# dbt Developer Hub > End user documentation, guides and technical reference for dbt ## API Reference ### About the Discovery API dbt platform | Starter, Enterprise, Enterprise+ Every time dbt runs a project, it generates and stores information about the project. The metadata includes details about your project’s models, sources, and other nodes along with their execution results. With the dbt Discovery API, you can query this comprehensive information to gain a better understanding of your DAG and the data it produces. By leveraging the metadata in dbt, you can create systems for data monitoring and alerting, lineage exploration, and automated reporting. This can help you improve data discovery, data quality, and pipeline operations within your organization. You can access the Discovery API through [ad hoc queries](https://docs.getdbt.com/docs/dbt-apis/discovery-querying.md), custom applications, a wide range of [partner ecosystem integrations](https://www.getdbt.com/product/integrations/) (like BI/analytics, catalog and governance, and quality and observability), and by using dbt features like [model timing](https://docs.getdbt.com/docs/deploy/run-visibility.md#model-timing-tab) and [data health tiles](https://docs.getdbt.com/docs/explore/data-tile.md). For request quotas and throttling behavior, refer to [API rate limits](https://docs.getdbt.com/docs/dbt-apis/rate-limits.md). [![A rich ecosystem for integration ](/img/docs/dbt-platform/discovery-api/discovery-api-figure.png?v=2 "A rich ecosystem for integration ")](#)A rich ecosystem for integration You can query the dbt metadata: * At the [environment](https://docs.getdbt.com/docs/environments-in-dbt.md) level for both the latest state (use the `environment` endpoint) and historical run results (use `modelHistoricalRuns`) of a dbt project in production. * At the job level for results on a specific dbt job run for a given resource type, like `models` or `test`. #### Prerequisites * You must have a dbt [multi-tenant](https://docs.getdbt.com/docs/platform/about-platform/tenancy.md#multi-tenant) or [single tenant](https://docs.getdbt.com/docs/platform/about-platform/tenancy.md#single-tenant) account. * You must be on a [Starter, Enterprise, or Enterprise+ plan](https://www.getdbt.com/pricing/). * Your projects must be on a dbt [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) or dbt version 1.0 or later. Refer to [Upgrade dbt version in Cloud](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) to upgrade. #### What you can use the Discovery API for Click the following tabs to learn more about the API's use cases, the analysis you can do, and the results you can achieve by integrating with it. To use the API directly or integrate your tool with it, refer to [Uses case and examples](https://docs.getdbt.com/docs/dbt-apis/discovery-use-cases-and-examples.md) for detailed information. ##### Performance Use the API to look at historical information like model build time to determine the health of your dbt projects. Finding inefficiencies in orchestration configurations can help decrease infrastructure costs and improve timeliness. To learn more about how to do this, refer to [Performance](https://docs.getdbt.com/docs/dbt-apis/discovery-use-cases-and-examples.md#performance). You can use, for example, the [model timing](https://docs.getdbt.com/docs/deploy/run-visibility.md#model-timing-tab) tab to help identify and optimize bottlenecks in model builds: [![Model timing visualization in dbt](/img/docs/dbt-platform/discovery-api/model-timing.png?v=2 "Model timing visualization in dbt")](#)Model timing visualization in dbt ##### Quality Use the API to determine if the data is accurate and up-to-date by monitoring test failures, source freshness, and run status. Accurate and reliable information is valuable for analytics, decisions, and monitoring to help prevent your organization from making bad decisions. To learn more about this, refer to [Quality](https://docs.getdbt.com/docs/dbt-apis/discovery-use-cases-and-examples.md#quality). When used with [webhooks](https://docs.getdbt.com/docs/deploy/webhooks.md), it can also help with detecting, investigating, and alerting issues. ##### Discovery Use the API to find and understand dbt assets in integrated tools using information like model and metric definitions, and column information. For more details, refer to [Discovery](https://docs.getdbt.com/docs/dbt-apis/discovery-use-cases-and-examples.md#discovery). Data producers must manage and organize data for stakeholders, while data consumers need to quickly and confidently analyze data on a large scale to make informed decisions that improve business outcomes and reduce organizational overhead. The API is useful for discovery data experiences in catalogs, analytics, apps, and machine learning (ML) tools. It can help you understand the origin and meaning of datasets for your analysis. [![Data lineage produced by dbt](/img/docs/collaborate/dbt-explorer/example-model-details.png?v=2 "Data lineage produced by dbt")](#)Data lineage produced by dbt ##### Governance Use the API to review who developed the models and who uses them to help establish standard practices for better governance. For more details, refer to [Governance](https://docs.getdbt.com/docs/dbt-apis/discovery-use-cases-and-examples.md#governance). ##### Development Use the API to review dataset changes and uses by examining exposures, lineage, and dependencies. From the investigation, you can learn how to define and build more effective dbt projects. For more details, refer to [Development](https://docs.getdbt.com/docs/dbt-apis/discovery-use-cases-and-examples.md#development). [![Use exposures to embed data health tiles in your dashboards to distill trust signals for data consumers.](/img/docs/collaborate/dbt-explorer/data-tile-pass.png?v=2 "Use exposures to embed data health tiles in your dashboards to distill trust signals for data consumers.")](#)Use exposures to embed data health tiles in your dashboards to distill trust signals for data consumers. #### Types of project state You can query these two types of [project state](https://docs.getdbt.com/docs/dbt-apis/project-state.md) at the environment level: * **Definition** — The logical state of a dbt project’s [resources](https://docs.getdbt.com/docs/build/projects.md) that update when the project is changed. * **Applied** — The output of successful dbt DAG execution that creates or describes the state of the database (for example: `dbt run`, `dbt test`, source freshness, and so on) These states allow you to easily examine the difference between a model’s definition and its applied state so you can get answers to questions like, did the model run? or did the run fail? Applied models exist as a table/view in the data platform given their most recent successful run. #### Related docs * [Use cases and examples for the Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-use-cases-and-examples.md) * [Query the Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-querying.md) * [Schema](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-job.md) --- ### About the Discovery API schema dbt platform | Starter, Enterprise, Enterprise+ With the Discovery API, you can query the metadata in dbt to learn more about your dbt deployments and the data they generate. You can analyze the data to make improvements. If you are new to the API, refer to [About the Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md) for an introduction. You might also find the [use cases and examples](https://docs.getdbt.com/docs/dbt-apis/discovery-use-cases-and-examples.md) helpful. The Discovery API *schema* provides all the pieces necessary to query and interact with the Discovery API. The most common queries use the `environment` endpoint: [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-environment.md) ###### [Environment schema](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-environment.md) [Query and compare a model’s definition (intended) and its applied (actual) state.](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-environment.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-environment-applied.md) ###### [Applied schema](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-environment-applied.md) [Query the actual state of objects and metadata in the warehouse after a \`dbt run\` or \`dbt build\`.](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-environment-applied.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-environment-definition.md) ###### [Definition schema](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-environment-definition.md) [Query intended state in project code and configuration defined in your dbt project.](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-environment-definition.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-environment-applied-modelHistoricalRuns.md) ###### [Model Historical Runs schema](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-environment-applied-modelHistoricalRuns.md) [Query information about a model's run history.](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-environment-applied-modelHistoricalRuns.md) --- ### Account-scoped personal access tokens dbt platform | Starter, Enterprise, Enterprise+ warning User API tokens have been deprecated and will no longer work. [Migrate](#migrate-deprecated-user-api-keys-to-personal-access-tokens) to personal access tokens to resume services. Each dbt user with a [Developer, Read-only, or IT license](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md) can create a new personal access token (PAT) to access the dbt API and dbt CLI. This token can execute queries against the dbt API on the user's behalf. To access dbt APIs and resources on behalf of the *account*, we recommend using service tokens instead. Learn more about [which token type you should use](https://docs.getdbt.com/docs/dbt-apis/authentication.md#which-token-type-should-you-use) to understand the token differences. PATs inherit the permissions of the user that created them. For example, if a developer-licensed user with Project Admin role access to specific projects creates a PAT, the token will get the Project Admin role with access to the same projects as the user. These tokens are also account-specific, so if a user has access to more than one dbt account with the same email address, they need to create a unique PAT for each one of these accounts. You can't use a PAT to create a [service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md). Requests to create a service token authenticated with a PAT return a `400` error — use an existing service token to create new ones instead. #### Create a personal access token Creating an account-scoped PAT requires only a few steps. 1. Navigate to your **Account Settings**, expand **API tokens** and click **Personal tokens**. 2. Click **Create personal access token**. 3. Give the token a descriptive name and click **Save**. 4. Copy the token before closing the window. *It will not be available after, and you will have to create a new token if you lose it.* To maintain best security practices, it's recommended that you regularly rotate your PATs. To do so, create a new token and delete the old one once it's in place. #### Delete a personal access token To permanently delete a PAT: 1. Navigate to your **Account Settings**, expand **API tokens** and click **Personal tokens**. 2. Find the token you want to delete and click "X" to the right of the token description fields. 3. **Confirm delete** and the token will no longer be valid. #### Migrate deprecated user API keys to personal access tokens The migration to PATs is critical if you are using user API keys today. The current API key is located under **Personal Settings → API Key**. There are a few things to understand if you are using a user API key today: * PATs are more secure. * To promote the least privilege and high-security assurance for your dbt accounts, we highly recommend moving to the new account-scoped PATs. * You must create and use unique tokens in each one of your dbt accounts that share the same email address. * For example, if belongs to two dbt accounts: Spice Harvesting Account and Guild Navigator Account. Before this release, the same API key was used to access both of these accounts. * After this release, Paul has to individually go into these accounts and create a unique PAT for each account he wants to access the API for. These PATs are account-specific and not user specific. * Cross-Account API endpoints will change in behavior when using PATs. * These are namely /v2/accounts and /v3/accounts. Since all PATs are now account specific, getting all accounts associated with a username cannot work. /v3/accounts will only return account metadata that’s relevant to the PAT that’s being used. * User account metadata will only contain information about the specific account under which the request is being made. * Any other accounts that belong to that user account will need to be requested through the PAT that belongs to that account. Undocumented APIs If you’re using any undocumented and unsupported API endpoints, please note that these can be deprecated without any notice. If you are using any undocumented endpoints and have use-cases that are not satisfied by the current API, please reach out to . ##### Using the personal access tokens Are you using a user API key today to access dbt APIs in any of your workflows? If not, you don’t have any action to take. If you are using a user API key, please follow the instructions below. 1. Make a list of all the places where you’re making a call to the dbt API using the dbt user API key. 2. Create a new PAT under **Account Settings → API Tokens → Personal Tokens.** For instructions, see [Create a personal access token](#create-a-personal-access-token). 3. Replace the API key in your APIs with the PAT you created. You can use a PAT wherever you previously used an API key. To replace the API key with a PAT, include the PAT in the Authorization header of your API requests. For example: `Authorization: Bearer `. Make sure to replace `` with the new PAT you created. note The option to rotate API keys is used for existing API keys, not for replacing them with PATs. You do not need to replace your API key with a PAT in the dbt UI. 4. Ensure that you’re using a PAT only where it's needed. For flows that require a service account, please use a service token. --- ### API rate limits dbt platform | Starter, Enterprise, Enterprise+ dbt applies request limits across the dbt platform APIs so integrations stay reliable and predictable. The [Administrative API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md) and the [Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md) do not share the same limit. You can send more requests per minute to admin endpoints under `/api/` and fewer to Discovery's GraphQL endpoints under `/graphql/`, because Discovery queries often return more data per call. You use the same credentials for both APIs: [personal access tokens](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md), [service account tokens](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md), and OAuth when your organization supports it. Refer to [Authentication tokens](https://docs.getdbt.com/docs/dbt-apis/authentication.md) to create and use those credentials, and the [APIs overview](https://docs.getdbt.com/docs/dbt-apis/overview.md) to learn what each API does. This page summarizes the default rate limits across the main API and integration surfaces: * [Administrative API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md): 5,000 requests per minute per account (`/api/`). * [Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md) (GraphQL): 500 requests per minute (`/graphql/`). * [SCIM and IdP provisioning](#scim-and-idp-provisioning): 20 requests every 5 seconds per account. * [Remote MCP](#remote-mcp): 5,000 requests per minute per IP (global API rate limit). * [Self-hosted MCP](#local-mcp): Uses the Administrative and Discovery API limits above. For SCIM scope, throttling responses, and identity provider behavior, see [SCIM and IdP provisioning](#scim-and-idp-provisioning) and [Set up SCIM](https://docs.getdbt.com/docs/platform/manage-access/scim.md). For MCP integration and usage patterns, see [Remote MCP](#remote-mcp) and [Self-hosted MCP](#local-mcp). #### Pagination and the Discovery API The most common way teams hit the Discovery API limit is to request very small pages in GraphQL (for example, `first: 1`) repeatedly. Each page is another request, so small pages require far more calls than fetching the same data in larger chunks. Use the largest page size that still works for you, up to what that query allows (often up to `500`). Full examples, including how to page with `after`, are in [Query the Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-querying.md). #### SCIM and IdP provisioning [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") For SCIM, the application rate limit of 20 requests every 5 seconds per account (Enterprise and Enterprise+) applies only to creating a user (`POST /api/v3/accounts/{account_id}/scim/v2/Users`) and replacing a user (`PUT /api/v3/accounts/{account_id}/scim/v2/Users/{user_id}`). dbt counts requests in a fixed 5-second window. It does not apply to `GET`, `PATCH`, or `DELETE` requests for users; to any Group operations (create, list, get, replace, patch, or delete); or to other SCIM endpoints such as Service Provider Config, schemas, resource types, or config token routes. Those operations are outside this limit but may still be subject to other platform rate limits. When you exceed that limit, dbt returns `429 Too Many Requests` with both `Retry-After` and `x-rate-limit-retry-after-seconds` (for example, Okta integrations often honor `Retry-After`, while SailPoint often honors `x-rate-limit-retry-after-seconds`). These headers indicate how long to wait before you send the next request. Configure your identity provider to read the header your provisioning stack supports, and to retry with exponential backoff when throttled. For configuration steps, use [Set up SCIM](https://docs.getdbt.com/docs/platform/manage-access/scim.md). For information on licenses, permissions, [SSO](https://docs.getdbt.com/docs/platform/manage-access/sso-overview.md), and how SCIM fits into account access, refer to [About user access in dbt](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md). #### Remote MCP [Remote MCP](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md) uses the same default global API rate limit as other dbt APIs: 5,000 requests per minute per IP. It uses the same authentication as other integrations (for example, personal access tokens, service account tokens, or OAuth where supported). Treat remote MCP automation like any other API client: avoid retrying without pausing between attempts. When you receive a `429` response, wait before trying again, and wait longer between retries if you continue to receive `429` responses. #### Self-hosted MCP [Self-hosted MCP](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md) calls the public [Administrative API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md) and [Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md) directly, so those limits apply: 5,000 requests per minute per account for `/api/` and 500 requests per minute for `/graphql/`, as summarized above. #### Exceeding the rate limit For the [Administrative API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md) and [Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md), if you exceed the limit, dbt returns `429 Too Many Requests` and enforces a five-minute cooldown. After five minutes, you can send requests again as usual. --- ### APIs overview dbt platform | Starter, Enterprise, Enterprise+ Accounts on the Starter, Enterprise, and Enterprise+ plans can query the dbt APIs. dbt provides the following APIs: * The [dbt Administrative API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md) can be used to administrate a dbt account. It can be called manually or with [the dbt Terraform provider](https://registry.terraform.io/providers/dbt-labs/dbtcloud/latest). * The [dbt Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md) can be used to fetch metadata related to the state and health of your dbt project. * The [Semantic Layer APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md) provides multiple API options which allow you to query your metrics defined in the Semantic Layer. If you want to learn more about webhooks, refer to [Webhooks for your jobs](https://docs.getdbt.com/docs/deploy/webhooks.md). For request quotas and throttling behavior, refer to [API rate limits](https://docs.getdbt.com/docs/dbt-apis/rate-limits.md). #### How to Access the APIs dbt supports two types of API Tokens: [personal access tokens](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) and [service account tokens](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md). Requests to the dbt APIs can be authorized using these tokens. --- ### Applied object schema dbt platform | Starter, Enterprise, Enterprise+ The applied object allows you to query information about a particular model based on `environmentId`. The [Example queries](#example/docs/dbt-apis-queries) illustrate a few fields you can query with this `environment` object. Refer to [Fields](#fields) to view the entire schema, which provides all possible fields you can query. `executionInfo` fields track two different concepts Within `executionInfo`, `lastRun*` fields reflect the most recent run *attempt* (regardless of outcome), while `execute*`, `executionTime`, `runGeneratedAt`, and `lastSuccess*` fields reflect the most recent *successful* materialization. Refer to [Project state](https://docs.getdbt.com/docs/dbt-apis/project-state.md#definition-logical-vs-applied-state-of-dbt-nodes) for more information. ##### Example queries You can use your production environment's `id`: ```graphql query Example { environment(id: 834){ # Get the latest state of the production environment applied { # The state of an executed node as it exists as an object in the database models(first: 100){ # Pagination to ensure manageable response for large projects edges { node { uniqueId, name, description, rawCode, compiledCode, # Basic properties database, schema, alias, # Table/view identifier (can also filter by) executionInfo {executeCompletedAt, executionTime}, # Metadata from when the model was built tests {name, executionInfo{lastRunStatus, lastRunError}}, # Latest test results catalog {columns {name, description, type}, stats {label, value}}, # Catalog info ancestors(types:[Source]) {name, ...on SourceAppliedStateNestedNode {freshness{maxLoadedAt, freshnessStatus}}}, # Source freshness } children {name, resourceType}}} # Immediate dependencies in lineage totalCount } # Number of models in the project } } } ``` ##### Fields When querying the `applied` field of `environment`, you can use the following fields. ### Fetching data... --- ### Authentication tokens dbt platform | Starter, Enterprise, Enterprise+ [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) ###### [Personal access tokens](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) [Learn about user tokens and how to use them to execute queries against the dbt API.](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) ###### [Service account tokens](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) [Learn how to use service account tokens to securely authenticate with dbt APIs for system-level integrations.](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) #### Types of API access tokens **Personal access tokens:** Preferred and secure way of accessing dbt APIs on behalf of a user. PATs are scoped to an account and can be enhanced with more granularity and control. **Service tokens:** Service tokens are similar to service accounts and are the preferred method to enable access on behalf of the dbt account. ##### Which token type should you use You should use service tokens broadly for any production workflow where you need a service account. You should use PATs only for developmental workflows *or* dbt client workflows that require user context. The following examples show you when to use a personal access token (PAT) or a service token: * **Connecting a partner integration to dbt** — Some examples include the [Semantic Layer Google Sheets integration](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md), Hightouch, Datafold, a custom app you’ve created, etc. These types of integrations should use a service token instead of a PAT because service tokens give you visibility, and you can scope them to only what the integration needs and ensure the least privilege. We highly recommend switching to a service token if you’re using a personal access token for these integrations today. * **Production Terraform** — Use a service token since this is a production workflow and is acting as a service account and not a user account. * **dbt platform CLI** — Use a PAT since the dbt platform CLI works within the context of a user (the user is making the requests and has to operate within the context of their user account). * **Testing a custom script and staging Terraform or Postman** — We recommend using a PAT as this is a developmental workflow and is scoped to the user making the changes. When you push this script or Terraform into production, use a service token instead. * **API endpoints requiring user context** — Use PATs to authenticate to any API endpoint that requires user context (for example, endpoints to create and update user credentials). --- ### dbt Administrative API dbt platform | Starter, Enterprise, Enterprise+ The dbt Administrative API is enabled by default for [Starter, Enterprise, and Enterprise+ plans](https://www.getdbt.com/pricing/). It can be used to: * Download artifacts after a job has completed * Kick off a job run from an orchestration tool * Manage your dbt account * and more dbt currently supports two versions of the Administrative API: v2 and v3. In general, v3 is the recommended version to use, but we don't yet have all our v2 routes upgraded to v3. We're currently working on this. If you can't find something in our v3 docs, check out the shorter list of v2 endpoints because you might find it there. Many endpoints of the Administrative API can also be called through the [dbt Terraform provider](https://registry.terraform.io/providers/dbt-labs/dbtcloud/latest). The built-in documentation on the Terraform registry contains [a guide on how to get started with the provider](https://registry.terraform.io/providers/dbt-labs/dbtcloud/latest/docs/guides/1_getting_started) as well as [a page showing all the Terraform resources available](https://registry.terraform.io/providers/dbt-labs/dbtcloud/latest/docs/guides/99_list_resources) to configure. For request quotas and throttling behavior, refer to [API rate limits](https://docs.getdbt.com/docs/dbt-apis/rate-limits.md). [![](/img/icons/pencil-paper.svg)](https://docs.getdbt.com/dbt-cloud/api-v2) ###### [API v2](https://docs.getdbt.com/dbt-cloud/api-v2) [Our legacy API version, with limited endpoints and features. Contains information not available in v3.](https://docs.getdbt.com/dbt-cloud/api-v2) [![](/img/icons/pencil-paper.svg)](https://docs.getdbt.com/dbt-cloud/api-v3) ###### [API v3](https://docs.getdbt.com/dbt-cloud/api-v3) [Our latest API version, with new endpoints and features.](https://docs.getdbt.com/dbt-cloud/api-v3) [![](/img/icons/pencil-paper.svg)](https://registry.terraform.io/providers/dbt-labs/dbtcloud/latest) ###### [dbt Terraform provider](https://registry.terraform.io/providers/dbt-labs/dbtcloud/latest) [The Terraform provider maintained by dbt Labs which can be used to manage a dbt account.](https://registry.terraform.io/providers/dbt-labs/dbtcloud/latest) [](https://registry.terraform.io/providers/dbt-labs/dbtcloud/latest) --- ### Definition object schema dbt platform | Starter, Enterprise, Enterprise+ The definition object allows you to query the logical state of a given project node given its most recent manifest generated models. The [Example queries](#example-queries) illustrate a few fields you can query with this `definition` object. Refer to [Fields](#fields) to view the entire schema, which provides all possible fields you can query. ##### Example queries You can use your production environment's `id`: ```graphql query Example { environment(id: 834){ # Get the latest state of the production environment definition { # The logical state of a given project node given its most recent manifest generated models(first: 100, filter:{access:public}){ # Filter on model access (or other properties) edges { node { rawCode, # Compare to see if/how the model has changed since the last build jobDefinitionId, runGeneratedAt, # When the code was last compiled or run contractEnforced, group, version}}} # Model governance } } } ``` ##### Fields When querying the `definition` field of `environment`, you can use the following fields. ### Fetching data... --- ### Environment object schema dbt platform | Starter, Enterprise, Enterprise+ You can use the environment object to query and compare definition (intended) and applied (actual) states for nodes (models, seeds, snapshots, models, and more) in your dbt project. For example, you specify an `environmentId` to learn more about a particular model (or other node type) in that environment. The [Example queries](#example-queries) illustrate a few fields you can query with this `environment` object. Refer to [Fields](#fields) to view the entire schema, which provides all possible fields you can query. ##### Arguments When querying for `environment`, you can use the following arguments. ### Fetching data... ##### Example queries You can use your production environment's `id`: ```graphql query Example { environment(id: 834){ # Get the latest state of the production environment applied { # The state of an executed node as it exists as an object in the database models(first: 100){ # Pagination to ensure manageable response for large projects edges { node { uniqueId, name, description, rawCode, compiledCode, # Basic properties database, schema, alias, # Table/view identifier (can also filter by) executionInfo {executeCompletedAt, executionTime}, # Metadata from when the model was built tests {name, executionInfo{lastRunStatus, lastRunError}}, # Latest test results catalog {columns {name, description, type}, stats {label, value}}, # Catalog info ancestors(types:[Source]) {name, ...on SourceAppliedStateNode {freshness{maxLoadedAt, freshnessStatus}}}, # Source freshness } children {name, resourceType}}} # Immediate dependencies in lineage totalCount } # Number of models in the project } definition { # The logical state of a given project node given its most recent manifest generated models(first: 100, filter:{access:public}){ # Filter on model access (or other properties) edges { node { rawCode, # Compare to see if/how the model has changed since the last build jobDefinitionId, runGeneratedAt, # When the code was last compiled or run contractEnforced, group, version}}} # Model governance } } ``` With the deprecation of the data type `Int` for `id`, below is an example of replacing it with `BigInt`: ```graphql query ($environmentId: BigInt!, $first: Int!) { environment(id: $environmentId) { applied { models(first: $first) { edges { node { uniqueId executionInfo { lastRunId } } } } } } } ``` With the deprecation of `modelByEnvironment`, below is an example of replacing it with `environment`: ```graphql query ($environmentId: BigInt!, $uniqueId: String) { environment(id: $environmentId) { applied { modelHistoricalRuns(uniqueId: $uniqueId) { uniqueId executionTime executeCompletedAt } } } } ``` ##### Fields When querying an `environment`, you can use the following fields. ### Fetching data... For details on querying the `applied` field of `environment`, you can visit: [Applied](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-environment-applied.md) For details querying the `definition` field of `environment`, you can visit: [Definition](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-environment-definition.md) --- ### Exposure object schema dbt platform | Starter, Enterprise, Enterprise+ The exposure object allows you to query information about a particular exposure. To learn more, refer to [Add Exposures to your DAG](https://docs.getdbt.com/docs/build/exposures.md). ##### Arguments When querying for an `exposure`, the following arguments are available. ### Fetching data... Below we show some illustrative example queries and outline the schema of the exposure object. ##### Example query The example below queries information about an exposure including the owner's name and email, the URL, and information about parent sources and parent models. ```graphql { job(id: 123) { exposure(name: "my_awesome_exposure") { runId projectId name uniqueId resourceType ownerName url ownerEmail parentsSources { uniqueId sourceName name state maxLoadedAt criteria { warnAfter { period count } errorAfter { period count } } maxLoadedAtTimeAgoInS } parentsModels { uniqueId } } } } ``` ##### Fields When querying for an `exposure`, the following fields are available: ### Fetching data... --- ### Exposure tile object schema dbt platform | Starter, Enterprise, Enterprise+ [Exposure health tiles](https://docs.getdbt.com/docs/explore/data-tile.md) distill data health signals for data consumers and can be embedded in downstream tools. You can query information on these tiles from the Discovery API. The [Example query](#example-query) illustrates a few fields you can query with the `exposureTile` object. Refer to [Fields](#fields) to view the entire schema, which provides all possible fields you can query. ##### Arguments When querying for `exposureTile`, you can use the following arguments: ### Fetching data... ##### Example query You can specify the `environmentId` and filter by a model's `uniqueId` to understand the data quality and metadata information for the exposure tile associated with the `customers` model in the `marketing` package: ```graphql query { environment(id: 834) { applied { exposureTile( filter: {uniqueId: "model.marketing.customers"} # Use this format for unique ID: RESOURCE_TYPE.PACKAGE_NAME.RESOURCE_NAME ) { accountId # The account ID of this node environmentId projectId exposureType filePath quality } } } } ``` ##### Fields When querying for `exposureTile`, you can use the following fields: ### Fetching data... --- ### Exposures object schema dbt platform | Starter, Enterprise, Enterprise+ [Exposures](https://docs.getdbt.com/docs/build/exposures.md) are dbt resources that represent downstream uses of your project, such as dashboards, applications, or data science pipelines. You can query exposures through the Discovery API to understand which assets depend on your models. The [Example query](#example-query) illustrates a few fields you can query with the `exposures` object. Refer to [Fields](#fields) to view the entire schema, which provides all possible fields you can query. ##### Arguments When querying for `exposures`, you can use the following arguments: ### Fetching data... ##### Example query You can specify the `environmentId`, `first: 100`, and filter by model `uniqueIds` to return all the downstream exposures (dashboards, applications, etc.) that depend on the `customers` model in the `marketing` package, limited to the first 100 results: ```graphql query { environment(id: 834) { applied { exposures( filter: { uniqueIds: ["model.marketing.customers"] # Use this format for unique ID: RESOURCE_TYPE.PACKAGE_NAME.RESOURCE_NAME }, first: 100 ) { edges { node { accountId exposureType fqn projectId url } } } } } } ``` ##### Fields When querying for `exposures`, you can use the following fields: ### Fetching data... ##### Key fields from nodes ### Fetching data... --- ### Exposures object schema dbt platform | Starter, Enterprise, Enterprise+ The exposures object allows you to query information about all exposures in a given job. To learn more, refer to [Add Exposures to your DAG](https://docs.getdbt.com/docs/build/exposures.md). ##### Arguments When querying for `exposures`, the following arguments are available. ### Fetching data... Below we show some illustrative example queries and outline the schema of the exposures object. ##### Example query The example below queries information about all exposures in a given job including the owner's name and email, the URL, and information about parent sources and parent models for each exposure. ```graphql { job(id: 123) { exposures(first: 10, after: "{somePaginationCursorValue}") { runId projectId name uniqueId resourceType ownerName url ownerEmail paginationCursor parentsSources { uniqueId sourceName name state maxLoadedAt criteria { warnAfter { period count } errorAfter { period count } } maxLoadedAtTimeAgoInS } parentsModels { uniqueId } } } } ``` ##### Fields When querying for `exposures`, the following fields are available: ### Fetching data... --- ### GraphQL dbt platform | Starter, Enterprise, Enterprise+ [GraphQL](https://graphql.org/) (GQL) is an open-source query language for APIs. It offers a more efficient and flexible approach compared to traditional RESTful APIs. With GraphQL, users can request specific data using a single query, reducing the need for many server round trips. This improves performance and minimizes network overhead. GraphQL has several advantages, such as self-documenting, having a strong typing system, supporting versioning and evolution, enabling rapid development, and having a robust ecosystem. These features make GraphQL a powerful choice for APIs prioritizing flexibility, performance, and developer productivity. #### dbt Semantic Layer GraphQL API The Semantic Layer GraphQL API allows you to explore and query metrics and dimensions. Due to its self-documenting nature, you can explore the calls conveniently through a schema explorer. The schema explorer URLs vary depending on your [deployment region](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). Use the following table to find the right link for your region: | Deployment type | Schema explorer URL | | -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | North America multi-tenant | | | EMEA multi-tenant | | | APAC multi-tenant | | | Single tenant | `https://semantic-layer.YOUR_ACCESS_URL/api/graphql`

Replace `YOUR_ACCESS_URL` with your specific account prefix followed by the appropriate Access URL for your region and plan. | | Multi-cell | `https://YOUR_ACCOUNT_PREFIX.semantic-layer.REGION.dbt.com/api/graphql`

Replace `YOUR_ACCOUNT_PREFIX` with your specific account identifier and `REGION` with your location, which could be `us1.dbt.com`. | **Example** * If your Single tenant access URL is `ABC123.getdbt.com`, your schema explorer URL will be `https://semantic-layer.ABC123.getdbt.com/api/graphql`. dbt Partners can use the Semantic Layer GraphQL API to build an integration with the Semantic Layer. Note that the Semantic Layer GraphQL API doesn't support `ref` to call dbt objects. Instead, use the complete qualified table name. If you're using dbt macros at query time to calculate your metrics, you should move those calculations into your Semantic Layer metric definitions as code. #### Requirements to use the GraphQL API * A dbt project on dbt v1.6 or higher * Metrics are defined and configured * A dbt [service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) with "Semantic Layer Only” and "Metadata Only" permissions or a [personal access token](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) #### Using the GraphQL API If you're a dbt user or partner with access to dbt and the [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md), you can [set up](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md) and test this API with data from your own instance by configuring the Semantic Layer and obtaining the right GQL connection parameters described in this document. Refer to [Get started with the Semantic Layer](https://docs.getdbt.com/guides/sl-qs.md) for more info. Authentication uses either a dbt [service account token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) or a [personal access token](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) passed through a header as follows. To explore the schema, you can enter this information in the "header" section. ```shell {"Authorization": "Bearer "} ``` Each GQL request also requires a dbt `environmentId`. The API uses both the service or personal token in the header and `environmentId` for authentication. ##### Metadata calls ###### Fetch data platform dialect In some cases in your application, it may be useful to know the dialect or data platform that's internally used for the Semantic Layer connection (such as if you are building `where` filters from a user interface rather than user-inputted SQL). The GraphQL API has an easy way to fetch this with the following query: ```graphql { environmentInfo(environmentId: BigInt!) { dialect } } ``` ###### Fetch available metrics ```graphql metricsPaginated( environmentId: BigInt! search: String = null groupBy: [GroupByInput!] = null pageNum: Int! = 1 pageSize: Int = null ): MetricResultPage! { items: [Metric!]! pageNum: Int! pageSize: Int totalItems: Int! totalPages: Int! } ``` ###### Fetch available dimensions for metrics ```graphql dimensionsPaginated( environmentId: BigInt! metrics: [MetricInput!]! search: String = null pageNum: Int! = 1 pageSize: Int = null ): DimensionResultPage! { items: [Dimension!]! pageNum: Int! pageSize: Int totalItems: Int! totalPages: Int! } ``` ###### Fetch available granularities given metrics Note: This call for `queryableGranularities` returns only queryable granularities for metric time - the primary time dimension across all metrics selected. ```graphql queryableGranularities( environmentId: BigInt! metrics: [MetricInput!]! ): [TimeGranularity!]! ``` You can also get queryable granularities for all other dimensions using the `dimensions` call: ```graphql { dimensionsPaginated(environmentId: BigInt!, metrics:[{name:"order_total"}]) { items { name queryableGranularities # --> ["DAY", "WEEK", "MONTH", "QUARTER", "YEAR"] } } } ``` You can also optionally access it from the metrics endpoint: ```graphql { metricsPaginated(environmentId: BigInt!) { items { name dimensions { name queryableGranularities } } } } ``` ###### Fetch entities ```graphql entitiesPaginated( environmentId: BigInt! metrics: [MetricInput!] = null search: String = null pageNum: Int! = 1 pageSize: Int = null ): EntityResultPage! { items: [Entity!]! pageNum: Int! pageSize: Int totalItems: Int! totalPages: Int! } ``` ###### Fetch entities and dimensions to group metrics ```graphql groupBysPaginated( environmentId: BigInt! metrics: [MetricInput!] = null search: String = null pageNum: Int! = 1 pageSize: Int = null ): EntityDimensionResultPage! { items: [EntityDimension!]! pageNum: Int! pageSize: Int totalItems: Int! totalPages: Int! } ``` ###### Metric types ```graphql Metric { name: String! description: String type: MetricType! typeParams: MetricTypeParams! filter: WhereFilter dimensions: [Dimension!]! queryableGranularities: [TimeGranularity!]! } ``` ```text MetricType = [SIMPLE, RATIO, CUMULATIVE, DERIVED] ``` ###### Metric type parameters (Applies to dbt v1.12 and later) ```graphql MetricTypeParams { numerator: MetricInput denominator: MetricInput expr: String window: MetricTimeWindow grainToDate: TimeGranularity metrics: [MetricInput!] } ``` ###### Dimension types ```graphql Dimension { name: String! description: String type: DimensionType! typeParams: DimensionTypeParams isPartition: Boolean! expr: String queryableGranularities: [TimeGranularity!]! } ``` ```text DimensionType = [CATEGORICAL, TIME] ``` ###### List saved queries List all saved queries for the specified environment: ```graphql savedQueriesPaginated( environmentId: BigInt! search: String = null pageNum: Int! = 1 pageSize: Int = null ): SavedQueryResultPage! { items: [SavedQuery!]! pageNum: Int! pageSize: Int totalItems: Int! totalPages: Int! } ``` ###### List a saved query List a single saved query using environment ID and query name: ```graphql { savedQuery(environmentId: "123", savedQueryName: "query_name") { name description label queryParams { metrics { name } groupBy { name grain datePart } where { whereSqlTemplate } } } } ``` ##### Querying When querying for data, *either* a `groupBy` *or* a `metrics` selection is required. The following section provides examples of how to query metrics: * [Create query](#create-metric-query) * [Fetch query result](#fetch-query-result) ###### Create query ```graphql createQuery( environmentId: BigInt! metrics: [MetricInput!]! groupBy: [GroupByInput!] = null limit: Int = null where: [WhereInput!] = null order: [OrderByInput!] = null ): CreateQueryResult ``` ```graphql MetricInput { name: String! alias: String! } GroupByInput { name: String! grain: TimeGranularity = null } WhereInput { sql: String! } OrderByinput { # -- pass one and only one of metric or groupBy metric: MetricInput = null groupBy: GroupByInput = null descending: Boolean! = false } ``` ###### Fetch query result ```graphql query( environmentId: BigInt! queryId: String! ): QueryResult! ``` The GraphQL API uses a polling process for querying since queries can be long-running in some cases. It works by first creating a query with a mutation, \`createQuery, which returns a query ID. This ID is then used to continuously check (poll) for the results and status of your query. The typical flow would look as follows: 1. Kick off a query ```graphql mutation { createQuery( environmentId: 123456 metrics: [{name: "order_total"}] groupBy: [{name: "metric_time"}] ) { queryId # => Returns 'QueryID_12345678' } } ``` 2. Poll for results ```graphql { query(environmentId: 123456, queryId: "QueryID_12345678") { sql status error totalPages jsonResult arrowResult } } ``` 3. Keep querying 2. at an appropriate interval until status is `FAILED` or `SUCCESSFUL` ##### Output format and pagination ###### Output format By default, the output is in Arrow format. You can switch to JSON format using the following parameter. However, due to performance limitations, we recommend using the JSON parameter for testing and validation. The JSON received is a base64 encoded string. To access it, you can decode it using a base64 decoder. The JSON is created from pandas, which means you can change it back to a dataframe using `pandas.read_json(json, orient="table")`. Or you can work with the data directly using `json["data"]`, and find the table schema using `json["schema"]["fields"]`. Alternatively, you can pass `encoded:false` to the jsonResult field to get a raw JSON string directly. ```graphql { query(environmentId: BigInt!, queryId: Int!, pageNum: Int! = 1) { sql status error totalPages arrowResult jsonResult(orient: PandasJsonOrient! = TABLE, encoded: Boolean! = true) } } ``` The results default to the table but you can change it to any [pandas](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_json.html) supported value. ###### Pagination By default, we return 1024 rows per page. If your result set exceeds this, you need to increase the page number using the `pageNum` option. ##### Run a Python query The `arrowResult` in the GraphQL query response is a byte dump, which isn't visually useful. You can convert this byte data into an Arrow table using any Arrow-supported language. Refer to the following Python example explaining how to query and decode the arrow result: ```python import base64 import pyarrow as pa import time headers = {"Authorization":"Bearer "} query_result_request = """ { query(environmentId: 70, queryId: "12345678") { sql status error arrowResult } } """ while True: gql_response = requests.post( "https://semantic-layer.cloud.getdbt.com/api/graphql", json={"query": query_result_request}, headers=headers, ) if gql_response.json()["data"]["status"] in ["FAILED", "SUCCESSFUL"]: break # Set an appropriate interval between polling requests time.sleep(1) """ gql_response.json() => { "data": { "query": { "sql": "SELECT\n ordered_at AS metric_time__day\n , SUM(order_total) AS order_total\nFROM semantic_layer.orders orders_src_1\nGROUP BY\n ordered_at", "status": "SUCCESSFUL", "error": null, "arrowResult": "arrow-byte-data" } } } """ def to_arrow_table(byte_string: str) -> pa.Table: """Get a raw base64 string and convert to an Arrow Table.""" with pa.ipc.open_stream(base64.b64decode(byte_string)) as reader: return pa.Table.from_batches(reader, reader.schema) arrow_table = to_arrow_table(gql_response.json()["data"]["query"]["arrowResult"]) # Perform whatever functionality is available, like convert to a pandas table. print(arrow_table.to_pandas()) """ order_total ordered_at 3 2023-08-07 112 2023-08-08 12 2023-08-09 5123 2023-08-10 """ ``` ##### Additional create query examples The following section provides query examples for the GraphQL API, such as how to query metrics, dimensions, where filters, and more: * [Query metric alias](#query-metric-alias) — Query with metric alias, which allows you to use simpler or more intuitive names for metrics instead of their full definitions. * [Query with a time grain](#query-with-a-time-grain) — Fetch multiple metrics with a change in time dimension granularities. * [Query multiple metrics and multiple dimensions](#query-multiple-metrics-and-multiple-dimensions) — Select common dimensions for multiple metrics. * [Query a categorical dimension on its own](#query-a-categorical-dimension-on-its-own) — Group by a categorical dimension. * [Query with a where filter](#query-with-a-where-filter) — Use the `where` parameter to filter on dimensions and entities using parameters. * [Query with order](#query-with-order) — Query with `orderBy`, accepts basic string that's a Dimension, Metric, or Entity. Defaults to ascending order. * [Query with limit](#query-with-limit) — Query using a `limit` clause. * [Query saved queries](#query-saved-queries) — Query using a saved query using the `savedQuery` parameter for frequently used queries. * [Query with just compiling SQL](#query-with-just-compiling-sql) — Query using a compile keyword using the `compileSql` mutation. * [Query records](#query-records) — View all the queries made in your project. ###### Query metric alias ```graphql mutation { createQuery( environmentId: "123" metrics: [{name: "metric_name", alias: "metric_alias"}] ) { ... } } ``` ###### Query with a time grain ```graphql mutation { createQuery( environmentId: "123" metrics: [{name: "order_total"}] groupBy: [{name: "metric_time", grain: MONTH}] ) { queryId } } ``` Note that when using granularity in the query, the output of a time dimension with a time grain applied to it always takes the form of a dimension name appended with a double underscore and the granularity level - `{time_dimension_name}__{DAY|WEEK|MONTH|QUARTER|YEAR}`. Even if no granularity is specified, it will also always have a granularity appended to it and will default to the lowest available (usually daily for most data sources). It is encouraged to specify a granularity when using time dimensions so that there won't be any unexpected results with the output data. ###### Query multiple metrics and multiple dimensions ```graphql mutation { createQuery( environmentId: "123" metrics: [{name: "food_order_amount"}, {name: "order_gross_profit"}] groupBy: [{name: "metric_time", grain: MONTH}, {name: "customer__customer_type"}] ) { queryId } } ``` ###### Query a categorical dimension on its own ```graphql mutation { createQuery( environmentId: "123" groupBy: [{name: "customer__customer_type"}] ) { queryId } } ``` ###### Query with a where filter The `where` filter takes a list argument (or a string for a single input). Depending on the object you are filtering, there are a couple of parameters: * `Dimension()` — Used for any categorical or time dimensions. For example, `Dimension('metric_time').grain('week')` or `Dimension('customer__country')`. * `Entity()` — Used for entities like primary and foreign keys, such as `Entity('order_id')`. Note: If you prefer a `where` clause with a more explicit path, you can optionally use `TimeDimension()` to separate categorical dimensions from time ones. The `TimeDimension` input takes the time dimension and optionally the granularity level. `TimeDimension('metric_time', 'month')`. ```graphql mutation { createQuery( environmentId: "123" metrics:[{name: "order_total"}] groupBy:[{name: "customer__customer_type"}, {name: "metric_time", grain: month}] where:[{sql: "{{ Dimension('customer__customer_type') }} = 'new'"}, {sql:"{{ Dimension('metric_time').grain('month') }} > '2022-10-01'"}] ) { queryId } } ``` ###### Multi-hop joins In cases where you need to query across multiple related tables (multi-hop joins), use the `entity_path` argument to specify the path between related entities. The following are examples of how you can define these joins: * In this example, you're querying the `location_name` dimension but specifying that it should be joined using the `order_id` field. ```sql {{Dimension('location__location_name', entity_path=['order_id'])}} ``` * In this example, the `salesforce_account_owner` dimension is joined to the `region` field, with the path going through `salesforce_account`. ```sql {{ Dimension('salesforce_account_owner__region',['salesforce_account']) }} ``` ###### Query with order ```graphql mutation { createQuery( environmentId: "123" metrics: [{name: "order_total"}] groupBy: [{name: "metric_time", grain: MONTH}] orderBy: [{metric: {name: "order_total"}}, {groupBy: {name: "metric_time", grain: MONTH}, descending:true}] ) { queryId } } ``` ###### Query with limit ```graphql mutation { createQuery( environmentId: "123" metrics: [{name:"food_order_amount"}, {name: "order_gross_profit"}] groupBy: [{name:"metric_time", grain: MONTH}, {name: "customer__customer_type"}] limit: 10 ) { queryId } } ``` ###### Query saved queries This takes the same inputs as the `createQuery` mutation, but includes the field `savedQuery`. You can use this for frequently used queries. ```graphql mutation { createQuery( environmentId: "123" savedQuery: "new_customer_orders" ) { queryId } } ``` A note on querying saved queries When querying [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md),you can use parameters such as `where`, `limit`, `order`, `compile`, and so on. However, keep in mind that you can't access `metric` or `group_by` parameters in this context. This is because they are predetermined and fixed parameters for saved queries, and you can't change them at query time. If you would like to query more metrics or dimensions, you can build the query using the standard format. ###### Query with just compiling SQL This takes the same inputs as the `createQuery` mutation. ```graphql mutation { compileSql( environmentId: "123" metrics: [{name:"food_order_amount"} {name:"order_gross_profit"}] groupBy: [{name:"metric_time", grain: MONTH}, {name:"customer__customer_type"}] ) { sql } } ``` ###### Query records Use this endpoint to view all the queries made in your project. This covers both Insights and Semantic Layer queries. ```graphql { queryRecords( environmentId:123 ) { items { queryId status startTime endTime connectionDetails sqlDialect connectionSchema error queryDetails { ... on SemanticLayerQueryDetails { params { type metrics { name } groupBy { name grain } limit where { sql } orderBy { groupBy { name grain } metric { name } descending } savedQuery } } ... on RawSqlQueryDetails { queryStr compiledSql numCols queryDescription queryTitle } } } totalItems pageNum pageSize } } ``` --- ### JDBC API dbt platform | Starter, Enterprise, Enterprise+ The Semantic Layer Java Database Connectivity (JDBC) API enables users to query metrics and dimensions using the JDBC protocol, while also providing standard metadata functionality. A JDBC driver is a software component enabling a Java application to interact with a data platform. Here's some more information about our JDBC API: * The Semantic Layer JDBC API utilizes the open-source JDBC driver with ArrowFlight SQL protocol. * You can download the JDBC driver from [Maven](https://search.maven.org/remotecontent?filepath=org/apache/arrow/flight-sql-jdbc-driver/12.0.0/flight-sql-jdbc-driver-12.0.0.jar). * The Semantic Layer supports ArrowFlight SQL driver version 12.0.0 and higher. * You can embed the driver into your application stack as needed, and you can use dbt Labs' [example project](https://github.com/dbt-labs/example-semantic-layer-clients) for reference. * If you’re a partner or user building a homegrown application, you’ll need to install an AWS root CA to the Java Trust [documentation](https://www.amazontrust.com/repository/) (specific to Java and JDBC call). dbt Labs partners can use the JDBC API to build integrations in their tools with the Semantic Layer #### Using the JDBC API If you are a dbt user or partner with access to dbt and the [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md), you can [setup](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md) and test this API with data from your own instance by configuring the Semantic Layer and obtaining the right JDBC connection parameters described in this document. You *may* be able to use our JDBC API with tools that do not have an official integration with the Semantic Layer. If the tool you use allows you to write SQL and either supports a generic JDBC driver option (such as DataGrip) or supports Dremio and uses ArrowFlightSQL driver version 12.0.0 or higher, you can access the Semantic Layer API. Refer to [Get started with the Semantic Layer](https://docs.getdbt.com/guides/sl-qs.md) for more info. Note that the Semantic Layer GraphQL API doesn't support `ref` to call dbt objects. Instead, use the complete qualified table name. If you're using dbt macros at query time to calculate your metrics, you should move those calculations into your Semantic Layer metric definitions as code. #### Authentication dbt authorizes requests to the Semantic Layer API. You need to provide an Environment ID, Host, and [service account tokens](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) or [personal access tokens](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md). #### Connection parameters The JDBC connection requires a few different connection parameters. This is an example of a URL connection string and the individual components: ```text jdbc:arrow-flight-sql://semantic-layer.cloud.getdbt.com:443?&environmentId=202339&token=AUTHENTICATION_TOKEN ``` | JDBC parameter | Description | Example | | --------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | | `jdbc:arrow-flight-sql://` | The protocol for the JDBC driver. | `jdbc:arrow-flight-sql://` | | `semantic-layer.cloud.getdbt.com` | The [access URL](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for your account's dbt region. You must always add the `semantic-layer` prefix before the access URL. | For dbt deployment hosted in North America, use `semantic-layer.cloud.getdbt.com` | | `environmentId` | The unique identifier for the dbt production environment, you can retrieve this from the dbt URL
when you navigate to **Environments** under **Deploy**. | If your URL ends with `.../environments/222222`, your `environmentId` is `222222`

| | `AUTHENTICATION_TOKEN` | You can use either a dbt [service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) with “Semantic Layer Only” and "Metadata Only" permissions or a dbt [personal access token](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md). Create a new service or personal token on the **Account Settings** page. | `token=AUTHENTICATION_TOKEN` | \*Note — If you're testing locally on a tool like DataGrip, you may also have to provide the following variable at the end or beginning of the JDBC URL `&disableCertificateVerification=true`. #### Querying the API for metadata The Semantic Layer JDBC API has built-in metadata calls which can provide a user with information about their metrics and dimensions. Expand the following toggles for examples and metadata commands:  Fetch defined metrics You can use this query to fetch all defined metrics in your dbt project: ```bash select * from {{ semantic_layer.metrics() }} ```  Fetch dimension for a metric You can use this query to fetch all dimensions for a metric. Note, metrics is a required argument that lists one or multiple metrics in it. ```bash select * from {{ semantic_layer.dimensions(metrics=['food_order_amount'])}} ```  Fetch granularities for metrics You can use this query to fetch queryable granularities for a list of metrics. This API request allows you to only show the time granularities that make sense for the primary time dimension of the metrics (such as metric\_time), but if you want queryable granularities for other time dimensions, you can use the dimensions() call, and find the column queryable\_granularities. Note, metrics is a required argument that lists one or multiple metrics. ```bash select * from {{ semantic_layer.queryable_granularities(metrics=['food_order_amount', 'order_gross_profit'])}} ```  Fetch available metrics given dimensions You can use this query to fetch available metrics given dimensions. This command is essentially the opposite of getting dimensions given a list of metrics. Note, group\_by is a required argument that lists one or multiple dimensions. ```bash select * from {{ semantic_layer.metrics_for_dimensions(group_by=['customer__customer_type']) }} ```  Fetch granularities for all time dimensions You can use this example query to fetch available granularities for all time dimensions (the similar queryable granularities API call only returns granularities for the primary time dimensions for metrics). The following call is a derivative of the dimensions() call and specifically selects the granularity field. ```bash select NAME, QUERYABLE_GRANULARITIES from {{ semantic_layer.dimensions( metrics=["order_total"] ) }} ```  Paginate metadata calls In the case when you don't want to return the full result set from a metadata call, you can paginate the results for both `semantic_layer.metrics()` and `semantic_layer.dimensions()` calls using the `page_size` and `page_number` parameters. * `page_size`: This is an optional variable which sets the number of records per page. If left as None, there is no page limit. * `page_number`: This is an optional variable which specifies the page number to retrieve. Defaults to `1` (first page) if not specified. Examples: ```sql -- Retrieves the 5th page with a page size of 10 metrics select * from {{ semantic_layer.metrics(page_size=10, page_number=5) }} -- Retrieves the 1st page with a page size of 10 metrics select * from {{ semantic_layer.metrics(page_size=10) }} -- Retrieves all metrics without pagination select * from {{ semantic_layer.metrics() }} ``` You can use the same pagination parameters for `semantic_layer.dimensions(...)`.  List saved queries You can use this example query to list all available saved queries in your dbt project. **Command** ```bash select * from semantic_layer.saved_queries() ``` **Output** ```bash | NAME | DESCRIPTION | LABEL | METRICS | GROUP_BY | WHERE_FILTER | ```  Fetch metric aliases You can query metrics using aliases for simpler or more intuitive names, even if the alias isn't defined in the metric configuration. The query returns the alias as the metric name, for example: ```sql select * from {{ semantic_layer.query(metrics=[Metric("metric_name", alias="metric_alias")]) }} ``` In this example, if you define an alias for `revenue` as `banana`, the query will return a column named `banana` even if `banana` isn't defined in the metric configuration. However, when using `where` Jinja clauses, you need to reference the *actual* metric name (`revenue` in this case) instead of the alias. For more a more detailed example, see [Query metric alias](#query-metric-alias). #### Querying the API for values To query values, the following parameters are available. Your query must have *either* a `metric` **or** a `group_by` parameter to be valid. | Parameter | Description | Example | | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `metrics` | The metric name as defined in your dbt metric configuration | `metrics=['revenue']` | | `group_by` | Dimension names or entities to group by. We require a reference to the entity of the dimension (other than for the primary time dimension), which is pre-appended to the front of the dimension name with a double underscore. | `group_by=['user__country', 'metric_time']` | | `grain` | A parameter specific to any time dimension and changes the grain of the data from the default for the metric. | `group_by=[Dimension('metric_time')`
`grain('week\|day\|month\|quarter\|year')]` | | `where` | A where clause that allows you to filter on dimensions and entities using parameters. This takes a filter list OR string. Inputs come with `Dimension`, and `Entity` objects. Granularity is required if the `Dimension` is a time dimension | `"{{ where=Dimension('customer__country') }} = 'US')"` | | `limit` | Limit the data returned | `limit=10` | | `order` | Order the data returned by a particular field | `order_by=['order_gross_profit']`, use `-` for descending, or full object notation if the object is operated on: `order_by=[Metric('order_gross_profit').descending(True)`] | | `compile` | If true, returns generated SQL for the data platform but does not execute | `compile=True` | | `saved_query` | A saved query you can use for frequently used queries. | `select * from {{ semantic_layer.query(saved_query="new_customer_orders"` | ##### Note on time dimensions and `metric_time` You will notice that in the list of dimensions for all metrics, there is a dimension called `metric_time`. `Metric_time` is a reserved keyword for any metric's default aggregation time dimension. For any time-series metric, the `metric_time` keyword should always be available for use in queries. This is a common dimension across *all* metrics in a semantic graph. You can look at a single metric or hundreds of metrics, and if you group by `metric_time`, it will always give you the correct time series. Additionally, when performing granularity calculations that are global (not specific to a particular time dimension), we recommend you always operate on `metric_time` and you will get the correct answer. Note that `metric_time` should be available in addition to any other time dimensions that are available for the metric(s). In the case where you are looking at one metric (or multiple metrics from the same data source), the values in the series for the primary time dimension and `metric_time` are equivalent. #### Examples The following sections provide examples of how to query metrics using the JDBC API: * [Fetch metadata for metrics](#fetch-metadata-for-metrics) — Filter/add any SQL outside of the templating syntax. * [Query common dimensions](#query-common-dimensions) — Select common dimensions for multiple metrics. * [Query grouped by time](#query-grouped-by-time) — Fetch revenue and new customers grouped by time. * [Query with a time grain](#query-with-a-time-grain) — Fetch multiple metrics with a change in time dimension granularities. * [Group by categorical dimension](#group-by-categorical-dimension) — Group by a categorical dimension. * [Query only a dimension](#query-only-a-dimension) — Get the full list of dimension values for the chosen dimension. * [Query by all dimensions](#query-by-all-dimensions) — Query by all valid dimensions. * [Query with where filters](#query-with-where-filters) — Use the `where` parameter to filter on dimensions and entities using parameters. * [Query with a limit](#query-with-a-limit) — Query using a `limit` or `order_by` clause. * [Query with order by examples](#query-with-order-by-examples) — Query with `order_by`, accepts basic string that's a Dimension, Metric, or Entity. Defaults to ascending order. Add a `-` sign in front of the object for descending order. * [Query with compile keyword](#query-with-compile-keyword) — Query using a compile keyword to preview the final SQL before execution. * [Query a saved query](#query-a-saved-query) — Query using a saved query with optional parameters like `limit` or `where`. * [Query metric alias](#query-metric-alias) — Query metrics using aliases, which allow you to use simpler or more intuitive names for metrics instead of their full definitions. * [Multi-hop joins](#multi-hop-joins) — Query across multiple related tables (multi-hop joins) using the `entity_path` argument to specify the path between related entities. ##### Fetch metadata for metrics You can filter/add any SQL outside of the templating syntax. For example, you can use the following query to fetch the name and dimensions for a metric: ```bash select name, dimensions from {{ semantic_layer.metrics() }} WHERE name='food_order_amount' ``` ##### Query common dimensions You can select common dimensions for multiple metrics. Use the following query to fetch the name and dimensions for multiple metrics: ```bash select * from {{ semantic_layer.dimensions(metrics=['food_order_amount', 'order_gross_profit']) }} ``` ##### Query grouped by time The following example query uses the [shorthand method](#faqs) to fetch revenue and new customers grouped by time: ```bash select * from {{ semantic_layer.query(metrics=['food_order_amount','order_gross_profit'], group_by=['metric_time']) }} ``` ##### Query with a time grain Use the following example query to fetch multiple metrics with a change in time dimension granularities: ```bash select * from {{ semantic_layer.query(metrics=['food_order_amount', 'order_gross_profit'], group_by=[Dimension('metric_time').grain('month')]) }} ``` ##### Group by categorical dimension Use the following query to group by a categorical dimension: ```bash select * from {{ semantic_layer.query(metrics=['food_order_amount', 'order_gross_profit'], group_by=[Dimension('metric_time').grain('month'), 'customer__customer_type']) }} ``` ##### Query only a dimension In this case, you'll get the full list of dimension values for the chosen dimension. ```bash select * from {{ semantic_layer.query(group_by=['customer__customer_type']) }} ``` ##### Query by all dimensions You can use the `semantic_layer.query_with_all_group_bys` endpoint to query by all valid dimensions. ```sql select * from {{ semantic_layer.query_with_all_group_bys(metrics =['revenue','orders','food_orders'], compile= True) }} ``` This returns all dimensions that are valid for the set of metrics in the request. ##### Query with where filters Where filters in API allow for a filter list or string. We recommend using the filter list for production applications as this format will realize all benefits from the Predicate pushdown where possible. Where Filters have a few objects that you can use: * `Dimension()` — Used for any categorical or time dimensions. `Dimension('metric_time').grain('week')` or `Dimension('customer__country')`. * `TimeDimension()` — Used as a more explicit definition for time dimensions, optionally takes in a granularity `TimeDimension('metric_time', 'month')`. * `Entity()` — Used for entities like primary and foreign keys - `Entity('order_id')`. You can use the following example to query using a `where` filter with the string format: ```bash select * from {{ semantic_layer.query(metrics=['food_order_amount', 'order_gross_profit'], group_by=[Dimension('metric_time').grain('month'),'customer__customer_type'], where="{{ Dimension('metric_time').grain('month') }} >= '2017-03-09' AND {{ Dimension('customer__customer_type' }} in ('new') AND {{ Entity('order_id') }} = 10") }} ``` * (Recommended for better performance) Use the following example to query using a `where` filter with a filter list format: ```bash select * from {{ semantic_layer.query(metrics=['food_order_amount', 'order_gross_profit'], group_by=[Dimension('metric_time').grain('month'),'customer__customer_type'], where=["{{ Dimension('metric_time').grain('month') }} >= '2017-03-09'", "{{ Dimension('customer__customer_type') }} in ('new')", "{{ Entity('order_id') }} = 10"]) }} ``` ##### Query with a limit Use the following example to query using a `limit` or `order_by` clause: ```bash select * from {{ semantic_layer.query(metrics=['food_order_amount', 'order_gross_profit'], group_by=[Dimension('metric_time')], limit=10) }} ``` ##### Query with order by examples Order By can take a basic string that's a Dimension, Metric, or Entity, and this will default to ascending order ```bash select * from {{ semantic_layer.query(metrics=['food_order_amount', 'order_gross_profit'], group_by=[Dimension('metric_time')], limit=10, order_by=['order_gross_profit']) }} ``` For descending order, you can add a `-` sign in front of the object. However, you can only use this short-hand notation if you aren't operating on the object or using the full object notation. ```bash select * from {{ semantic_layer.query(metrics=['food_order_amount', 'order_gross_profit'], group_by=[Dimension('metric_time')], limit=10, order_by=['-order_gross_profit']) }} ``` If you are ordering by an object that's been operated on (for example, you changed the granularity of the time dimension), or you are using the full object notation, descending order must look like: ```bash select * from {{ semantic_layer.query(metrics=['food_order_amount', 'order_gross_profit'], group_by=[Dimension('metric_time').grain('week')], limit=10, order_by=[Metric('order_gross_profit').descending(True), Dimension('metric_time').grain('week').descending(True) ]) }} ``` Similarly, this will yield ascending order: ```bash select * from {{ semantic_layer.query(metrics=['food_order_amount', 'order_gross_profit'], group_by=[Dimension('metric_time').grain('week')], limit=10, order_by=[Metric('order_gross_profit'), Dimension('metric_time').grain('week')]) }} ``` ##### Query with compile keyword * Use the following example to query using a `compile` keyword: ```sql select * from {{ semantic_layer.query(metrics=['food_order_amount', 'order_gross_profit'], group_by=[Dimension('metric_time').grain('month'),'customer__customer_type'], compile=True) }} ``` * Use the following example to compile SQL with a [saved query](https://docs.getdbt.com/docs/build/saved-queries.md). You can use this for frequently used queries. ```sql select * from {{ semantic_layer.query(saved_query="new_customer_orders", limit=5, compile=True}} ``` A note on querying saved queries When querying [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md),you can use parameters such as `where`, `limit`, `order`, `compile`, and so on. However, keep in mind that you can't access `metric` or `group_by` parameters in this context. This is because they are predetermined and fixed parameters for saved queries, and you can't change them at query time. If you would like to query more metrics or dimensions, you can build the query using the standard format. ##### Query a saved query Use the following example to query a [saved query](https://docs.getdbt.com/docs/build/saved-queries.md): ```sql select * from {{ semantic_layer.query(saved_query="new_customer_orders", limit=5}} ``` The JDBC API will use the saved query (`new_customer_orders`) as defined and apply a limit of 5 records. ##### Query metric alias You can query metrics using aliases, which allow you to use simpler or more intuitive names for metrics instead of their full definitions. ```sql select * from {{ semantic_layer.query(metrics=[Metric("revenue", alias="metric_alias")]) }} ``` For example, let's say your metric configuration includes an alias like `total_revenue_global` for the `order_total` metric. You can query the metric using the alias instead of the original name: ```sql select * from {{ semantic_layer.query(metrics=[Metric("order_total", alias="total_revenue_global")], group_by=['metric_time']) }} ``` The result will be: ```text | METRIC_TIME | TOTAL_REVENUE_GLOBAL | |:-------------:|:------------------: | | 2023-12-01 | 1500.75 | | 2023-12-02 | 1725.50 | | 2023-12-03 | 1850.00 | ``` tip Note that you need to use the actual metric name when using the `where` Jinja clauses. For example, if you used `banana` as an alias for `revenue`, you need to use the actual metric name, `revenue`, in the `where` clause, not `banana`. ```graphql semantic_layer.query(metrics=[Metric("revenue", alias="banana")], where="{{ Metric('revenue') }} > 0") ``` ##### Multi-hop joins In cases where you need to query across multiple related tables (multi-hop joins), use the `entity_path` argument to specify the path between related entities. The following are examples of how you can define these joins: * In this example, you're querying the `location_name` dimension but specifying that it should be joined using the `order_id` field. ```sql {{Dimension('location__location_name', entity_path=['order_id'])}} ``` * In this example, the `salesforce_account_owner` dimension is joined to the `region` field, with the path going through `salesforce_account`. ```sql {{ Dimension('salesforce_account_owner__region',['salesforce_account']) }} ``` #### FAQs I'm receiving an \`Failed ALPN\` error when trying to connect to the dbt Semantic Layer. If you're receiving a `Failed ALPN` error when trying to connect the dbt Semantic Layer with the various [data integration tools](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) (such as Tableau, DBeaver, Datagrip, ADBC, or JDBC), it typically happens when connecting from a computer behind a corporate VPN or Proxy (like Zscaler or Check Point). The root cause is typically the proxy interfering with the TLS handshake as the Semantic Layer uses gRPC/HTTP2 for connectivity. To resolve this: * If your proxy supports gRPC/HTTP2 but isn't configured to allow ALPN, adjust its settings accordingly to allow ALPN. Or create an exception for the dbt domain. * If your proxy does not support gRPC/HTTP2, add an SSL interception exception for the dbt domain in your proxy settings This should help in successfully establishing the connection without the Failed ALPN error.  Why do some dimensions use different syntax, like \`metric\_time\` versus \`Dimension('metric\_time')\`? When you select a dimension on its own, such as `metric_time` you can use the shorthand method which doesn't need the “Dimension” syntax. However, when you perform operations on the dimension, such as adding granularity, the object syntax `[Dimension('metric_time')` is required.  What does the double underscore \`'\_\_'\` syntax in dimensions mean? The double underscore `"__"` syntax indicates a mapping from an entity to a dimension, as well as where the dimension is located. For example, `user__country` means someone is looking at the `country` dimension from the `user` table.  What is the default output when adding granularity? The default output follows the format `{{time_dimension_name}__{granularity_level}}`. So for example, if the `time_dimension_name` is `ds` and the granularity level is yearly, the output is `ds__year`. #### Related docs * [Semantic Layer integration best practices](https://docs.getdbt.com/guides/sl-partner-integration-guide.md) --- ### Job object schema dbt platform | Starter, Enterprise, Enterprise+ The job object allows you to query information about a particular model based on `jobId` and, optionally, a `runId`. If you don't provide a `runId`, the API returns information on the latest runId of a job. The [example query](#example-query) illustrates a few fields you can query in this `job` object. Refer to [Fields](#fields) to see the entire schema, which provides all possible fields you can query. ##### Arguments When querying for `job`, you can use the following arguments. ### Fetching data... ##### Example Query You can use your production job's `id`. ```graphql query JobQueryExample { # Provide runId for looking at specific run, otherwise it defaults to latest run job(id: 940) { # Get all models from this job's latest run models(schema: "analytics", first: 10, after: "{somePaginationCursorValue}") { uniqueId executionTime paginationCursor } # Or query a single node source(uniqueId: "source.jaffle_shop.snowplow.event") { uniqueId sourceName name state maxLoadedAt criteria { warnAfter { period count } errorAfter { period count } } maxLoadedAtTimeAgoInS } } } ``` ##### Fields When querying an `job`, you can use the following fields. ### Fetching data... --- ### Lineage object schema dbt platform | Starter, Enterprise, Enterprise+ The lineage object allows you to query lineage across your resources. The [Example query](#example-query) illustrates a few fields you can query with the `lineage` object. Refer to [Fields](#fields) to view the entire schema, which provides all possible fields you can query. ##### Arguments When querying for `lineage`, you can use the following arguments: ### Fetching data... ##### Example query You can specify the `environmentId` and filter by "Model" as the resource type to see lineage information for all models in this environment, including their dependencies, materialization type, and metadata: ```graphql query { environment(id: 834) { applied { lineage( filter: {"types": ["Model"]} # Return results for the Model type ) { name resourceType filePath projectId materializationType parentIds tags uniqueId } } } } ``` ##### Fields When querying for `lineage`, you can use the following fields: ### Fetching data... --- ### Model Historical Runs object schema dbt platform | Starter, Enterprise, Enterprise+ The model historical runs object allows you to query information about a model's run history. The [Example query](#example-query) illustrates a few fields you can query with the `modelHistoricalRuns` object. Refer to [Fields](#fields) to view the entire schema, which provides all possible fields you can query. ##### Arguments When querying for `modelHistoricalRuns`, you can use the following arguments: ### Fetching data... ##### Example query You can specify the `environmentId` and the model's `uniqueId` to return the model and its execution history for the `customers` model in the `marketing` package, including performance metrics and test results for the last 20 times it was run, regardless of which job ran it. ```graphql query { environment(id: 834) { applied { modelHistoricalRuns( uniqueId: "model.marketing.customers" # Use this format for unique ID: RESOURCE_TYPE.PACKAGE_NAME.RESOURCE_NAME lastRunCount: 20 ) { runId # Get historical results for a particular model runGeneratedAt executionTime # View build time across runs status tests { name status executeCompletedAt } # View test results across runs } } } } ``` ##### Fields When querying for `modelHistoricalRuns`, you can use the following fields: ### Fetching data... --- ### Model object schema dbt platform | Starter, Enterprise, Enterprise+ The model object allows you to query information about a particular model in a given job. ##### Arguments When querying for a `model`, the following arguments are available. ### Fetching data... Below we show some illustrative example queries and outline the schema (all possible fields you can query) of the model object. ##### Example query for finding parent models and sources The example query below uses the `parentsModels` and `parentsSources` fields to fetch information about a model’s parent models and parent sources. The jobID and uniqueID fields are placeholders that you will need to replace with your own values. ```graphql { job(id: 123) { model(uniqueId: "model.jaffle_shop.dim_user") { parentsModels { runId uniqueId executionTime } parentsSources { runId uniqueId state } } } } ``` ##### Example query for model timing The example query below could be useful if you want to understand information around execution timing on a given model (start, end, completion). ```graphql { job(id: 123) { model(uniqueId: "model.jaffle_shop.dim_user") { runId projectId name uniqueId resourceType executeStartedAt executeCompletedAt executionTime } } } ``` ##### Example query for column-level information You can use the following example query to understand more about the columns of a given model. This query will only work if the job has generated documentation; that is, it will work with the command `dbt docs generate`. ```graphql { job(id: 123) { model(uniqueId: "model.jaffle_shop.dim_user") { columns { name index type comment description tags meta } } } } ``` ##### Fields When querying for a `model`, the following fields are available: ### Fetching data... --- ### Models object schema dbt platform | Starter, Enterprise, Enterprise+ [Models](https://docs.getdbt.com/docs/build/models.md) are the foundational dbt resource that transform raw data into curated datasets using SQL (or Python). Each model represents a single SELECT statement, typically materialized as a table or view in your warehouse. You can query information about models through the Discovery API. The [Example query](#example-query) illustrates a few fields you can query with the `models` object. Refer to [Fields](#fields) to view the entire schema, which provides all possible fields you can query. ##### Arguments When querying for `models`, you can use the following arguments: ### Fetching data... ##### Example query You can specify the `environmentId` to return model information for all models in the given environment, including their metadata, configuration, tests, and ownership details, limited to the first 100 results: ```graphql query { environment(id: 834) { applied { models (first: 100) { edges { node { name description access accountId catalog { owner } config environmentId tests { name description } } } } } } } ``` ##### Fields When querying for `models`, you can use the following fields: ### Fetching data... ##### Key fields from nodes ### Fetching data... --- ### Models object schema dbt platform | Starter, Enterprise, Enterprise+ The models object allows you to query information about all models in a given job. ##### Arguments When querying for `models`, the following arguments are available. ### Fetching data... Below we show some illustrative example queries and outline the schema of the models object. ##### Example queries The database, schema, and identifier arguments are all optional. This means that with this endpoint you can: * Find a specific model by providing `..` * Find all of the models in a database and/or schema by providing `` and/or `` ###### Find models by their database, schema, and identifier The example query below finds a model by its unique database, schema, and identifier. ```graphql { job(id: 123) { models( database: "analytics" schema: "analytics" identifier: "dim_customers" first: 10 after: "{somePaginationCursorValue}" ) { uniqueId paginationCursor } } } ``` ###### Find models by their schema The example query below finds all models in this schema and their respective execution times. ```graphql { job(id: 123) { models(schema: "analytics", first: 10, after: "{somePaginationCursorValue}") { uniqueId executionTime paginationCursor } } } ``` ###### Paginate models in a job run Job-based `models` queries support `first` and `after` for cursor pagination. Each model returns a `paginationCursor` field to fetch the next page. Refer to [Job-based queries](https://docs.getdbt.com/docs/dbt-apis/discovery-querying.md#job-based-queries) for paging behavior, supported resource types, and legacy default behavior. ```graphql query JobModelsPage($jobId: BigInt!, $runId: BigInt, $first: Int!, $after: String) { job(id: $jobId, runId: $runId) { models(first: $first, after: $after) { uniqueId executionTime paginationCursor } } } ``` ##### Fields The models object can access the *same fields* as the [Model node](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-job-model.md). The difference is that the models object can output a list so instead of querying for fields for one specific model, you can query for those parameters for all models within a jobID, database, and so on. When querying for `models`, the following fields are available: ### Fetching data... --- ### Owners object schema dbt platform | Starter, Enterprise, Enterprise+ [Owners](https://docs.getdbt.com/docs/build/groups.md) help you identify the user or domain responsible for a dbt asset. For most assets, owners are defined in your project code using groups. Exposures are an exception: for downstream exposures that represent BI assets, owners are automatically pulled from the downstream tool based on who owns that asset. You can query ownership information through the Discovery API. The [Example query](#example-query) illustrates a few fields you can query with the `owners` object. Refer to [Fields](#fields) to view the entire schema, which provides all possible fields you can query. ##### Arguments When querying for `owners`, you can use the following arguments: ### Fetching data... ##### Example query You can specify the `environmentId` and "exposure" as the `OwnerResourceType` to return people who own exposures (downstream BI assets) in this environment, including their contact information. ```graphql query { environment(id: 834) { applied { owners(resource: exposure) { email name } } } } ``` ##### Fields When querying for `owners`, you can use the following fields: ### Fetching data... --- ### Packages object schema dbt platform | Starter, Enterprise, Enterprise+ [dbt packages](https://docs.getdbt.com/docs/build/packages.md) are libraries with models, macros, and other resources that tackle a specific problem area utilized by dbt projects. You can query project packages through the Discovery API. The [Example query](#example-query) illustrates a few fields you can query with the `packages` object. Refer to [Fields](#fields) to view the entire schema, which provides all possible fields you can query. ##### Arguments When querying for `packages`, you can use the following arguments: ### Fetching data... ##### Example query You can specify the `environmentId` and "model" as the resource to see all dbt packages in this environment that contain model resources: ```graphql query { environment(id: 834) { applied { packages(resource: "model") } } } ``` --- ### Project state in dbt dbt provides a stateful way of deploying dbt. Artifacts are accessible programmatically via the [Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-querying.md) in the metadata platform. With the implementation of the `environment` endpoint in the Discovery API, we've introduced the idea of multiple states. The Discovery API provides a single API endpoint that returns the latest state of models, sources, and other nodes in the DAG. A single [deployment environment](https://docs.getdbt.com/docs/environments-in-dbt.md) should represent the production state of a given dbt project. There are two states that can be queried in dbt: * **Applied state** refers to what exists in the data warehouse after a successful `dbt run`. The model build succeeds and now exists as a table in the warehouse. * **Definition state** depends on what exists in the project given the code defined in it (for example, manifest state), which hasn’t necessarily been executed in the data platform (maybe just the result of `dbt compile`). #### Definition (logical) vs. applied state of dbt nodes In a dbt project, the state of a node *definition* represents the configuration, transformations, and dependencies defined in the SQL and YAML files. It captures how the node should be processed in relation to other nodes and tables in the data warehouse and may be produced by a `dbt build`, `run`, `parse`, or `compile`. It changes whenever the project code changes. A node’s *applied state* refers to the node’s actual state after it has been successfully executed in the DAG; for example, models are executed; thus, their state is applied to the data warehouse via `dbt run` or `dbt build`. It changes whenever a node is executed. This state represents the result of the transformations and the actual data stored in the database, which for models can be a table or a view based on the defined logic. The applied state includes execution info, which contains metadata about how the node arrived in the applied state. The fields within `executionInfo` track two related but distinct concepts: | Concept | Description | Fields | | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------- | | Most recent run attempt | The latest run regardless of outcome (success, error, or skip) | `lastRunId`, `lastRunStatus`, `lastRunError`, `lastRunGeneratedAt`, `lastJobDefinitionId` | | Most recent successful materialization | The last run in which the node was built in the data warehouse.
When a run errors out, the node isn't rebuilt, so these fields remain pinned to the prior successful run. | `executeStartedAt`, `executeCompletedAt`, `executionTime`, `runGeneratedAt`, `lastSuccessRunId`, `lastSuccessJobDefinitionId` | For example, if a model's most recent run errors out, `lastRunStatus` will be `error` and `lastRunGeneratedAt` will reference that failed run, while `executeCompletedAt` and `lastSuccessRunId` will still reference the prior run in which the model was successfully materialized. Here’s how you can query and compare the definition vs. applied state of a model using the Discovery API: ```graphql query Compare($environmentId: Int!, $first: Int!) { environment(id: $environmentId) { definition { models(first: $first) { edges { node { name rawCode } } } } applied { models(first: $first) { edges { node { name rawCode executionInfo { executeCompletedAt } } } } } } } ``` Most Discovery API use cases will favor the *applied state* since it pertains to what has actually been run and can be analyzed. #### Affected states by node type The following table shows the states of dbt nodes and how they are affected by the Discovery API. | Node | Executed in DAG | Created by execution | Exists in database | Lineage | States | | -------------------------------------------------------------------------------------- | --------------- | -------------------- | ------------------ | --------------------- | -------------------- | | [Analysis](https://docs.getdbt.com/docs/build/analyses.md) | No | No | No | Upstream | Definition | | [Data test](https://docs.getdbt.com/docs/build/data-tests.md) | Yes | Yes | No | Upstream | Applied & definition | | [Exposure](https://docs.getdbt.com/docs/build/exposures.md) | No | No | No | Upstream | Definition | | [Group](https://docs.getdbt.com/docs/build/groups.md) | No | No | No | Downstream | Definition | | [Macro](https://docs.getdbt.com/docs/build/jinja-macros.md) | Yes | No | No | N/A | Definition | | [Metric](https://docs.getdbt.com/docs/build/metrics-overview.md) | No | No | No | Upstream & downstream | Definition | | [Model](https://docs.getdbt.com/docs/build/models.md) | Yes | Yes | Yes | Upstream & downstream | Applied & definition | | [Saved queries](https://docs.getdbt.com/docs/build/saved-queries.md)
(not in API) | N/A | N/A | N/A | N/A | N/A | | [Seed](https://docs.getdbt.com/docs/build/seeds.md) | Yes | Yes | Yes | Downstream | Applied & definition | | [Semantic model](https://docs.getdbt.com/docs/build/semantic-models.md) | No | No | No | Upstream & downstream | Definition | | [Snapshot](https://docs.getdbt.com/docs/build/snapshots.md) | Yes | Yes | Yes | Upstream & downstream | Applied & definition | | [Source](https://docs.getdbt.com/docs/build/sources.md) | Yes | No | Yes | Downstream | Applied & definition | | [Unit tests](https://docs.getdbt.com/docs/build/unit-tests.md) | Yes | Yes | No | Downstream | Definition | #### Caveats about state/metadata updates Over time, Cloud Artifacts will provide information to maintain state for features/services in dbt and enable you to access state in dbt and its downstream ecosystem. Cloud Artifacts is currently focused on the latest production state, but this focus will evolve. Here are some limitations of the state representation in the Discovery API: * Users must access the default production environment to know the latest state of a project. * The API gets the definition from the latest manifest generated in a given deployment environment, but that often won’t reflect the latest project code state. * Compiled code results may be outdated depending on dbt run step order and failures. * Catalog info can be outdated, or incomplete (in the applied state), based on if/when `docs generate` was last run. * Source freshness checks can be out of date (in the applied state) depending on when the command was last run, and it’s not included in `build`. --- ### Python SDK dbt platform | Starter, Enterprise, Enterprise+ The [`dbt-sl-sdk` Python software development kit](https://github.com/dbt-labs/semantic-layer-sdk-python) (SDK) is a Python library that provides you with easy access to the dbt Semantic Layer with Python. It allows developers to interact with the dbt Semantic Layer APIs and query metrics and dimensions in downstream tools. #### Installation To install the Python SDK, you'll need to specify optional dependencies depending on whether you want to use it synchronously, backed by [requests](https://github.com/psf/requests/), or with asynchronous ([asyncio](https://docs.python.org/3/library/asyncio.html) backed by [aiohttp](https://github.com/aio-libs/aiohttp/)). The Python SDK supports the Long-Term Support (LTS) versions of Python, such as 3.9, 3.10, 3.11, and 3.12. When Python discontinues support for a version, the Python SDK will also discontinue support for that version. If you’re using a non-supported version, you may experience compatibility issues and won’t receive updates or security patches from the SDK. ##### Sync installation Sync installation means your program waits for each task to finish before moving on to the next one. It's simpler, easier to understand, and suitable for smaller tasks or when your program doesn't need to handle many tasks at the same time. ```bash pip install "dbt-sl-sdk[sync]" ``` If you're using async frameworks like [FastAPI](https://fastapi.tiangolo.com/) or [Strawberry](https://github.com/strawberry-graphql/strawberry), installing the sync version of the SDK will block your event loop and can significantly slow down your program. In this case, we strongly recommend using async installation. ##### Async installation Async installation means your program can start a task and then move on to other tasks while waiting for the first one to finish. This can handle many tasks at once without waiting, making it faster and more efficient for larger tasks or when you need to manage multiple tasks at the same time. For more details, refer to [asyncio](https://docs.python.org/3/library/asyncio.html). ```bash pip install "dbt-sl-sdk[async]" ``` Since the [Python ADBC driver](https://github.com/apache/arrow-adbc/tree/main/python/adbc_driver_manager) doesn't yet support asyncio natively, `dbt-sl-sdk` uses a [`ThreadPoolExecutor`](https://github.com/dbt-labs/semantic-layer-sdk-python/blob/5e52e1ca840d20a143b226ae33d194a4a9bc008f/dbtsl/api/adbc/client/asyncio.py#L62) to run `query` and `list dimension-values` (all operations that are done with ADBC). This is why you might see multiple Python threads spawning. If you're using async frameworks like [FastAPI](https://fastapi.tiangolo.com/) or [Strawberry](https://github.com/strawberry-graphql/strawberry), installing the sync version of the Python SDK will block your event loop and can significantly slow down your program. In this case, we strongly recommend using async installation. #### Usage To run operations against the Semantic Layer APIs, instantiate (create an instance of) a `SemanticLayerClient` with your specific [API connection parameters](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md): ```python from dbtsl import SemanticLayerClient client = SemanticLayerClient( environment_id=123, auth_token="", host="semantic-layer.cloud.getdbt.com", ) # query the first metric by `metric_time` def main(): with client.session(): metrics = client.metrics() table = client.query( metrics=[metrics[0].name], group_by=["metric_time"], ) print(table) main() ``` **Note**: All method calls that reach out to the APIs need to be within a `client.session()` context manager. This allows the client to establish a connection to the APIs only once and reuse the same connection between API calls. We recommend creating an application-wide session and reusing the same session throughout the application for optimal performance. Creating a session per request is discouraged and inefficient. ##### asyncio usage If you're using asyncio, import `AsyncSemanticLayerClient` from `dbtsl.asyncio`. The `SemanticLayerClient` and `AsyncSemanticLayerClient` APIs are identical, but the async version has async methods that you need to `await`. ```python import asyncio from dbtsl.asyncio import AsyncSemanticLayerClient client = AsyncSemanticLayerClient( environment_id=123, auth_token="", host="semantic-layer.cloud.getdbt.com", ) async def main(): async with client.session(): metrics = await client.metrics() table = await client.query( metrics=[metrics[0].name], group_by=["metric_time"], ) print(table) asyncio.run(main()) ``` ##### Lazy loading for large fields By default, the Python SDK eagerly loads nested lists of objects such as `dimensions`, `entities`, and `measures` for each `Metric` — even if you don't need them. This is generally convenient, but in large projects, it can lead to slower responses due to the amount of data returned. To improve performance, you can opt into lazy loading by passing `lazy=True` when creating the client. With lazy loading enabled, the SDK skips fetching large nested fields until you explicitly request them on a per-model basis. (Applies to dbt v1.12 and later) Lazy loading is currently only supported for `dimensions` and `entities` on `Metric` objects. For example, the following code fetches all available metrics from the metadata API and displays only the dimensions of certain metrics: list\_metrics\_lazy\_sync.py ```python """Fetch all available metrics from the metadata API and display only the dimensions of certain metrics.""" from argparse import ArgumentParser from dbtsl import SemanticLayerClient def get_arg_parser() -> ArgumentParser: p = ArgumentParser() p.add_argument("--env-id", required=True, help="The dbt environment ID", type=int) p.add_argument("--token", required=True, help="The API auth token") p.add_argument("--host", required=True, help="The API host") return p def main() -> None: arg_parser = get_arg_parser() args = arg_parser.parse_args() client = SemanticLayerClient( environment_id=args.env_id, auth_token=args.token, host=args.host, lazy=True, ) with client.session(): metrics = client.metrics() for i, m in enumerate(metrics): print(f"📈 {m.name}") print(f" type={m.type}") print(f" description={m.description}") assert len(m.dimensions) == 0 # skip if index is odd if i & 1: print(" dimensions=skipped") continue # load dimensions only if index is even m.load_dimensions() print(" dimensions=[") for dim in m.dimensions: print(f" {dim.name},") print(" ]") if __name__ == "__main__": main() ``` Refer to the [lazy loading example](https://github.com/dbt-labs/semantic-layer-sdk-python/blob/main/examples/list_metrics_lazy_sync.py) for more details. #### Integrate with dataframe libraries The Python SDK returns all query data as [pyarrow](https://arrow.apache.org/docs/python/index.html) tables. The Python SDK library doesn't come bundled with [Polars](https://pola.rs/) or [Pandas](https://pandas.pydata.org/). If you use these libraries, add them as dependencies in your project. To use the data with libraries like Polars or Pandas, manually convert the data into the desired format. For example: ###### If you're using pandas ```python # ... initialize client arrow_table = client.query(...) pandas_df = arrow_table.to_pandas() ``` ###### If you're using polars ```python import polars as pl # ... initialize client arrow_table = client.query(...) polars_df = pl.from_arrow(arrow_table) ``` #### Usage examples For additional usage examples, check out the [usage examples](https://github.com/dbt-labs/semantic-layer-sdk-python/tree/main/examples), some of which include: * [Fetching dimension values sync](https://github.com/dbt-labs/semantic-layer-sdk-python/blob/main/examples/fetch_dimension_values_sync.py) * Fetching metrics [async](https://github.com/dbt-labs/semantic-layer-sdk-python/blob/main/examples/fetch_metric_async.py) and [sync](https://github.com/dbt-labs/semantic-layer-sdk-python/blob/main/examples/fetch_metric_sync.py) * [List saved queries async](https://github.com/dbt-labs/semantic-layer-sdk-python/blob/main/examples/list_saved_queries_async.py) #### Disable telemetry By default, the Python SDK sends some [platform-related information](https://github.com/dbt-labs/semantic-layer-sdk-python/blob/main/dbtsl/env.py) to dbt Labs. To opt-out, set the `PLATFORM.anonymous` attribute to `True`: ```python from dbtsl.env import PLATFORM PLATFORM.anonymous = True # ... initialize client ``` #### Contribute To contribute to this project, check out our [contribution guidelines](https://github.com/dbt-labs/semantic-layer-sdk-python/blob/main/CONTRIBUTING.md) and open a GitHub [issue](https://github.com/dbt-labs/semantic-layer-sdk-python/issues) or [pull request](https://github.com/dbt-labs/semantic-layer-sdk-python/pulls). --- ### Query the Discovery API dbt platform | Starter, Enterprise, Enterprise+ The Discovery API supports ad-hoc queries and integrations. If you are new to the API, refer to [About the Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md) for an introduction. For request quotas and throttling behavior, refer to [API rate limits](https://docs.getdbt.com/docs/dbt-apis/rate-limits.md). Use the Discovery API to evaluate data pipeline health and project state across runs or at a moment in time. dbt Labs provide a default [GraphQL explorer](https://metadata.cloud.getdbt.com/graphql) for this API, enabling you to run queries and browse the schema. However, you can also use any GraphQL client of your choice to query the API. Since GraphQL describes the data in the API, the schema displayed in the GraphQL explorer accurately represents the graph and fields available to query. #### Prerequisites * You must have a dbt [multi-tenant](https://docs.getdbt.com/docs/platform/about-platform/tenancy.md#multi-tenant) or [single tenant](https://docs.getdbt.com/docs/platform/about-platform/tenancy.md#single-tenant) account. * You must be on a [Starter, Enterprise, or Enterprise+ plan](https://www.getdbt.com/pricing/). * Your projects must be on a dbt [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) or dbt version 1.0 or later. Refer to [Upgrade dbt version in Cloud](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) to upgrade. #### Authorization Currently, authorization of requests takes place [using a service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md). dbt admin users can generate a Metadata Only service token that is authorized to execute a specific query against the Discovery API. Once you've created a token, you can use it in the Authorization header of requests to the dbt Discovery API. Be sure to include the Token prefix in the Authorization header, or the request will fail with a `401 Unauthorized` error. Note that `Bearer` can be used instead of `Token` in the Authorization header. Both syntaxes are equivalent. #### Access the Discovery API 1. Create a [service account token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) to authorize requests. dbt Admin users can generate a *Metadata Only* service token, which can be used to execute a specific query against the Discovery API to authorize requests. 2. Find the API URL to use from the [Discovery API endpoints](#discovery-api-endpoints) table. 3. For specific query points, refer to the [schema documentation](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-job.md). #### Run queries using HTTP requests You can run queries by sending a `POST` request to the Discovery API, making sure to replace: * `YOUR_API_URL` with the appropriate [Discovery API endpoint](#discovery-api-endpoints) for your region and plan. * `YOUR_TOKEN` in the Authorization header with your actual API token. Be sure to include the Token prefix. * `QUERY_BODY` with a GraphQL query, for example `{ "query": "", "variables": "" }` * `VARIABLES` with a dictionary of your GraphQL query variables, such as a job ID or a filter. * `ENDPOINT` with the endpoint you're querying, such as environment. ```shell curl 'YOUR_API_URL' \ -H 'authorization: Bearer YOUR_TOKEN' \ -H 'content-type: application/json' -X POST --data QUERY_BODY ``` Python example: ```python response = requests.post( 'YOUR_API_URL', headers={"authorization": "Bearer "+YOUR_TOKEN, "content-type": "application/json"}, json={"query": QUERY_BODY, "variables": VARIABLES} ) metadata = response.json()['data'][ENDPOINT] ``` Every query will require an environment ID or job ID. You can get the ID from a dbt URL or using the Admin API. There are several illustrative example queries on this page. For more examples, refer to [Use cases and examples for the Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-use-cases-and-examples.md). #### Discovery API endpoints Find your Discovery API endpoint in **Account settings** under **Access URLs** in dbt platform. The format depends on your deployment type and region: | Deployment type | Discovery API URL | | --------------- | ------------------------------------------------------------------- | | Multi-tenant | `https://ACCOUNT_PREFIX.metadata.REGION.dbt.com/graphql` | | Single-tenant | `https://ACCOUNT_PREFIX.metadata.ROUTING_SUBDOMAIN.dbt.com/graphql` | * Replace the following placeholders with your actual values. Refer to [Access, Regions, & IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for more information on the regions and subdomains: * `ACCOUNT_PREFIX` with your account identifier (found in **Account settings** under **Access URLs**) * `REGION` with your deployment region (for example, `us1` for North America AWS, `eu1` for EMEA, `jp1` for Japan, `au1` for APAC, and so on) * `ROUTING_SUBDOMAIN` with your single-tenant routing subdomain (typically your company name. Please contact your account team if unsure) * For example, if you're on North America AWS with account prefix `abc123`, your Discovery API URL is `https://abc123.metadata.us1.dbt.com/graphql`. #### Reasonable use Discovery (GraphQL) API usage is subject to request rate and response size limits to maintain the performance and stability of the metadata platform and prevent abuse. Job-level endpoints are subject to query complexity limits. Nested nodes (like parents), code (like rawCode), and catalog columns are considered as most complex. Overly complex queries should be broken up into separate queries with only necessary fields included. dbt Labs recommends using the environment endpoint instead for most use cases to get the latest descriptive and result metadata for a dbt project. #### Retention limits You can use the Discovery API to query data from the previous two months. For example, if today was April 1st, you could query data back to February 1st. #### Run queries with the GraphQL explorer You can run ad-hoc queries directly in the [GraphQL API explorer](https://metadata.cloud.getdbt.com/graphql) and use the document explorer on the left-hand side to see all possible nodes and fields. Refer to the [Apollo explorer documentation](https://www.apollographql.com/docs/graphos/explorer/explorer) for setup and authorization information for GraphQL. 1. Access the [GraphQL API explorer](https://metadata.cloud.getdbt.com/graphql) and select fields you want to query. 2. Select **Variables** at the bottom of the explorer and replace any `null` fields with your unique values. 3. [Authenticate](https://www.apollographql.com/docs/graphos/explorer/connecting-authenticating#authentication) using Bearer auth with `YOUR_TOKEN`. Select **Headers** at the bottom of the explorer and select **+New header**. 4. Select **Authorization** in the **header key** dropdown list and enter your Bearer auth token in the **value** field. Remember to include the Token prefix. Your header key should be in this format: `{"Authorization": "Bearer }`.
[![Enter the header key and Bearer auth token values](/img/docs/dbt-platform/discovery-api/graphql_header.jpg?v=2 "Enter the header key and Bearer auth token values")](#)Enter the header key and Bearer auth token values 1. Run your query by clicking the blue query button in the top right of the **Operation** editor (to the right of the query). You should see a successful query response on the right side of the explorer. [![Run queries using the Apollo Server GraphQL explorer](/img/docs/dbt-platform/discovery-api/graphql.jpg?v=2 "Run queries using the Apollo Server GraphQL explorer")](#)Run queries using the Apollo Server GraphQL explorer ##### Fragments Use the [`... on`](https://www.apollographql.com/docs/react/data/fragments/) notation to query across lineage and retrieve results from specific node types. ```graphql query ($environmentId: BigInt!, $first: Int!) { environment(id: $environmentId) { applied { models(first: $first, filter: { uniqueIds: "MODEL.PROJECT.MODEL_NAME" }) { edges { node { name ancestors(types: [Model, Source, Seed, Snapshot]) { ... on ModelAppliedStateNestedNode { name resourceType materializedType executionInfo { executeCompletedAt } } ... on SourceAppliedStateNestedNode { sourceName name resourceType freshness { maxLoadedAt } } ... on SnapshotAppliedStateNestedNode { name resourceType executionInfo { executeCompletedAt } } ... on SeedAppliedStateNestedNode { name resourceType executionInfo { executeCompletedAt } } } } } } } } } ``` ##### Pagination Querying large datasets can impact performance on multiple functions in the API pipeline. Pagination eases the burden by returning smaller data sets one page at a time. This is useful for returning a particular portion of the dataset or the entire dataset piece-by-piece to enhance performance. dbt utilizes cursor-based pagination, which makes it easy to return pages of constantly changing data. Use the `PageInfo` object to return information about the page. The available fields are: * `startCursor` string type — Corresponds to the first `node` in the `edge`. * `endCursor` string type — Corresponds to the last `node` in the `edge`. * `hasNextPage` boolean type — Whether or not there are more `nodes` after the returned results. There are connection variables available when making the query: * `first` integer type — Returns the first n `nodes` for each page, up to 500. * `after` string type — Sets the cursor to retrieve `nodes` after. It's best practice to set the `after` variable with the object ID defined in the `endCursor` of the previous page. Below is an example that returns the `first` 500 models `after` the specified Object ID in the variables. The `PageInfo` object returns where the object ID where the cursor starts, where it ends, and whether there is a next page. [![Example of pagination](/img/Paginate.png?v=2 "Example of pagination")](#)Example of pagination Below is a code example of the `PageInfo` object: ```graphql pageInfo { startCursor endCursor hasNextPage } totalCount # Total number of records across all pages ``` The previously described `PageInfo` and `totalCount` pattern applies to the `environment` endpoints, which return results as a connection of `edges` and `nodes`. ###### Job-based endpoint pagination Job-based list endpoints also support cursor-based pagination. Use this pattern when you query resources under the [`job`](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-job.md) object. These endpoints return a flat list rather than an `edges`/`nodes` connection, so they do not expose a `PageInfo` object. The following job-based list fields accept `first` and `after`: * `models` * `sources` * `seeds` * `snapshots` * `tests` * `macros` * `metrics` * `exposures` Pass `first` to set the page size (capped at 100). Each returned item includes a `paginationCursor` field—an opaque, per-element value you select in your query. Pass the last item's `paginationCursor` as `after` on the next request to fetch the following page. Unlike environment queries, these job-based endpoints do not return a `PageInfo` object or `hasNextPage` field. You have reached the last page when a page returns fewer rows than `first` (or fewer than the default page size). We recommend always specifying `first` to keep response sizes manageable. Historically, omitting `first` and `after` would cause the API to return all matching rows. For large jobs, best practice is to pass `first` and `after` explicitly so your integration does not depend on unpaginated responses. Upcoming change to unpaginated requests In a future update, requests that omit `first` and `after` will return at most 100 rows. dbt Labs will provide advance notice before that change rolls out. To run the example, use your [Discovery API endpoint](#discovery-api-endpoints) and a [Metadata Only service token](#authorization). Get `jobId` and optional `runId` from the job or run URL, or with the [Admin API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md). If you omit `runId`, the API uses the job's latest run. Run the query in the [GraphQL explorer](#run-queries-with-the-graphql-explorer) or via [HTTP requests](#run-queries-using-http-requests). The example below uses a job's `models` list. The same `first` and `after` arguments work for the other resource types listed above. For a use-case example with more fields, refer to [Use cases and examples for the Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-use-cases-and-examples.md). ```graphql query JobModelsPage($jobId: BigInt!, $runId: BigInt, $first: Int!, $after: String) { job(id: $jobId, runId: $runId) { models(first: $first, after: $after) { uniqueId paginationCursor } } } ``` First page variables: ```json { "jobId": 12345, "runId": 67890, "first": 10, "after": null } ``` For the next page, set `after` to the `paginationCursor` from the *last* row of the previous page. The cursor is an opaque encoded string, not the `uniqueId`. Repeat until a page returns fewer rows than `first` (or fewer than the default page size). ```json { "jobId": 12345, "runId": 67890, "first": 10, "after": "Y3Vyc29yOm1vZGVsLm15X3Byb2plY3QuZGltX2N1c3RvbWVycw==" } ``` ##### Filters Filtering helps to narrow down the results of an API query. If you want to query and return only models and tests that are failing or find models that are taking too long to run, you can fetch execution details such as [`executionTime`](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-job-models.md#fields), [`runElapsedTime`](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-job-models.md#fields), or [`status`](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-job-models.md#fields). This helps data teams monitor the performance of their models, identify bottlenecks, and optimize the overall data pipeline. Below is an example that filters for results of models that have succeeded on their `lastRunStatus`: [![Example of filtering](/img/Filtering.png?v=2 "Example of filtering")](#)Example of filtering Below is an example that filters for models that have an error on their last run and tests that have failed: ```graphql query ModelsAndTests($environmentId: BigInt!, $first: Int!) { environment(id: $environmentId) { applied { models(first: $first, filter: { lastRunStatus: error }) { edges { node { name executionInfo { lastRunId } } } } tests(first: $first, filter: { status: "fail" }) { edges { node { name executionInfo { lastRunId } } } } } } } ``` #### Related content * [Use cases and examples for the Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-use-cases-and-examples.md) * [Schema](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-job.md) --- ### Resources object schema dbt platform | Starter, Enterprise, Enterprise+ The resources object allows you to paginate across all resources in your environment. The [Example query](#example-query) illustrates a few fields you can query with the `resources` object. Refer to [Fields](#fields) to view the entire schema, which provides all possible fields you can query. ##### Arguments When querying for `resources`, you can use the following arguments: ### Fetching data... ##### Example query You can specify the `environmentId`, filter by "Model" as the type, and limit to the first 100 results to see comprehensive information about the first 100 model resources in this environment, including their metadata, tags, and file locations: ```graphql query { environment(id: 834) { applied { resources( filter: { types: [ Model ] }, first: 100 ) { edges { node { accountId description environmentId filePath meta name projectId resourceType uniqueId tags } } } } } } ``` ##### Fields When querying for `resources`, you can use the following fields: ### Fetching data... ##### Key fields from nodes ### Fetching data... --- ### Seed object schema dbt platform | Starter, Enterprise, Enterprise+ The seed object allows you to query information about a particular seed in a given job. ##### Arguments When querying for a `seed`, the following arguments are available. ### Fetching data... Below we show some illustrative example queries and outline the schema of the seed object. ##### Example query The example query below pulls relevant information about a given seed. For instance, you can view the load time. ```graphql { job(id: 123) { seed(uniqueId: "seed.jaffle_shop.raw_customers") { database schema uniqueId name status error } } } ``` ##### Fields When querying for a `seed`, the following fields are available: ### Fetching data... --- ### Seeds object schema dbt platform | Starter, Enterprise, Enterprise+ [Seeds](https://docs.getdbt.com/docs/build/seeds.md) are CSV files in your dbt project that dbt can load into your data warehouse. You can query seeds through the Discovery API. The [Example query](#example-query) illustrates a few fields you can query with the `seeds` object. Refer to [Fields](#fields) to view the entire schema, which provides all possible fields you can query. ##### Arguments When querying for `seeds`, you can use the following arguments: ### Fetching data... ##### Example query You can specify the `environmentId`, filter by the database, and limit to the first 100 to show information about the first 100 seed files in the `analytics` database, including their metadata and file locations: ```graphql query ($environmentId: BigInt!, $first: Int!, $filter: GenericMaterializedFilter) { environment(id: $environmentId) { applied { seeds( first: 100, filter: { database: "analytics" } ) { edges { node { description name filePath projectId fqn tags uniqueId resourceType } } } } } } ``` ##### Fields When querying for `seeds`, you can use the following fields: ### Fetching data... ##### Key fields from nodes ### Fetching data... --- ### Seeds object schema dbt platform | Starter, Enterprise, Enterprise+ The seeds object allows you to query information about all seeds in a given job. ##### Arguments When querying for `seeds`, the following arguments are available. ### Fetching data... Below we show some illustrative example queries and outline the schema of the seeds object. ##### Example query The example query below pulls relevant information about all seeds in a given job. For instance, you can view load times. ```graphql { job(id: 123) { seeds(first: 10, after: "{somePaginationCursorValue}") { uniqueId name executionTime status paginationCursor } } } ``` ##### Fields When querying for `seeds`, the following fields are available: ### Fetching data... --- ### Semantic Layer APIs dbt platform | Starter, Enterprise, Enterprise+ The rapid growth of different tools in the modern data stack has helped data professionals address the diverse needs of different teams. The downside of this growth is the fragmentation of business logic across teams, tools, and workloads.

The [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) allows you to define metrics in code (with [MetricFlow](https://docs.getdbt.com/docs/build/about-metricflow.md)) and dynamically generate and query datasets in downstream tools based on their dbt governed assets, such as metrics and models. Integrating with the Semantic Layer will help organizations that use your product make more efficient and trustworthy decisions with their data. It also helps you to avoid duplicative coding, optimize development workflow, ensure data governance, and guarantee consistency for data consumers. You can use the Semantic Layer for a variety of tools and applications of data. Some common use cases are: * Business intelligence (BI), reporting, and analytics * Data quality and monitoring * Governance and privacy * Data discovery and cataloging * Machine learning and data science [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md) ###### [GraphQL API](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md) [Use GraphQL to query metrics and dimensions in downstream tools.](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/dbt-apis/sl-jdbc.md) ###### [JDBC API](https://docs.getdbt.com/docs/dbt-apis/sl-jdbc.md) [Use a JDBC driver to query metrics and dimensions in downstream tools, while also providing standard metadata functionality.](https://docs.getdbt.com/docs/dbt-apis/sl-jdbc.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/dbt-apis/sl-python.md) ###### [Python SDK](https://docs.getdbt.com/docs/dbt-apis/sl-python.md) [Use the Python SDK to interact with the dbt Semantic Layer using Python.](https://docs.getdbt.com/docs/dbt-apis/sl-python.md) --- ### Service account tokens dbt platform | Starter, Enterprise, Enterprise+ Service account tokens enable you to securely authenticate with the dbt API by assigning each token a narrow set of permissions that more precisely manages access to the API. While similar to [personal access tokens](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md), service account tokens belong to an account rather than a user. You can use service account tokens for system-level integrations that do not run on behalf of any one user. Assign any permission sets available in dbt to your service account token, which can vary slightly depending on your plan: * Enterprise and Enterprise+ plans can apply any permission sets available to service tokens. * Developer and Starter plans can apply Semantic Layer permissions set to service tokens. * Legacy Team plans can apply Account Admin, Member, Job Admin, Read-Only, Metadata, and Semantic Layer permissions set to service tokens. You can assign as many permission sets as needed to one token. For more on permissions sets, see "[Enterprise Permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md)." #### Generate service account tokens You can generate service tokens if you have a Developer [license](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md) and account admin [permissions](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#permission-sets). To create a service token in dbt, follow these steps: 1. From dbt, click on your account name in the left side menu and select **Account settings**. 2. On the left sidebar, click **Service Tokens**. 3. Click the **+ Create service token** button to generate a new token. 1. Enter a name for your token. 2. Add the necessary [permissions](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md#permissions-for-service-account-tokens). 4. Once the token is generated, you won't be able to view this token again so make sure to save it somewhere safe. #### Permissions for service account tokens You can assign service account tokens to any permission set available in dbt. When you assign a permission set to a token, you will also be able to choose whether to grant those permissions to all projects in the account or to specific projects. ##### Team plans using service account tokens The following permissions can be assigned to a service account token on a Team plan. Refer to [Enterprise permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) for more information about these roles. * Account Admin — Account Admin service tokens have full `read + write` access to an account, so please use them with caution. A Team plan refers to this permission set as an "Owner role." * Billing Admin * Job Admin * Metadata Only * Member * Read-only * Semantic Layer Only ##### Enterprise plans using service account tokens Refer to [Enterprise permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) for more information about these roles. * Account Admin — Account Admin service tokens have full `read + write` access to an account, so please use them with caution. * Account Viewer * Admin * Analyst * Billing Admin * Database Admin * Developer * Git Admin * Job Admin * Job Runner * Job Viewer * Manage marketplace apps * Metadata Only * Read-only * Semantic Layer Only * Security Admin * Stakeholder * Team Admin #### Service token update On July 18, 2023, dbt Labs changed how tokens are generated and validated to increase performance. These improvements only apply to tokens created after July 18, 2023. Old tokens remain valid, but if they are used in high-frequency API invocations, we recommend you rotate them for reduced latency. To rotate your token: 1. Navigate to **Account settings** and click **Service tokens** on the left side pane. 2. Verify the **Created** date for the token is *on or before* July 18, 2023. 3. Click **+ New Token** on the top right side of the screen. Ensure the new token has the same permissions as the old one. 4. Copy the new token and replace the old one in your systems. Store it in a safe place, as it will not be available again once the creation screen is closed. 5. Delete the old token in dbt by clicking the **trash can icon**. *Only take this action after the new token is in place to avoid service disruptions*. #### FAQs I'm receiving a 403 error 'Forbidden: Access denied' when using service tokens All [service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) traffic is subject to IP restrictions. When using a service token, the following 403 response error indicates the IP is not on the allowlist. To resolve this, you should add your third-party integration CIDRs (network addresses) to your allowlist. The following is an example of the 403 response error: ```json { "status": { "code": 403, "is_success": False, "user_message": ("Forbidden: Access denied"), "developer_message": None, }, "data": { "account_id": , "user_id": , "is_service_token": , "account_access_denied": True, }, } ``` --- ### Snapshots object schema dbt platform | Starter, Enterprise, Enterprise+ [Snapshots](https://docs.getdbt.com/docs/build/snapshots.md) represent point-in-time copies of your data, allowing you to track historical changes. You can query your snapshots from the Discovery API. The [Example query](#example-query) illustrates a few fields you can query with the `snapshots` object. Refer to [Fields](#fields) to view the entire schema, which provides all possible fields you can query. ##### Arguments When querying for `snapshots`, you can use the following arguments: ### Fetching data... ##### Example query You can specify the `environmentId`, filter by the database, and limit to the first 100 to see the first 100 snapshots in the `analytics` database, including their execution performance and status information: ```graphql query { environment(id: 834) { applied { snapshots( filter: { database: "analytics" }, first: 100 ) { edges { node { executionInfo { compileCompletedAt compileStartedAt executeCompletedAt executeStartedAt executionTime lastRunStatus lastRunId } fqn name } } } } } } ``` ##### Fields When querying for `snapshots`, you can use the following fields: ### Fetching data... ##### Key fields from nodes ### Fetching data... --- ### Snapshots object schema dbt platform | Starter, Enterprise, Enterprise+ The snapshots object allows you to query information about all snapshots in a given job. ##### Arguments When querying for `snapshots`, the following arguments are available. ### Fetching data... Below we show some illustrative example queries and outline the schema of the snapshots object. ##### Example query The database, schema, and identifier arguments are optional. This means that with this endpoint you can: * Find a specific snapshot by providing `..` * Find all of the snapshots in a database and/or schema by providing `` and/or `` ###### Find snapshots information for a job The example query returns information about all snapshots in this job. ```graphql { job(id: 123) { snapshots(first: 10, after: "{somePaginationCursorValue}") { uniqueId name executionTime environmentId executeStartedAt executeCompletedAt paginationCursor } } } ``` ##### Fields When querying for `snapshots`, the following fields are available: ### Fetching data... --- ### Source object schema dbt platform | Starter, Enterprise, Enterprise+ The source object allows you to query information about a particular source in a given job. ##### Arguments When querying for a `source`, the following arguments are available. ### Fetching data... Below we show some illustrative example queries and outline the schema of the source object. ##### Example query The query below pulls relevant information about a given source. For instance, you can view the load time and the state (pass, fail, error) of that source. ```graphql { job(id: 123) { source(uniqueId: "source.jaffle_shop.snowplow.event") { uniqueId sourceName name state maxLoadedAt criteria { warnAfter { period count } errorAfter { period count } } maxLoadedAtTimeAgoInS } } } ``` ##### Fields When querying for a `source`, the following fields are available: ### Fetching data... --- ### Sources object schema dbt platform | Starter, Enterprise, Enterprise+ [Sources](https://docs.getdbt.com/docs/build/sources.md) make it possible to name and describe the data loaded into your warehouse by your extract and load tools. You can query sources through the Discovery API. The [Example query](#example-query) illustrates a few fields you can query with the `sources` object. Refer to [Fields](#fields) to view the entire schema, which provides all possible fields you can query. ##### Arguments When querying for `sources`, you can use the following arguments: ### Fetching data... ##### Example query You can specify the `environmentId` and filter on the database name, to return the freshness and execution status, for the first 100 sources from the given database: ```graphql query { environment(id: 834) { applied { sources( filter: { database: "analytics" }, first: 100 ) { edges { node { name fqn description filePath freshness { freshnessChecked freshnessStatus } sourceName sourceDescription tests { name description testType executionInfo { lastRunStatus } } } } } } } } ``` ##### Fields When querying for `sources`, you can use the following fields: ### Fetching data... ##### Key fields from nodes ### Fetching data... --- ### Sources object schema dbt platform | Starter, Enterprise, Enterprise+ The sources object allows you to query information about all sources in a given job. ##### Arguments When querying for `sources`, the following arguments are available. ### Fetching data... Below we show some illustrative example queries and outline the schema of the sources object. ##### Example queries The database, schema, and identifier arguments are optional. This means that with this endpoint you can: * Find a specific source by providing `..` * Find all of the sources in a database and/or schema by providing `` and/or `` ###### Finding sources by their database, schema, and identifier The example query below finds a source by its unique database, schema, and identifier. ```graphql { job(id: 123) { sources( database: "analytics" schema: "analytics" identifier: "dim_customers" first: 10 after: "{somePaginationCursorValue}" ) { uniqueId paginationCursor } } } ``` ###### Finding sources by their schema The example query below finds all sources in this schema and their respective states (pass, error, fail). ```graphql { job(id: 123) { sources(schema: "analytics", first: 10, after: "{somePaginationCursorValue}") { uniqueId state paginationCursor } } } ``` ##### Fields The sources object can access the *same fields* as the [source node](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-job-source.md). The difference is that the sources object can output a list so instead of querying for fields for one specific source, you can query for those parameters for all sources within a jobID, database, and so on. When querying for `sources`, the following fields are available: ### Fetching data... --- ### Tags object schema dbt platform | Starter, Enterprise, Enterprise+ [Tags](https://docs.getdbt.com/reference/resource-configs/tags.md) provide a mechanism to categorize and group resources within a dbt project, enabling selective execution and management of these resources. You can query tags through the Discovery API. The [Example query](#example-query) illustrates a few fields you can query with the `tags` object. Refer to [Fields](#fields) to view the entire schema, which provides all possible fields you can query. ##### Example query You can use the `environmentId` to return the name of all the tags in your environment: ```graphql query { environment(id: 834) { applied { tags { name } } } } ``` ##### Fields When querying for `tags`, you can use the following fields: ### Fetching data... --- ### Test object schema dbt platform | Starter, Enterprise, Enterprise+ The test object allows you to query information about a particular test. ##### Arguments When querying for a `test`, the following arguments are available. ### Fetching data... Below we show some illustrative example queries and outline the schema (all possible fields you can query) of the test object. ##### Example query The example query below outputs information about a test including the state of the test result. In order of severity, the result can be one of these: "error", "fail", "warn", or "pass". ```graphql { job(id: 123) { test(uniqueId: "test.internal_analytics.not_null_metrics_id") { runId accountId projectId uniqueId name columnName state } } } ``` ##### Fields When querying for a `test`, the following fields are available: ### Fetching data... --- ### Tests object schema dbt platform | Starter, Enterprise, Enterprise+ [Tests](https://docs.getdbt.com/docs/build/data-tests.md) are assertions you make about your models and other resources in your dbt project. When you run `dbt test`, dbt will tell you if each test in your project passes or fails. You can query tests through the Discovery API to understand information about them. The [Example query](#example-query) illustrates a few fields you can query with the `tests` object and how to filter by last-known result. Refer to [Fields](#fields) to view the entire schema, which provides all possible fields you can query. ##### Arguments When querying for `tests`, you can use the following arguments: ### Fetching data... Deprecation: `lastKnownResult` The single-value filter field `lastKnownResult: TestStatus` is deprecated. You should now use `lastKnownResults: [TestStatus]` to filter tests by one or more result statuses. If you pass both fields in the same query, `lastKnownResults` takes precedence. Refer to [Filter tests by last-known result](#filter-tests-by-last-known-result) for more information. ##### Filter tests by last-known result You can filter tests by one or more last-known result values. Use `lastKnownResults` with any combination of these `TestStatus` values: * `pass` * `fail` * `error` * `skipped` * `warn` You can still use the deprecated `lastKnownResult`, but If you pass both fields in the same query, `lastKnownResults` takes precedence. An empty list `[]`, omitting `lastKnownResults`, or passing `null` applies no filter on last-known result. All tests are returned regardless of result status. ##### Example query You can return tests whose last-known result is `error` or `fail`: ```graphql query { environment(id: 834) { applied { tests( filter: { lastKnownResults: [error, fail] }, first: 100 ) { edges { node { name model description expect resourceType testType given lastKnownResult } } } } } } ``` You can combine `lastKnownResults` with other filters on `TestAppliedFilter`, such as `testTypes`: ```graphql query { environment(id: 834) { applied { tests( filter: { testTypes: [GENERIC_DATA_TEST, SINGULAR_DATA_TEST], lastKnownResults: [fail] }, first: 100 ) { edges { node { name model description expect resourceType testType given lastKnownResult } } } } } } ``` ##### Fields When querying for `tests`, you can use the following fields: ### Fetching data... ##### Key fields from nodes ### Fetching data... --- ### Tests object schema dbt platform | Starter, Enterprise, Enterprise+ The tests object allows you to query information about all tests in a given job. ##### Arguments When querying for `tests`, the following arguments are available. ### Fetching data... Below we show some illustrative example queries and outline the schema (all possible fields you can query) of the tests object. ##### Example query The example query below finds all tests in this job and includes information about those tests. ```graphql { job(id: 123) { tests(first: 10, after: "{somePaginationCursorValue}") { runId accountId projectId uniqueId name columnName state paginationCursor } } } ``` ##### Fields When querying for `tests`, the following fields are available: ### Fetching data... --- ### Use cases and examples for the Discovery API dbt platform | Starter, Enterprise, Enterprise+ With the Discovery API, you can query the metadata in dbt to learn more about your dbt deployments and the data it generates to analyze them and make improvements. You can use the API in a variety of ways to get answers to your business questions. Below describes some of the uses of the API and is meant to give you an idea of the questions this API can help you answer. | Use case | Outcome | Example questions | | --------------------------- | ---------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | [Performance](#performance) | Identify inefficiencies in pipeline execution to reduce infrastructure costs and improve timeliness. | - What’s the latest status of each model?
- Do I need to run this model?
- How long did my DAG take to run? | | [Quality](#quality) | Monitor data source freshness and test results to resolve issues and drive trust in data. | - How fresh are my data sources?
- Which tests and models failed?
- What’s my project’s test coverage? | | [Discovery](#discovery) | Find and understand relevant datasets and semantic nodes with rich context and metadata. | - What do these tables and columns mean?
- What's the full data lineage at a model level?
- Which metrics can I query? | | [Governance](#governance) | Audit data development and facilitate collaboration within and between teams. | - Who is responsible for this model?
- How do I contact the model’s owner?
- Who can use this model? | | [Development](#development) | Understand dataset changes and usage and gauge impacts to inform project definition. | - How is this metric used in BI tools?
- Which nodes depend on this data source?
- How has a model changed? What impact? | #### Performance You can use the Discovery API to identify inefficiencies in pipeline execution to reduce infrastructure costs and improve timeliness. Below are example questions and queries you can run. For performance use cases, people typically query the historical or latest applied state across any part of the DAG (for example, models) using the `environment`, `modelHistoricalRuns`, or job-level endpoints. ##### How long did each model take to run? It’s helpful to understand how long it takes to build models (tables) and tests to execute during a dbt run. Longer model build times result in higher infrastructure costs and fresh data arriving later to stakeholders. Analyses like these can be in observability tools or ad-hoc queries, like in a notebook. [![Model timing visualization in dbt](/img/docs/dbt-platform/discovery-api/model-timing.png?v=2 "Model timing visualization in dbt")](#)Model timing visualization in dbt Example query with code Data teams can monitor the performance of their models, identify bottlenecks, and optimize the overall data pipeline by fetching execution details like `executionTime` and `runElapsedTime`: 1. Use latest state environment-level API to get a list of all executed models and their execution time. Then, sort the models by `executionTime` in descending order. ```graphql query AppliedModels($environmentId: BigInt!, $first: Int!) { environment(id: $environmentId) { applied { models(first: $first) { edges { node { name uniqueId materializedType executionInfo { lastSuccessRunId executionTime executeStartedAt } } } } } } } ``` 2. Get the most recent 20 run results for the longest running model. Review the results of the model across runs or you can go to the job/run or commit itself to investigate further. ```graphql query ModelHistoricalRuns( $environmentId: BigInt! $uniqueId: String $lastRunCount: Int ) { environment(id: $environmentId) { applied { modelHistoricalRuns( uniqueId: $uniqueId lastRunCount: $lastRunCount ) { name runId runElapsedTime runGeneratedAt executionTime executeStartedAt executeCompletedAt status } } } } ``` 3. Use the query results to plot a graph of the longest running model’s historical run time and execution time trends. ```python # Import libraries import os import matplotlib.pyplot as plt import pandas as pd import requests # Set API key auth_token = *[SERVICE_TOKEN_HERE]* # Query the API def query_discovery_api(auth_token, gql_query, variables): response = requests.post('https://metadata.cloud.getdbt.com/graphql', headers={"authorization": "Bearer "+auth_token, "content-type": "application/json"}, json={"query": gql_query, "variables": variables}) data = response.json()['data'] return data # Get the latest run metadata for all models models_latest_metadata = query_discovery_api(auth_token, query_one, variables_query_one)['environment'] # Convert to dataframe models_df = pd.DataFrame([x['node'] for x in models_latest_metadata['applied']['models']['edges']]) # Unnest the executionInfo column models_df = pd.concat([models_df.drop(['executionInfo'], axis=1), models_df['executionInfo'].apply(pd.Series)], axis=1) # Sort the models by execution time models_df_sorted = models_df.sort_values('executionTime', ascending=False) print(models_df_sorted) # Get the uniqueId of the longest running model longest_running_model = models_df_sorted.iloc[0]['uniqueId'] # Define second query variables variables_query_two = { "environmentId": *[ENVR_ID_HERE]* "lastRunCount": 10, "uniqueId": longest_running_model } # Get the historical run metadata for the longest running model model_historical_metadata = query_discovery_api(auth_token, query_two, variables_query_two)['environment']['applied']['modelHistoricalRuns'] # Convert to dataframe model_df = pd.DataFrame(model_historical_metadata) # Filter dataframe to only successful runs model_df = model_df[model_df['status'] == 'success'] # Convert the runGeneratedAt, executeStartedAt, and executeCompletedAt columns to datetime model_df['runGeneratedAt'] = pd.to_datetime(model_df['runGeneratedAt']) model_df['executeStartedAt'] = pd.to_datetime(model_df['executeStartedAt']) model_df['executeCompletedAt'] = pd.to_datetime(model_df['executeCompletedAt']) # Plot the runElapsedTime over time plt.plot(model_df['runGeneratedAt'], model_df['runElapsedTime']) plt.title('Run Elapsed Time') plt.show() # # Plot the executionTime over time plt.plot(model_df['executeStartedAt'], model_df['executionTime']) plt.title(model_df['name'].iloc[0]+" Execution Time") plt.show() ``` Plotting examples: [![The plot of runElapsedTime over time](/img/docs/dbt-platform/discovery-api/plot-of-runelapsedtime.png?v=2 "The plot of runElapsedTime over time")](#)The plot of runElapsedTime over time [![The plot of executionTime over time](/img/docs/dbt-platform/discovery-api/plot-of-executiontime.png?v=2 "The plot of executionTime over time")](#)The plot of executionTime over time ##### What’s the latest state of each model? The Discovery API provides information about the applied state of models and how they arrived in that state. You can retrieve the status information from the most recent run and most recent successful run (execution) from the `environment` endpoint and dive into historical runs using job-based and `modelByEnvironment` endpoints. Example query The API returns full identifier information (`database.schema.alias`) and the `executionInfo` for both the most recent run and most recent successful run from the database: ```graphql query ($environmentId: BigInt!, $first: Int!) { environment(id: $environmentId) { applied { models(first: $first) { edges { node { uniqueId compiledCode database schema alias materializedType executionInfo { executeCompletedAt lastJobDefinitionId lastRunGeneratedAt lastRunId lastRunStatus lastRunError lastSuccessJobDefinitionId runGeneratedAt lastSuccessRunId } } } } } } } ``` ##### What happened with my job run? You can query the metadata at the job level to review results for specific runs. This is helpful for historical analysis of deployment performance or optimizing particular jobs. Example query Deprecated example: ```graphql query ($jobId: Int!, $runId: Int!) { models(jobId: $jobId, runId: $runId) { name status tests { name status } } } ``` New example: ```graphql query ($jobId: BigInt!, $runId: BigInt!) { job(id: $jobId, runId: $runId) { models(first: 10, after: "{somePaginationCursorValue}") { name status tests { name status } paginationCursor } } } ``` For jobs with many models, paginate with `first`, `after`, and `paginationCursor`. Refer to [Job-based queries](https://docs.getdbt.com/docs/dbt-apis/discovery-querying.md#job-based-queries) in the Discovery API querying guide. ```graphql query JobModelsPage($jobId: BigInt!, $runId: BigInt, $first: Int!, $after: String) { job(id: $jobId, runId: $runId) { models(first: $first, after: $after) { uniqueId name status paginationCursor } } } ``` ##### What’s changed since the last run? Unnecessary runs incur higher infrastructure costs and load on the data team and their systems. A model doesn’t need to be run if it’s a view and there's no code change since the last run, or if it’s a table/incremental with no code change since last run and source data has not been updated since the last run. Example query With the API, you can compare the `rawCode` between the definition and applied state, and review when the sources were last loaded (source `maxLoadedAt` relative to model `executeCompletedAt`) given the `materializedType` of the model: ```graphql query ($environmentId: BigInt!, $first: Int!) { environment(id: $environmentId) { applied { models( first: $first filter: { uniqueIds: "MODEL.PROJECT.MODEL_NAME" } ) { edges { node { rawCode ancestors(types: [Source]) { ... on SourceAppliedStateNestedNode { freshness { maxLoadedAt } } } executionInfo { runGeneratedAt executeCompletedAt } materializedType } } } } definition { models( first: $first filter: { uniqueIds: "MODEL.PROJECT.MODEL_NAME" } ) { edges { node { rawCode runGeneratedAt materializedType } } } } } } ``` #### Quality You can use the Discovery API to monitor data source freshness and test results to diagnose and resolve issues and drive trust in data. When used with [webhooks](https://docs.getdbt.com/docs/deploy/webhooks.md), can also help with detecting, investigating, and alerting issues. Below lists example questions the API can help you answer. Below are example questions and queries you can run. For quality use cases, people typically query the historical or latest applied state, often in the upstream part of the DAG (for example, sources), using the `environment` or `environment { applied { modelHistoricalRuns } }` endpoints. ##### Which models and tests failed to run? By filtering on the latest status, you can get lists of models that failed to build and tests that failed during their most recent execution. This is helpful when diagnosing issues with the deployment that result in delayed or incorrect data. Example query with code 1. Get the latest run results across all jobs in the environment and return only the models and tests that errored/failed. ```graphql query ($environmentId: BigInt!, $first: Int!) { environment(id: $environmentId) { applied { models(first: $first, filter: { lastRunStatus: error }) { edges { node { name executionInfo { lastRunId } } } } tests(first: $first, filter: { status: "fail" }) { edges { node { name executionInfo { lastRunId } } } } } } } ``` 2. Review the historical execution and test failure rate (up to 20 runs) for a given model, such as a frequently used and important dataset. ```graphql query ($environmentId: BigInt!, $uniqueId: String!, $lastRunCount: Int) { environment(id: $environmentId) { applied { modelHistoricalRuns(uniqueId: $uniqueId, lastRunCount: $lastRunCount) { name executeStartedAt status tests { name status } } } } } ``` 3. Identify the runs and plot the historical trends of failure/error rates. ##### When was the data my model uses last refreshed? You can get the metadata on the latest execution for a particular model or across all models in your project. For instance, investigate when each model or snapshot that's feeding into a given model was last executed or the source or seed was last loaded to gauge the *freshness* of the data. Example query with code ```graphql query ($environmentId: BigInt!, $first: Int!) { environment(id: $environmentId) { applied { models( first: $first filter: { uniqueIds: "MODEL.PROJECT.MODEL_NAME" } ) { edges { node { name ancestors(types: [Model, Source, Seed, Snapshot]) { ... on ModelAppliedStateNestedNode { name resourceType materializedType executionInfo { executeCompletedAt } } ... on SourceAppliedStateNestedNode { sourceName name resourceType freshness { maxLoadedAt } } ... on SnapshotAppliedStateNestedNode { name resourceType executionInfo { executeCompletedAt } } ... on SeedAppliedStateNestedNode { name resourceType executionInfo { executeCompletedAt } } } } } } } } } ``` ```python # Extract graph nodes from response def extract_nodes(data): models = [] sources = [] groups = [] for model_edge in data["applied"]["models"]["edges"]: models.append(model_edge["node"]) for source_edge in data["applied"]["sources"]["edges"]: sources.append(source_edge["node"]) for group_edge in data["definition"]["groups"]["edges"]: groups.append(group_edge["node"]) models_df = pd.DataFrame(models) sources_df = pd.DataFrame(sources) groups_df = pd.DataFrame(groups) return models_df, sources_df, groups_df # Construct a lineage graph with freshness info def create_freshness_graph(models_df, sources_df): G = nx.DiGraph() current_time = datetime.now(timezone.utc) for _, model in models_df.iterrows(): max_freshness = pd.Timedelta.min if "meta" in models_df.columns: freshness_sla = model["meta"]["freshness_sla"] else: freshness_sla = None if model["executionInfo"]["executeCompletedAt"] is not None: model_freshness = current_time - pd.Timestamp(model["executionInfo"]["executeCompletedAt"]) for ancestor in model["ancestors"]: if ancestor["resourceType"] == "SourceAppliedStateNestedNode": ancestor_freshness = current_time - pd.Timestamp(ancestor["freshness"]['maxLoadedAt']) elif ancestor["resourceType"] == "ModelAppliedStateNestedNode": ancestor_freshness = current_time - pd.Timestamp(ancestor["executionInfo"]["executeCompletedAt"]) if ancestor_freshness > max_freshness: max_freshness = ancestor_freshness G.add_node(model["uniqueId"], name=model["name"], type="model", max_ancestor_freshness = max_freshness, freshness = model_freshness, freshness_sla=freshness_sla) for _, source in sources_df.iterrows(): if source["maxLoadedAt"] is not None: G.add_node(source["uniqueId"], name=source["name"], type="source", freshness=current_time - pd.Timestamp(source["maxLoadedAt"])) for _, model in models_df.iterrows(): for parent in model["parents"]: G.add_edge(parent["uniqueId"], model["uniqueId"]) return G ``` Graph example: [![A lineage graph with source freshness information](/img/docs/dbt-platform/discovery-api/lineage-graph-with-freshness-info.png?v=2 "A lineage graph with source freshness information")](#)A lineage graph with source freshness information ##### Are my data sources fresh? Checking [source freshness](https://docs.getdbt.com/docs/build/sources.md#source-data-freshness) allows you to ensure that sources loaded and used in your dbt project are compliant with expectations. The API provides the latest metadata about source loading and information about the freshness check criteria. [![Source freshness page in dbt](/img/docs/dbt-platform/discovery-api/source-freshness-page.png?v=2 "Source freshness page in dbt")](#)Source freshness page in dbt Example query ```graphql query ($environmentId: BigInt!, $first: Int!) { environment(id: $environmentId) { applied { sources( first: $first filter: { freshnessChecked: true, database: "production" } ) { edges { node { sourceName name identifier loader freshness { freshnessJobDefinitionId freshnessRunId freshnessRunGeneratedAt freshnessStatus freshnessChecked maxLoadedAt maxLoadedAtTimeAgoInS snapshottedAt criteria { errorAfter { count period } warnAfter { count period } } } } } } } } } ``` ##### What’s the test coverage and status? [Data tests](https://docs.getdbt.com/docs/build/data-tests.md) are an important way to ensure that your stakeholders are reviewing high-quality data. You can execute tests during a dbt run. The Discovery API provides complete test results for a given environment or job, which it represents as the `children` of a given node that’s been tested (for example, a `model`). Example query For the following example, the `parents` are the nodes (code) that's being tested and `executionInfo` describes the latest test results: ```graphql query ($environmentId: BigInt!, $first: Int!) { environment(id: $environmentId) { applied { tests(first: $first) { edges { node { name columnName parents { name resourceType } executionInfo { lastRunStatus lastRunError executeCompletedAt executionTime } } } } } } } ``` ##### How is this model contracted and versioned? To enforce the shape of a model's definition, you can define contracts on models and their columns. You can also specify model versions to keep track of discrete stages in its evolution and use the appropriate one. Example query ```graphql query { environment(id: 123) { applied { models(first: 100, filter: { access: public }) { edges { node { name latestVersion contractEnforced constraints { name type expression columns } catalog { columns { name type } } } } } } } } ``` #### Discovery You can use the Discovery API to find and understand relevant datasets and semantic nodes with rich context and metadata. Below are example questions and queries you can run. For discovery use cases, people typically query the latest applied or definition state, often in the downstream part of the DAG (for example, mart models or metrics), using the `environment` endpoint. ##### What does this dataset and its columns mean? Query the Discovery API to map a table/view in the data platform to the model in the dbt project; then, retrieve metadata about its meaning, including descriptive metadata from its YAML file and catalog information from its YAML file and the schema. Example query ```graphql query ($environmentId: BigInt!, $first: Int!) { environment(id: $environmentId) { applied { models( first: $first filter: { database: "analytics" schema: "prod" identifier: "customers" } ) { edges { node { name description tags meta catalog { columns { name description type } } } } } } } } ``` ##### What's the full data lineage at a model level? The Discovery API enables access to comprehensive model-level data lineage by exposing: * Upstream dependencies of models, including relationships to [sources](https://docs.getdbt.com/docs/build/sources.md), [seeds](https://docs.getdbt.com/docs/build/seeds.md), and [snapshots](https://docs.getdbt.com/docs/build/snapshots.md) * Model execution metadata such as run status, execution time, and freshness * Column-level details, including tests and descriptions * References between models to reconstruct lineage across your project Example query Here's a GraphQL query example that retrieves full model-level data lineage using the Discovery API: ```graphql query ($environmentId: BigInt!, $first: Int!) { environment(id: $environmentId) { applied { models(first: $first) { edges { node { name ancestors(types: [Model, Source, Seed, Snapshot]) { ... on ModelAppliedStateNestedNode { name resourceType } ... on SourceAppliedStateNestedNode { sourceName name resourceType } } } } } } } } ``` ##### Which metrics are available? You can define and query metrics using the [Semantic Layer](https://docs.getdbt.com/docs/build/about-metricflow.md), use them for documentation purposes (like for a data catalog), and calculate aggregations (like in a BI tool that doesn’t query the SL). Example query ```graphql query ($environmentId: BigInt!, $first: Int!) { environment(id: $environmentId) { definition { metrics(first: $first) { edges { node { name description type formula filter tags parents { name resourceType } } } } } } } ``` #### Governance You can use the Discovery API to audit data development and facilitate collaboration within and between teams. For governance use cases, people tend to query the latest definition state, often in the downstream part of the DAG (for example, public models), using the `environment` endpoint. ##### Who is responsible for this model? You can define and surface the groups each model is associated with. Groups contain information like owner. This can help you identify which team owns certain models and who to contact about them. Example query ```graphql query ($environmentId: BigInt!, $first: Int!) { environment(id: $environmentId) { applied { models(first: $first, filter: { uniqueIds: ["MODEL.PROJECT.NAME"] }) { edges { node { name description resourceType access group } } } } definition { groups(first: $first) { edges { node { name resourceType models { name } ownerName ownerEmail } } } } } } ``` ##### Who can use this model? You can enable people the ability to specify the level of access for a given model. In the future, public models will function like APIs to unify project lineage and enable reuse of models using cross-project refs. Example query ```graphql query ($environmentId: BigInt!, $first: Int!) { environment(id: $environmentId) { definition { models(first: $first) { edges { node { name access } } } } } } ``` *** ```graphql query ($environmentId: BigInt!, $first: Int!) { environment(id: $environmentId) { definition { models(first: $first, filter: { access: public }) { edges { node { name } } } } } } ``` #### Development You can use the Discovery API to understand dataset changes and usage and gauge impacts to inform project definition. Below are example questions and queries you can run. For development use cases, people typically query the historical or latest definition or applied state across any part of the DAG using the `environment` endpoint. ##### How is this model or metric used in downstream tools? [Exposures](https://docs.getdbt.com/docs/build/exposures.md) provide a method to define how a model or metric is actually used in dashboards and other analytics tools and use cases. You can query an exposure’s definition to see how project nodes are used and query its upstream lineage results to understand the state of the data used in it, which powers use cases like a freshness and quality status tile. [![Embed data health tiles in your dashboards to distill trust signals for data consumers.](/img/docs/collaborate/dbt-explorer/data-tile-pass.png?v=2 "Embed data health tiles in your dashboards to distill trust signals for data consumers.")](#)Embed data health tiles in your dashboards to distill trust signals for data consumers. Example query Below is an example that reviews an exposure and the models used in it including when they were last executed. ```graphql query ($environmentId: BigInt!, $first: Int!) { environment(id: $environmentId) { applied { exposures(first: $first) { edges { node { name description ownerName url parents { name resourceType ... on ModelAppliedStateNestedNode { executionInfo { executeCompletedAt lastRunStatus } } } } } } } } } ``` ##### How has this model changed over time? The Discovery API provides historical information about any resource in your project. For instance, you can view how a model has evolved over time (across recent runs) given changes to its shape and contents. Example query Review the differences in `compiledCode` or `columns` between runs or plot the “Approximate Size” and “Row Count” `stats` over time: ```graphql query ( $environmentId: BigInt! $uniqueId: String! $lastRunCount: Int! $withCatalog: Boolean! ) { environment(id: $environmentId) { applied { modelHistoricalRuns( uniqueId: $uniqueId lastRunCount: $lastRunCount withCatalog: $withCatalog ) { name compiledCode columns { name } stats { label value } } } } } ``` ##### Which nodes depend on this data source? dbt lineage begins with data sources. For a given source, you can look at which nodes are its children then iterate downstream to get the full list of dependencies. Currently, querying beyond 1 generation (defined as a direct parent-to-child) is not supported. To see the grandchildren of a node, you need to make two queries: one to get the node and its children, and another to get the children nodes and their children. Example query ```graphql query ($environmentId: BigInt!, $first: Int!) { environment(id: $environmentId) { applied { sources( first: $first filter: { uniqueIds: ["SOURCE_NAME.TABLE_NAME"] } ) { edges { node { loader children { uniqueId resourceType ... on ModelAppliedStateNestedNode { database schema alias } } } } } } } } ``` #### Related docs * [Query Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-querying.md) --- ## Best Practices ### Adding data-informed tests with dbt Wizard Use dbt Wizard CLI to identify meaningful test gaps, validate assumptions against current warehouse data, and write focused dbt data tests. This workflow uses the built-in `test_writer` agent. Unlike a request that names tests in advance, `test_writer` starts with project metadata and model importance, forms candidate assertions, and checks those assertions against available data before proposing YAML changes. CLI workflow The built-in `test_writer` agent is available in dbt Wizard CLI. Refer to [Use subagents with dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-subagents.md) for availability and agent behavior. #### Prerequisites Before you begin: * [Install and configure dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md). * Start dbt Wizard from a dbt project with a current `target/manifest.json`. * Configure a development connection that can query the models you want to inspect. * Build the relevant models if their development relations don't exist. * Decide whether sensitive columns or expensive relations should be excluded from profiling. Warehouse queries can consume compute and can expose data values in your terminal session. Review proposed queries and apply your organization's data-access policies. #### Choose a focused scope Start with a model or a small project area. This keeps the proposed tests reviewable and the warehouse queries bounded: ```text Use test_writer to improve test coverage for stg_customers. Inspect its existing tests and downstream importance, validate candidate assumptions against current data, and propose only high-signal tests. Do not change model SQL. ``` You can also ask dbt Wizard to find a starting point: ```text Use test_writer to find three important models with weak test coverage. Prioritize high fan-out models and models that define a business grain. Explain the ranking before writing tests. ``` State constraints in the prompt. For example, name columns that must not be queried, limit the number of models, or ask dbt Wizard to avoid custom generic tests. #### Review the coverage analysis Before it writes YAML, ask `test_writer` to explain: * Which models and columns already have tests. * Why the selected model is important to downstream resources. * What grain or business rule each candidate test represents. * Which candidates are based on code or metadata and which are inferred from data. * What warehouse query would validate each assumption. The strongest candidates protect model contracts and business behavior, not just the shape of today's sample. | Candidate test | Evidence to review | | ----------------- | ------------------------------------------------------------------------------------------------------------------- | | `not_null` | The column is required by the model grain, contract, or downstream logic, and current nulls have been investigated. | | `unique` | The column or column combination represents the intended grain, and duplicate checks support that assumption. | | `relationships` | The parent resource and key are authoritative, and orphaned values have been investigated. | | `accepted_values` | The field is a governed enumeration, not merely a list of values observed in one query. | If the data disproves a candidate assertion, investigate the discrepancy. Don't automatically add a filter or weaken the test to make it pass. #### Approve the warehouse checks `test_writer` can use warehouse previews to check assumptions such as uniqueness, nullability, relationships, and observed values. Before approving a query, confirm that: * It uses the expected target and relation. * Its selected columns comply with data-access policies. * Its filters and limits won't hide the condition the test is meant to protect. * Its expected scan and run time are reasonable. You can ask for a smaller or more targeted query before approving it: ```text Validate the proposed grain without selecting customer attributes. Query only customer_id and aggregate counts, and explain any sampling or date filter. ``` #### Review the proposed tests For each proposed YAML change, confirm that: * The test expresses an intended rule, not an accidental property of current data. * The test is attached to the correct model and column. * The YAML syntax matches the dbt version used by the project. * Existing tests aren't duplicated. * Test names, arguments, configuration, and severity follow project conventions. * The diff doesn't include unrelated documentation or model changes. The `test_writer` agent writes the tests. The normal dbt Wizard validation flow is responsible for running them. Select an appropriate [validation level](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-3-validate-changes.md#choose-a-validation-level) and review any failing rows before accepting the final change. #### Investigate a failed candidate When a new test fails, keep the original assumption visible while you determine what the data means: ```text The proposed unique test on customer_id fails. Show the duplicate pattern, check whether the intended grain uses another column, and tell me whether this is a model defect, a source-data issue, or a bad test assumption. Do not modify the test until you explain the evidence. ``` Possible outcomes include fixing a model defect, documenting a source limitation, selecting the correct compound grain, or rejecting the candidate test. #### Understand the limits Data-informed test generation still requires engineering judgment: * Current data can support or disprove an assumption, but it doesn't define the business rule. * A passing test can miss future values, late-arriving data, and unqueried partitions. * `accepted_values` tests need a governed list when new legitimate values can appear. * Warehouse access, permissions, stale relations, and sampling can limit the evidence available. * High test counts can increase build time without improving meaningful coverage. * Tests don't replace model contracts, source freshness checks, monitoring, or review by a model owner. #### Related docs * [Validate dbt changes with dbt Wizard](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-3-validate-changes.md) * [Use subagents with dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-subagents.md) * [Data tests](https://docs.getdbt.com/docs/build/data-tests.md) * [Test best practices](https://docs.getdbt.com/best-practices/writing-custom-generic-tests.md) --- ### Available materializations Views and tables and incremental models, oh my! In this section we’ll start getting our hands dirty digging into the three basic materializations that ship with dbt. They are considerably less scary and more helpful than lions, tigers, or bears — although perhaps not as cute (can data be cute? We at dbt Labs think so). We’re going to define, implement, and explore: * 🔍 [**views**](https://docs.getdbt.com/docs/build/materializations.md#view) * ⚒️ [**tables**](https://docs.getdbt.com/docs/build/materializations.md#table) * 📚 [**incremental model**](https://docs.getdbt.com/docs/build/materializations.md#incremental) info 👻 There is a fourth default materialization available in dbt called [**ephemeral materialization**](https://docs.getdbt.com/docs/build/materializations.md#ephemeral). It is less broadly applicable than the other three, and better deployed for specific use cases that require weighing some tradeoffs. We chose to leave it out of this guide and focus on the three materializations that will power 99% of your modeling needs. **Views and Tables are the two basic categories** of object that we can create across warehouses. They exist natively as types of objects in the warehouse, as you can see from this screenshot of Snowflake (depending on your warehouse the interface will look a little different). **Incremental models** and other materializations types are a little bit different. They tell dbt to **construct tables in a special way**. ![Tables and views in the browser on Snowflake.](/assets/images/tables-and-views-d510f9a1eecc0c54f4352182389f3435.png) ##### Views * ✅ **The default materialization in dbt**. A starting project has no configurations defined for materializations, which means *everything* is by default built as a view. * 👩‍💻 **Store *only the SQL logic* of the transformation in the warehouse, *not the data***. As such, they make a great default. They build almost instantly and cost almost nothing to build. * ⏱️ Always reflect the **most up-to-date** version of the input data, as they’re run freshly every time they’re queried. * 👎 **Have to be processed every time they’re queried, so slower to return results than a table of the same data.** That also means they can cost more over time, especially if they contain intensive transformations and are queried often. ##### Tables * 🏗️ **Tables store the data itself** as opposed to views which store the query logic. This means we can pack all of the transformation compute into a single run. A view is storing a *query* in the warehouse. Even to preview that data we have to query it. A table is storing the literal rows and columns on disk. * 🏎️ Querying lets us **access that transformed data directly**, so we get better performance. Tables feel **faster and more responsive** compared to views of the same logic. * 💸 **Improves compute costs.** Compute is significantly more expensive than storage. So while tables use much more storage, it’s generally an economical tradeoff, as you only pay for the transformation compute when you build a table during a job, rather than every time you query it. * 🔍 **Ideal for models that get queried regularly**, due to the combination of these qualities. * 👎 **Limited to the source data that was available when we did our most recent run.** We’re ‘freezing’ the transformation logic into a table. So if we run a model as a table every hour, at 10:59a we still only have data up to 10a, because that was what was available in our source data when we ran the table last at 10a. Only at the next run will the newer data be included in our rebuild. ##### Incremental models * 🧱 **Incremental** models build a **table** in **pieces over time**, only adding and updating new or changed records. * 🏎️  **Builds more quickly** than a regular table of the same logic. * 🐢 **Initial runs are slow.** Typically we use incremental models on very large datasets, so building the initial table on the full dataset is time consuming and equivalent to the table materialization. * 👎 **Add complexity.** Incremental models require deeper consideration of layering and timing. * 👎 Can drift from source data over time. As we’re not processing all of the source data when we run an incremental model, extra effort is required to capture changes to historical data. ##### Comparing the materialization types | | view | table | incremental | | -------------------- | ------------------------------------ | -------------------------------------- | -------------------------------------- | | 🛠️⌛ **build time** | 💚  fastest — only stores logic | ❤️  slowest — linear to size of data | 💛  medium — builds flexible portion | | 🛠️💸 **build costs** | 💚  lowest — no data processed | ❤️  highest — all data processed | 💛  medium — some data processed | | 📊💸 **query costs** | ❤️  higher — reprocess every query | 💚  lower — data in warehouse | 💚  lower — data in warehouse | | 🍅🌱 **freshness** | 💚  best — up-to-the-minute of query | 💛  moderate — up to most recent build | 💛  moderate — up to most recent build | | 🧠🤔 **complexity** | 💚 simple - maps to warehouse object | 💚 simple - map to warehouse concept | 💛 moderate - adds logical complexity | info 🔑 **Time is money.** Notice in the above chart that the time and costs rows contain the same results. This is to highlight that when we’re talking about time in warehouses, we’re talking about compute time, which is the primary driver of costs. --- ### Best practice guides #### [🗃️ How we structure our dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md) [4 items](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md) --- ### Best practices #### Putting it all together * 📊 We've walked through **creating semantic models and metrics** for basic coverage of a key business area. * 🔁 In doing so we've looked at how to **refactor a frozen rollup** into a dynamic, flexible new life in the Semantic Layer. #### Best practices * ✅ **Prefer normalization** when possible to allow MetricFlow to denormalize dynamically for end users. * ✅ Use **marts to denormalize** when needed, for instance grouping tables together into richer components, or getting measures on dimensional tables attached to a table with a time spine. * ✅ When source data is **well normalized** you can **build semantic models on top of staging models**. * ✅ **Prefer** computing values in **measures and metrics** when possible as opposed to in frozen rollups. * ❌ **Don't directly refactor the code you have in production**, build in parallel so you can audit the Semantic Layer output and deprecate old marts gracefully. #### Key commands * 🔑 Use `dbt parse` to generate a fresh semantic manifest. * 🔑 Use `dbt sl list dimensions --metrics [metric name]` to check that you're increasing dimensionality as you progress. * 🔑 Use `dbt sl query [query options]` to preview the output from your metrics as you develop. #### Next steps * 🗺️ Use these best practices to map out your team's plan to **incrementally adopt the Semantic Layer**. * 🤗 Get involved in the community and ask questions, **help craft best practices**, and share your progress in building a Semantic Layer. * [Validate semantic nodes in CI](https://docs.getdbt.com/docs/deploy/ci-jobs.md#semantic-validations-in-ci) to ensure code changes made to dbt models don't break these metrics. The Semantic Layer is the biggest paradigm shift thus far in the young practice of analytics engineering. It's ready to provide value right away, but is most impactful if you move your project towards increasing normalization, and allow MetricFlow to do the denormalization for you with maximum dimensionality. We will be releasing more resources soon covering implementation of the Semantic Layer in dbt with various integrated BI tools. This is just the beginning, hopefully this guide has given you a path forward for building your data platform in this new era. Refer to [Semantic Layer FAQs](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-faqs.md) for more information. --- ### Best practices for dbt and Unity Catalog Your Databricks dbt project should be configured after following the ["How to set up your databricks dbt project guide"](https://docs.getdbt.com/guides/set-up-your-databricks-dbt-project.md). Now we’re ready to start building a dbt project using Unity Catalog. However, we should first consider how we want to allow dbt users to interact with our different catalogs. We recommend the following best practices to ensure the integrity of your production data: #### Isolate your Bronze (aka source) data We recommend using Unity Catalog because it allows you to reference data across your organization from any other catalog, legacy Hive metastore, external metastore, or Delta Live Table pipeline outputs. Additionally, Databricks offers the capability to [interact with external data](https://docs.databricks.com/external-data/index.html#interact-with-external-data-on-databricks) and supports query federation to many [database solutions](https://docs.databricks.com/query-federation/index.html#what-is-query-federation-for-databricks-sql). This means your dev and prod environments will have access to your source data, even if it is defined in another catalog or external data source. Raw data in your Bronze layer should be defined as dbt [sources](https://docs.getdbt.com/docs/build/sources.md) and should be read-only for all dbt interactions in both development and production. By default, we recommend that all of these inputs should be accessible by all dbt users in all dbt environments. This ensures that transformations in all environments begin with the same input data, and the results observed in development will be replicated when that code is deployed. That being said, there are times when your company’s data governance requirements necessitate using multiple workspaces or data catalogs depending on the environment. If you have different data catalogs/schemas for your source data depending on your environment, you can use the [target.name](https://docs.getdbt.com/reference/dbt-jinja-functions/target.md#use-targetname-to-change-your-source-database) to change the data catalog/schema you’re pulling from depending on the environment. If you use multiple Databricks workspaces to isolate development from production, you can use dbt’s [environment variables](https://docs.getdbt.com/docs/build/environment-variables.md) in your connection config strings to reference multiple workspaces from one dbt project. You can also do the same thing for your SQL warehouse so you can have different sizes based on your environments. To do so, use dbt's [environment variable syntax](https://docs.getdbt.com/docs/build/environment-variables.md#special-environment-variables) for Server Hostname of your Databricks workspace URL and HTTP Path for the SQL warehouse in your connection settings. Note that Server Hostname still needs to appear to be a valid domain name to pass validation checks, so you will need to hard-code the domain suffix on the URL, eg `{{env_var('DBT_HOSTNAME')}}.cloud.databricks.com` and the path prefix for your warehouses, eg `/sql/1.0/warehouses/{{env_var('DBT_HTTP_PATH')}}`. [![Using environment variable syntax in connection configs](/img/guides/databricks-guides/databricks-connection-env-vars.png?v=2 "Using environment variable syntax in connection configs")](#)Using environment variable syntax in connection configs When you create environments in dbt, you can assign environment variables to populate the connection information dynamically. Don’t forget to make sure the tokens you use in the credentials for those environments were generated from the associated workspace. [![Defining default environment variable values](/img/guides/databricks-guides/databricks-env-variables.png?v=2 "Defining default environment variable values")](#)Defining default environment variable values #### Access Control For granting access to data consumers, use dbt’s [grants config](https://docs.getdbt.com/reference/resource-configs/grants.md) to apply permissions to database objects generated by dbt models. This lets you configure grants as a structured dictionary rather than writing all the SQL yourself and lets dbt take the most efficient path to apply those grants. As for permissions to run dbt and read non-consumer-facing data sources, the table below summarizes an access model. Effectively, all developers should get no more than read access on the prod catalog and write access in the dev catalog. When using dbt, schema creation is taken care of for you; unlike traditional data warehousing workflows, you do not need to manually create any Unity Catalog assets other than the top-level catalogs. The **prod** service principal should have “read” access to raw source data, and “write” access to the prod catalog. If you add a **test** catalog and associated dbt environment, you should create a dedicated service principal. The test service principal should have *read* on raw source data, and *write* on the **test** catalog but no permissions on the prod or dev catalogs. A dedicated test environment should be used for [CI testing](https://www.getdbt.com/blog/adopting-ci-cd-with-dbt-cloud/) only. **Table-level grants:** | | Source Data | Development catalog | Production catalog | Test catalog | | ---------------------------- | ----------- | ------------------- | ------------------ | --------------- | | developers | select | select & modify | select or none | none | | production service principal | select | none | select & modify | none | | Test service principal | select | none | none | select & modify | **Schema-level grants:** | | Source Data | Development catalog | Production catalog | Test catalog | | ---------------------------- | ----------- | --------------------------------- | -------------------------------- | -------------------------------- | | developers | use | use, create schema, table, & view | use or none | none | | production service principal | use | none | use, create schema, table & view | none | | Test service principal | use | none | none | use, create schema, table & view | #### Next steps Ready to start transforming your Unity Catalog datasets with dbt? Check out the resources below for guides, tips, and best practices: * [How we structure our dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md) * [Self-paced dbt fundamentals training course](https://learn.getdbt.com/courses/dbt-fundamentals) * [Customizing CI/CD](https://docs.getdbt.com/guides/custom-cicd-pipelines.md) * [Debugging errors](https://docs.getdbt.com/guides/debug-errors.md) * [Writing custom generic tests](https://docs.getdbt.com/best-practices/writing-custom-generic-tests.md) * [dbt packages hub](https://hub.getdbt.com/) --- ### Best practices for materializations First, let’s consider some properties of various levels of our dbt project and materializations. * 🔍 **Views** return the freshest, real-time state of their input data when they’re queried, this makes them ideal as **building blocks** for larger models. * 🧶  When we’re building a model that stitches lots of other models together, we don’t want to worry about all those models having different states of freshness because they were built into tables at different times. We want all those inputs to give us all the underlying source data available. * 🤏 **Views** are also great for **small datasets** with minimally intensive logic that we want **near realtime** access to. * 🛠️ **Tables** are the **most performant** materialization, as they just return the transformed data when they’re queried, with no need to reprocess it. * 📊  This makes tables great for **things end users touch**, like a mart that services a popular dashboard. * 💪 Tables are also ideal for **frequently used, compute intensive** transformations. Making a table allows us to ‘freeze’ those transformations in place. * 📚  **Incremental models** are useful for the **same purposes as tables**, they just enable us to build them on larger datasets, so they can be **built** *and* **accessed** in a **performant** way. ##### Project-level configuration Keeping these principles in mind, we can applying these materializations to a project. Earlier we looked at how to configure an individual model's materializations. In practice though, we'll want to set materializations at the folder level, and use individual model configs to override those as needed. This will keep our code DRY and avoid repeating the same config blocks in every model. * 📂  In the `dbt_project.yml` we have a `models:` section (by default at the bottom of the file) we can use define various **configurations for entire directories**. * ⚙️  These are the **same configs that are passed to a `{{ config() }}` block** for individual models, but they get set for *every model in that directory and any subdirectories nested within it*. * ➕  We demarcate between a folder name and a configuration by using a `+`, so `marketing`, `paid_ads`, and `google` below are folder names, whereas **`+materialized` is a configuration** being applied to those folder and all folders nested below them. * ⛲  Configurations set in this way **cascade**, the **more specific scope** is the one that will be set. * 👇🏻  In the example below, all the models in the `marketing` and `paid_ads` folders would be views, but the `google` sub folder would be **tables.** ```yaml models: jaffle_shop: marketing: +materialized: view paid_ads: google: +materialized: table ``` ##### Staging views We’ll start off simple with staging models. Lets consider some aspects of staging models to determine the ideal materialization strategy: * 🙅‍♀️ Staging models are **rarely accessed** directly by our **end users.** * 🧱 They need to be always up-to-date and in sync with our source data as a **building blocks** for later models * 🔍  It’s clear we’ll want to keep our **staging models as views**. * 👍  Since views are the **default materialization** in dbt, we don’t *have* to do any specific configuration for this. * 💎  Still, for clarity, it’s a **good idea** to go ahead and **specify the configuration** to be explicit. We’ll want to make sure our `dbt_project.yml` looks like this: ```yaml models: jaffle_shop: staging: +materialized: view ``` ##### Intermediate models in larger projects The [Jaffle Shop](https://github.com/dbt-labs/jaffle-shop) example project uses a staging → marts flow and does not include an `intermediate/` folder. In larger projects, intermediate models often sit between staging and marts, breaking up complex transformations into manageable pieces: * 🚫 Intermediate models are not accessed directly by end users. They exist to simplify mart logic. * 🧩 They serve as building blocks that get referenced by marts or other intermediate models. * 👻 This makes them ideal candidates for ephemeral materialization, which doesn't create objects in your warehouse. Ephemeral models are interpolated as CTE into the models that reference them. This keeps your warehouse clean and avoids cluttering it with models that aren't meant for direct querying: ```yaml models: jaffle_shop: staging: +materialized: view intermediate: +materialized: ephemeral marts: +materialized: table ``` When to avoid ephemeral models Ephemeral models can make troubleshooting more difficult since they don't exist as queryable objects. If you need to inspect intermediate results during development, consider materializing them as views in a custom schema with restricted permissions instead. This gives you visibility while keeping them separate from production models. For more details on intermediate model patterns, refer to [How we structure our dbt projects: Intermediate](https://docs.getdbt.com/best-practices/how-we-structure/3-intermediate.md). ##### Table and incremental marts As we’ve learned, views store only the logic of the transformation in the warehouse, so our runs take only a couple seconds per model (or less). What happens when we go to query the data though? ![Long query time from Snowflake](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAXsAAABKCAMAAABtsQSFAAACAVBMVEX///+i0dzo6OhKSkoAu+b50pFNTU1QUFBXV1f+/v79/f2bm5vy8vJkZGSysrLNzc37+/torlLX19f29vZVVVXHx8d6enqIiIjx8fHv7+9aWlpTU1Pd3d1fX1/i4uLj4+OhoaGLi4uYmJhmZmbPz89wcHD09PS8vLzs7Oz39/fc3Nzq6urm5ubg4ODR0dHe3t5ycnJ3d3egzJJcXFx/f3+oqKiXl5dubm7IyMienp7u7u6dnZ24uLiQkJDT09NhYWHz8/OSkpKtra1eXl6rq6ucnJy0tLTFxcVxcXH6+vr4+PhpaWmwsLDBwcF+fn5qamr8/PyDg4PU1NTp6emjo6OMjIx4eHjt7e18fHyvr6++vr7Jycna2trLy8vj8N7k5OR1dXWAgIBtbW3KysrS0tLr6+vDw8Pf39+mpqanp6fV1dWNjY25ubmqqqr19fVra2uBgYHCwsLZ2dmWlpaampq2tra3t7eHh4ezs7N9fX21tbXGxsZ0dHTl5eW/v7+GhoaioqLh4eH5+fmurq6lpaWZmZmxsbGfn59xs1zAwMCUlJSkpKS6urqKioqgoKCpqanW1tb97NFlZWX73a32+vS9vb3Y7PHQ0NDG4b5xs13//Pbw8PC53eX/+/aFhYWNwn3Ozs73+/yEvXOVlZWsrKy7u7vY2Njn5+fs9ent+v3MzMwpLeKTAAAHzklEQVR42u3bZ1ObZxaA4Xs5m7eoISOhaoHoIMCAqaY30zsYDDY27r13x71u4pbNbjabzfa+/pWrBkYUj3E8ZBKe64tGOo800j2aV/py+N/GoHwyqv2P6K3yY0FRFEVRFEVRFEX5CbOhxG03DGO0u5E11Rplzvxs1mFGn1q/a4ZFpRUsMxGd9jlYrUSuocRkSOiVXQpYU65kNcoe1mGRoq/m6rtsJJ2qYpl0LaO1SbvNam8LURLtd8O0pBFp6btaiOnweBwWnt4Hx7QrV8orxf2Ksj0FV7av0X47mIEOnKf6Kof5vFqfLmR4+vCUmWgPtCzQ7biRQ3lLQVMJHZ9Dxd4x7wnMoapbrUx4oacFuoe2cPsqKS/XtEqxjzlk1CO15OngFV+0vUeMFjOo3XAHXWu2py6PK27HE6ko+7O9dyxTwjeDs0vtc6apkY63pXrLUPUIuQXw5R30HVwN3MyRihmZpE18uOe2avvqmmoJ0CLl/bWy/0vpcR6pWGyfvOa0iz0tbX6d9g0arhlcxu34Nccsw3loPN5e0vd1aEXUjMDUXsiQkixpNIP3ou0tUg6VddgdfskrypLCrdpebxPNx6jE9JQbIvUnVrTniog7bK7dvslOpEoXmY23t3kN0fYm2odCe4ugZheMiKZpUsHXnUe1/mj7h/H7HsLTZytP3j9Xv4Wv94fkODWSkV3+sB1X+lMJcUFKqEy2zwHLwWa79K7dvmsP1d4yczDR/ow94rQm2msAifYdC7Yok9nBPTlE2xfKRPR+CRXarc6KYEHnFm5fJgHLWfEcr5TyHLl+SfIIy0KLxNsPaIHeMilIH5Ejq9vn+0+Pu0sxas18mcVR7cPrcU4aq9ofkQanY9CJTbSJWHvyRo75B+fALXcJStoWbk+L3CSsi+HAXy9SfZnSPrEfElu0PVOaxmO3SIdldXuRgPcgFIkYgSFmqiWtIija4K2V7Xmji/E54DGIty/OE/E4ocUAbxeKjZhHhYkbFjldgN/kPZyliXEhmKVO1mD6SeV6hKIoiqIoivIz9csPh/KppMVtoH2a8omgKIqiKIqiKIqiKIqiKIqiKIqivMfO37PcvUrtwhDvN9GWsvmTasegVl0L7D+kd9UNwJszbJS5MEpMf47uzjUxYsZJ9Sd+sL+xDsuUkU9Mvl0bjYBvRDeeAP472uHedwfKL0zDt0ZUMx9gZnX6bduWxx+WB62dWi3vla6lbP6kOCa7I0VyjWz9fnpRV5WJ1cNGOcRNzFUjv0EfYn9GRobdS4pff7bkFx/td6ypxB6STACXPI14dZPKyhPD0gv28RPnNP/igbOa3Q6Otui7a+QDNLPktweS6bftZImpOYC34sN7Ap7mY9tVkDNJ8UhR/eXE7k5/ov3S5s9K93SgYR+eGqBYv/QR7We0q4n2d+rgohcgImVkXFzoNEn4w28+Rfs/sqbCXU493v6yuLgrxXSWwuhrvhIXXIssHnBEHHbIvUiSbba19i7z3T3lZPXAX5uhYZ6Dz7qPEHfyXfq/f3dgVXqyxQUQ3Ie+A/Kaxy70XeszSrLFPtQY393pY1l76vJYqV06spMvARyq+4j2t+6nJ9oPuTMvacMA0zcolLqXhoOkz354+yjWk2g/Fuw4Pd4FmMXX5TSzoSb7yP5lB2Lta9q+LjhLTLF1ar79m+fzc9Zsm/V7HljPs6t1wPrN0efpxDSx6C/btn13YGV6jruJqa9Ntj8ddNKvNWRLFjyM7e4UpbRv0Fhle5UYtWa7pAGEFzbePlO3JdtPBkRCPmC/lFEmrdjGNrU9J0WkB7CJjFu4KpXDXq00tf3JnONTMpdofwyLNQuaHUy9LbS+aD1mHWiMPuCyANERiw78Ktp9ZXoqpB8g8EWy/VB8TaczWwDs0d2dRyntm+yswdajP0DPB8g5s+H2LvdLku3tF9v9BR6g6gawW4x/lGxq+0wt3zksEWAsP1BHWHdCyJHSPq6uKtEejlkt0Brm0j93nOu9mVkL96zP/9VPTBMp8VemxyKXqPPEvmbuXjCae/XvbTabM9G+c3DPK1Lad+1hpYl0IHyYvhaOh+66b2+4/UtxG7oY7eCSDDiuwWUpAzjfW31mU9tfqQRC3a7hEjgZoigAHD6V2n5fKXSPLrYfsLZDwznSdj8+MtlUuw8Y+Pfra8S8IDX+Tlbo1ooO6nIGbhRkd8p1v/bElVF9MNHeJ9pEsv27zZ+VvtVazRPVdZyWcFq9tDmx9vl8PhcfzBf93zCrZ5iOIwQOlWYX1MMdLzDR1ViycHEzfmuTab+wFcn8+Tk56tTO+CMhLzbtjeusRKKDZe33Hi59aDxYbM+TZwPFTftgl3WGJqufrHB7f/czYnpIib+TVU4GRJMG2N8mnraz5BuivSbRniqDZPulzZ/VmgyRmhLIzBPRroBVog6zIeluaPNSERIZneSypAGMi3RlbcZ/TEBvwC+TZq4m+jnYERDxFMJ8UPTrsUHsQLJ96aCI17LU3vbCursZaH4OPWHgmdUaPkbMf1juv6zFZt4ucAJO4nwmSeZgmA9RPEbceWek7SAfywLgGuCd/nY20dEgYPoWPwpx/sQghcvJcgMWUjhL+DDyHq0XgoWyUg6p5mRt2/mpeZyz/mDTbb/uZwuxudYfKIqiKIqiKIqiKIqiKIqi/Dz9H3h3hQ8qA29FAAAAAElFTkSuQmCC) Our marts are slow to query! Let’s contrast the same aspects of marts that we considered for staging models to assess the best materialization strategy: * 📊  Marts are **frequently accessed directly by our end users**, and need to be **performant.** * ⌛  Can often **function with intermittently refreshed data**, end user decision making in many domains is **fine with hourly or daily data.** * 🛠️  Given the above properties we’ve got a great use case for **building the data itself** into the warehouse, not the logic. In other words, **a table**. * ❓ The only decision we need to make with our marts is whether we can **process the whole table at once or do we need to do it in chunks**, that is, are we going to use the `table` materialization or `incremental`. info 🔑 **Golden Rule of Materializations** Start with models as views, when they take too long to query, make them tables, when the tables take too long to build, make them incremental. --- ### Best practices for workflows This page contains the collective wisdom of experienced users of dbt on how to best use it in your analytics work. Observing these best practices will help your analytics team work as effectively as possible, while implementing the pro-tips will add some polish to your dbt projects! #### Best practice workflows ##### Version control your dbt project All dbt projects should be managed in version control. Git branches should be created to manage development of new features and bug fixes. All code changes should be reviewed by a colleague (or yourself) in a Pull Request prior to merging into your production branch, such as `main`. Git guide We've codified our best practices in Git, in our [Git guide](https://github.com/dbt-labs/corp/blob/main/git-guide.md). ##### Use separate development and production environments dbt makes it easy to maintain separate production and development environments through the use of targets within a profile. We recommend using a `dev` target when running dbt from your command line and only running against a `prod` target when running from a production deployment. You can read more [about managing environments here](https://docs.getdbt.com/docs/environments-in-dbt.md). ##### Use a style guide for your project SQL styles, field naming conventions, and other rules for your dbt project should be codified, especially on projects where multiple dbt users are writing code. Our style guide We've made our [style guide](https://docs.getdbt.com/best-practices/how-we-style/0-how-we-style-our-dbt-projects.md) public – these can act as a good starting point for your own style guide. #### Best practices in dbt projects ##### Use the ref function The [ref](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) function is what makes dbt so powerful! Using the `ref` function allows dbt to infer dependencies, ensuring that models are built in the correct order. It also ensures that your current model selects from upstream tables and views in the same environment that you're working in. Always use the `ref` function when selecting from another model, rather than using the direct relation reference (e.g. `my_schema.my_table`). ##### Limit references to raw data Your dbt project will depend on raw data stored in your database. Since this data is normally loaded by third parties, the structure of it can change over time – tables and columns may be added, removed, or renamed. When this happens, it is easier to update models if raw data is only referenced in one place. Using sources for raw data references We recommend defining your raw data as [sources](https://docs.getdbt.com/docs/build/sources.md), and selecting from the source rather than using the direct relation reference. Our dbt projects don't contain any direct relation references in any models. ##### Rename and recast fields once Raw data is generally stored in a source-conformed structure, that is, following the schema and naming conventions that the source defines. Not only will this structure differ between different sources, it is also likely to differ from the naming conventions you wish to use for analytics. The first layer of transformations in a dbt project should: * Select from only one source * Rename fields and tables to fit the conventions you wish to use within your project, for example, ensuring all timestamps are named `_at`. These conventions should be declared in your project coding conventions (see above). * Recast fields into the correct data type, for example, changing dates into UTC and prices into dollar amounts. All subsequent data models should be built on top of these models, reducing the amount of duplicated code. What happened to base models? Earlier versions of this documentation recommended implementing “base models” as the first layer of transformation, and gave advice on the SQL within these models. We realized that while the reasons behind this convention were valid, the specific advice around "base models" represented an opinion, so we moved it out of the official documentation. You can instead find our opinions on [how we structure our dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md). ##### Break complex models up into smaller pieces Complex models often include multiple Common Table Expressions (CTEs). In dbt, you can instead separate these CTEs into separate models that build on top of each other. It is often a good idea to break up complex models when: * A CTE is duplicated across two models. Breaking the CTE into a separate model allows you to reference the model from any number of downstream models, reducing duplicated code. * A CTE changes the grain of a the data it selects from. It's often useful to test any transformations that change the grain (as in, what one record represents) of your data. Breaking a CTE into a separate model allows you to test this transformation independently of a larger model. * The SQL in a query contains many lines. Breaking CTEs into separate models can reduce the cognitive load when another dbt user (or your future self) is looking at the code. ##### Group your models in directories Within your `models/` directory, you can have any number of nested subdirectories. We leverage directories heavily, since using a nested structure within directories makes it easier to: * Configure groups of models, by specifying configurations in your `dbt_project.yml` file. * Run subsections of your DAG, by using the [model selection syntax](https://docs.getdbt.com/reference/node-selection/syntax.md). * Communicate modeling steps to collaborators * Create conventions around the allowed upstream dependencies of a model, for example, "models in the `marts` directory can only select from other models in the `marts` directory, or from models in the `staging` directory". ##### Add tests to your models dbt provides a framework to test assumptions about the results generated by a model. Adding tests to a project helps provide assurance that both: * your SQL is transforming data in the way you expect, and * your source data contains the values you expect Recommended tests Our [style guide](https://github.com/dbt-labs/corp/blob/main/dbt_style_guide.md) recommends that at a minimum, every model should have a primary key that is tested to ensure it is unique, and not null. ##### Consider the information architecture of your data warehouse When a user connects to a data warehouse via a SQL client, they often rely on the names of schemas, relations, and columns, to understand the data they are presented with. To improve the information architecture of a data warehouse, we: * Use [custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md) to separate relations into logical groupings, or hide intermediate models in a separate schema. Generally, these custom schemas align with the directories we use to group our models, and are configured from the `dbt_project.yml` file. * Use prefixes in table names (for example, `stg_`, `fct_` and `dim_`) to indicate which relations should be queried by end users. ##### Choose your materializations wisely [materialization](https://docs.getdbt.com/docs/build/materializations.md) determine the way models are built through configuration. As a general rule: * Views are faster to build, but slower to query compared to tables. * Incremental models provide the same query performance as tables, are faster to build compared to the table materialization, however they introduce complexity into a project. We often: * Use views by default * Use ephemeral models for lightweight transformations that shouldn't be exposed to end-users * Use tables for models that are queried by BI tools * Use tables for models that have multiple descendants * Use incremental models when the build time for table models exceeds an acceptable threshold #### Pro-tips for workflows ##### Use the model selection syntax when running locally When developing, it often makes sense to only run the model you are actively working on and any downstream models. You can choose which models to run by using the [model selection syntax](https://docs.getdbt.com/reference/node-selection/syntax.md). ##### Run only modified models to test changes ("slim CI") To merge code changes with confidence, you want to know that those changes will not cause breakages elsewhere in your project. For that reason, we recommend running models and tests in a sandboxed environment, separated from your production data, as an automatic check in your git workflow. (If you use GitHub and dbt, read about [how to set up CI jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md). At the same time, it costs time (and money) to run and test all the models in your project. This inefficiency feels especially painful if your PR only proposes changes to a handful of models. By comparing to artifacts from a previous production run, dbt can determine which models are modified and build them on top of of their unmodified parents. ```bash dbt run -s state:modified+ --defer --state path/to/prod/artifacts dbt test -s state:modified+ --defer --state path/to/prod/artifacts ``` By comparing to artifacts from a previous production run, dbt can determine model and test result statuses. * `result:fail` * `result:error` * `result:warn` * `result:success` * `result:skipped` * `result:pass` For smarter reruns, use the `result:` selector instead of manually overriding dbt commands with the models in scope. ```bash dbt run --select state:modified+ result:error+ --defer --state path/to/prod/artifacts ``` * Rerun all my erroneous models AND run changes I made concurrently that may relate to the erroneous models for downstream use ```bash dbt build --select state:modified+ result:error+ --defer --state path/to/prod/artifacts ``` * Rerun and retest all my erroneous models AND run changes I made concurrently that may relate to the erroneous models for downstream use ```bash dbt build --select state:modified+ result:error+ result:fail+ --defer --state path/to/prod/artifacts ``` * Rerun all my erroneous models AND all my failed tests * Rerun all my erroneous models AND run changes I made concurrently that may relate to the erroneous models for downstream use * There's a failed test that's unrelated to modified or error nodes(think: source test that needs to refresh a data load in order to pass) ```bash dbt test --select result:fail --exclude --defer --state path/to/prod/artifacts ``` * Rerun all my failed tests and exclude tests that I know will still fail * This can apply to updates in source data during the "EL" process that need to be rerun after they are refreshed > Note: If you're using the `--state target/` flag, `result:error` and `result:fail` flags can only be selected concurrently(in the same command) if using the `dbt build` command. `dbt test` will overwrite the `run_results.json` from `dbt run` in a previous command invocation. Only supported by v1.1 or newer. By comparing to a `sources.json` artifact from a previous production run to a current `sources.json` artifact, dbt can determine which sources are fresher and run downstream models based on them. ```bash # job 1 dbt source freshness # must be run to get previous state ``` Test all my sources that are fresher than the previous run, and run and test all models downstream of them: ```bash # job 2 dbt source freshness # must be run again to compare current to previous state dbt build --select source_status:fresher+ --state path/to/prod/artifacts ``` To learn more, read the docs on [state](https://docs.getdbt.com/reference/node-selection/syntax.md#about-node-selection). #### Pro-tips for dbt Projects ##### Limit the data processed when in development In a development environment, faster run times allow you to iterate your code more quickly. We frequently speed up our runs by using a pattern that limits data based on the [target](https://docs.getdbt.com/reference/dbt-jinja-functions/target.md) name: ```sql select * from event_tracking.events {% if target.name == 'dev' %} where created_at >= dateadd('day', -3, current_date) {% endif %} ``` Another option is to use the [environment variable `DBT_CLOUD_INVOCATION_CONTEXT`](https://docs.getdbt.com/docs/build/environment-variables.md#dbt-platform-context). This environment variable provides metadata about the execution context of dbt. The possible values are `prod`, `dev`, `staging`, and `ci`. **Example usage**: ```text {% if env_var('DBT_CLOUD_INVOCATION_CONTEXT') != 'prod' %} ``` ##### Use grants to manage privileges on objects that dbt creates Use `grants` in [resource configs](https://docs.getdbt.com/reference/resource-configs/grants.md) to ensure that permissions are applied to the objects created by dbt. By codifying these grant statements, you can version control and repeatably apply these permissions. ##### Separate source-centric and business-centric transformations When modeling data, we frequently find there are two stages: 1. Source-centric transformations to transform data from different sources into a consistent structure, for example, re-aliasing and recasting columns, or unioning, joining or deduplicating source data to ensure your model has the correct grain; and 2. Business-centric transformations that transform data into models that represent entities and processes relevant to your business, or implement business definitions in SQL. We find it most useful to separate these two types of transformations into different models, to make the distinction between source-centric and business-centric logic clear. ##### Managing whitespace generated by Jinja If you're using macros or other pieces of Jinja in your models, your compiled SQL (found in the `target/compiled` directory) may contain unwanted whitespace. Check out the [Jinja documentation](http://jinja.pocoo.org/docs/2.10/templates/#whitespace-control) to learn how to control generated whitespace. #### Related docs * [Updating our permissioning guidelines: grants as configs in dbt v1.2](https://docs.getdbt.com/blog/configuring-grants) --- ### Building metrics tip Note that this best practices guide doesn't yet use the [new YAML specification](https://docs.getdbt.com/docs/build/latest-metrics-spec.md). We're working on updating this guide to use the new spec and file structure soon! To read more about the new spec, see [Creating metrics](https://docs.getdbt.com/docs/build/metrics-overview.md). #### How to build metrics * 💹 We'll start with one of the most important metrics for any business: **revenue**. * 📖 For now, our metric for revenue will be **defined as the sum of order totals excluding tax**. #### Defining revenue * 🔢 Metrics have four basic properties: * `name:` We'll use 'revenue' to reference this metric. * `description:` For documentation. * `label:` The display name for the metric in downstream tools. * `type:` one of `simple`, `ratio`, or `derived`. * 🎛️ Each type has different `type_params`. * 🛠️ We'll build a **simple metric** first to get the hang of it, and move on to ratio and derived metrics later. * 📏 Simple metrics are built on a **single measure defined as a type parameter**. * 🔜 Defining **measures as their own distinct component** on semantic models is critical to allowing the **flexibility of more advanced metrics**, though simple metrics act mainly as **pass-through that provide filtering** and labeling options. models/marts/orders.yml ```yml metrics: - name: revenue description: Sum of the order total. label: Revenue type: simple type_params: measure: order_total ``` #### Query your metric You can use the dbt platform CLI for metric validation or queries during development, via the `dbt sl` set of subcommands. Here are some useful examples: ```bash dbt sl query revenue --group-by metric_time__month dbt sl list dimensions --metrics revenue # list all dimensions available for the revenue metric ``` * It's best practice any time we're updating our Semantic Layer code to run `dbt parse` to update our development semantic manifest. * `dbt sl query` is not how you would typically use the tool in production, that's handled by the dbt Semantic Layer's features. It's available for testing results of various metric queries in development, exactly as we're using it now. * Note the structure of the above query. We select the metric(s) we want and the dimensions to group them by — we use dunders (double underscores e.g.`metric_time__[time bucket]`) to designate time dimensions or other non-unique dimensions that need a specified entity path to resolve (e.g. if you have an orders location dimension and an employee location dimension both named 'location' you would need dunders to specify `orders__location` or `employee__location`). --- ### Building Semantic Layer definitions with dbt Wizard Use dbt Wizard to turn a business question or an existing dbt model into version-compatible Semantic Layer definitions. dbt Wizard inspects model grain, columns, lineage, and nearby YAML before it proposes entities, dimensions, and metrics. This workflow uses your project files and available dbt metadata to guide the work. You can also add the `building-dbt-semantic-layer` skill from the [dbt Agent Skills repository](https://github.com/dbt-labs/dbt-agent-skills) for reusable Semantic Layer guidance. The prompts on this page work the same way in dbt platform. #### Before you begin Identify the following information before you ask dbt Wizard to edit files: * The business question or metric you want to support. * The model, or set of candidate models, that contains the relevant data. * The grain of each model, when you already know it. * Any naming, certification, or ownership conventions your team follows. If you don't know which model to use, start with the business question. For example: ```text We need to report gross revenue and order count by customer segment and month. Find the best models to build on, explain your choices, and propose the Semantic Layer definitions before editing files. ``` #### Ask Wizard to plan the definitions Give dbt Wizard a specific model when you know the starting point: ```text Build Semantic Layer definitions for fct_orders. First determine the dbt version and inspect the model grain, columns, lineage, and existing YAML. Propose the entities, dimensions, and metrics before making changes. Include total revenue, order count, and a monthly time dimension. ``` dbt Wizard should complete these planning steps before editing: 1. Determine the dbt version from the installed binary, `require-dbt-version`, or manifest metadata. 2. Inspect the target model's SQL, columns, and upstream and downstream lineage. 3. Check nearby YAML and existing semantic definitions so the new content follows project conventions. 4. Identify the model grain and at least one primary entity. 5. Propose time and categorical dimensions and metrics whose aggregations are meaningful for the underlying data. Ask dbt Wizard to sample data only when names, types, and SQL don't establish the meaning of a column. Review and approve the warehouse query before it runs. #### Review the generated YAML The correct YAML structure depends on the dbt version in your project. Review the generated file for the version-specific patterns in the following table. | Project version | Expected structure | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [dbt 1.12 and later](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.12.md?version=2.0#new-semantic-layer-yaml-spec), and the [dbt v2](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md?version=2.0) | Configure `semantic_model` on a model, annotate entities and dimensions on columns, and define metrics on the model. Don't use a top-level `semantic_models:` block for new definitions. | | [dbt 1.6 through 1.11](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.11.md?version=2.0) | Define semantic models in a top-level `semantic_models:` block and define metrics with `type_params` that reference measures. | For dbt 1.12 and later and dbt v2, a generated definition can resemble the following example: ```yaml models: - name: fct_orders semantic_model: enabled: true agg_time_dimension: ordered_at columns: - name: order_id entity: type: primary name: order - name: customer_id entity: type: foreign name: customer - name: ordered_at granularity: day dimension: type: time - name: order_status dimension: type: categorical metrics: - name: total_revenue type: simple label: Total Revenue agg: sum expr: order_total - name: order_count type: simple label: Order Count agg: count_distinct expr: order_id ``` For dbt 1.6 through 1.11, expect the top-level semantic model pattern: ```yaml semantic_models: - name: orders model: ref('fct_orders') defaults: agg_time_dimension: ordered_at entities: - name: order type: primary expr: order_id dimensions: - name: ordered_at type: time type_params: time_granularity: day measures: - name: total_revenue agg: sum expr: order_total metrics: - name: total_revenue type: simple label: Total Revenue type_params: measure: total_revenue ``` Confirm that every generated definition has a clear business meaning. A numeric column isn't automatically a useful metric, and an ID isn't automatically the correct primary entity. #### Validate the definitions Ask dbt Wizard to validate after you approve the YAML: ```text Validate the Semantic Layer changes. Parse the project, run the supported Semantic Layer validation for this environment, and report errors, warnings, and any checks you couldn't complete. Don't change the definitions to silence an error without explaining the root cause. ``` At minimum, validation should confirm that: * The project parses successfully. * Model, column, entity, measure, and metric references resolve. * The aggregate time dimension points to a declared time dimension. * The selected metric aggregation matches the model grain. * Semantic Layer validation passes when the project's dbt runtime supports it. When a validation command isn't available in the current environment, dbt Wizard should report that limitation instead of treating the work as fully validated. For a broader validation procedure, refer to [Validating dbt changes with dbt Wizard](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-3-validate-changes.md). #### Extend the first definitions After the initial definitions validate, continue with focused prompts rather than asking for an entire project conversion at once. For example: ```text Add a derived average order value metric using the existing revenue and order count metrics. Explain how the metric joins and time grain will behave before editing the YAML. ``` ```text Create a saved query for monthly revenue and order count grouped by customer segment. Reuse existing entities and dimensions, and validate every reference. ``` #### Related docs * [Semantic models](https://docs.getdbt.com/docs/build/semantic-models.md) * [Metrics overview](https://docs.getdbt.com/docs/build/metrics-overview.md) * [Use cases and examples](https://docs.getdbt.com/docs/dbt-ai/wizard-use-cases.md) * [Use skills with dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-skills.md) * [Validating dbt changes with dbt Wizard](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-3-validate-changes.md) --- ### Building semantic models tip Note that this best practices guide doesn't yet use the [new YAML specification](https://docs.getdbt.com/docs/build/latest-metrics-spec.md). We're working on updating this guide to use the new spec and file structure soon! To read more about the new spec, see [Creating metrics](https://docs.getdbt.com/docs/build/metrics-overview.md). #### How to build a semantic model A semantic model is the Semantic Layer equivalent to a logical layer model (what historically has just been called a 'model' in dbt land). Just as configurations for models are defined on the `models:` YAML key, configurations for semantic models are housed under `semantic models:`. A key difference is that while a logical model consists of configuration and SQL or Python code, a **semantic model is defined purely via YAML**. Rather than encoding a specific dataset, a **semantic model describes relationships and expressions** that let your end users select and refine their own datasets dynamically and reliably. * ⚙️ Semantic models are **comprised of three components**: * 🫂 **entities**: these describe the **relationships** between various semantic models (think ids) * 🔪 **dimensions**: these are the columns you want to **slice, dice, group, and filter by** (think timestamps, categories, booleans). * 📏 **measures**: these are the **quantitative values you want to aggregate** * 🪣 We define **columns as being an entity, dimension, or measure**. Columns will typically fit into one of these 3 buckets, or if they're a complex aggregation expression, they might constitute a metric. #### Defining orders Let's zoom in on how we might define an *orders* semantic model. * 📗 We define it as a **YAML dictionary in the `semantic_models` list**. * 📑 It will have a **name, entities list, dimensions list, and measures list**. * ⏬ We recommend defining them **in this order consistently** as a style best practice. models/marts/orders.yml ```yaml semantic_models: - name: orders entities: ... # we'll define these later dimensions: ... # we'll define these later measures: ... # we'll define these later ``` * Next we'll point to the corresponding logical model by supplying a [`ref`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) in the `model:` property, and a `description` for documentation. models/marts/orders.yml ```yml semantic_models: - name: orders description: | Model containing order data. The grain of the table is the order id. model: ref('stg_orders') entities: ... dimensions: ... measures: ... ``` #### Establishing our entities * 🫂 Entities are the **objects and concepts** in our data that *have* dimensions and measures. You can think of them as the **nouns** of our project, the **spines** of our queries that we may want to aggregate by, or simply the **join keys**. * 🔀 Entities help MetricFlow understand **how various semantic models relate to one another**. * ⛓️ Unlike many other semantic layers, in MetricFlow **we do not need to describe joins explicitly**, instead the **relationships are implicitly described by entities**. * 1️⃣ Each semantic model should have **one primary entity** defined for itself, and **any number of foreign entities** for other semantic models it may join to. * 🫂 Entities require a **name and type** * 🔑 Types available are **primary**, **foreign**, **unique** or **natural** — we'll be focused on the first two for now, but you can [read more about unique and natural keys](https://docs.getdbt.com/docs/build/entities.md#entity-types). ##### Entities in action If we look at an example staging model for orders, we see that it has 3 id columns, so we'll need three entities. models/staging/stg\_orders.sql ```sql renamed as ( select ---------- ids id as order_id, store_id as location_id, customer as customer_id, ---------- properties (order_total / 100.0) as order_total, (tax_paid / 100.0) as tax_paid, ---------- timestamps ordered_at from source ``` * 👉 We add them with a **`name`, `type`, and optional `expr`** (expression). The expression can be any valid SQL expression on your platform. * 📛 If you **don't add an expression**, MetricFlow will **assume the name is equal to the column name** in the underlying logical model. * 👍 Our best practices pattern is to, whenever possible, provide a `name` that is the singular form of the subject or grain of the table, and use `expr` to specify the precise column name (with `_id` etc). This will let us write **more readable metrics** on top of these semantic models. For example, we'll use `location` instead of `location_id`. models/marts/orders.yml ```yml semantic_models: - name: orders ... entities: # we use the column for the name here because order is a reserved word in SQL - name: order_id type: primary - name: location type: foreign expr: location_id - name: customer type: foreign expr: customer_id dimensions: ... measures: ... ``` #### Defining our dimensions * 🧮 Dimensions are the columns that we want to **filter and group by**, **the adjectives of our project**. They come in three types: * **categorical** * **time** * slowly changing dimensions — [these are covered in the documentation](https://docs.getdbt.com/docs/build/dimensions.md#scd-type-ii), and a little more complex. To focus on building your mental models of MetricFlow's fundamentals, we won't be using SCDs in this guide. * ➕ We're **not limited to existing columns**, we can use the `expr` property to add simple computations in our dimensions. * 📛 Categorical dimensions are the simplest, they simply require a `name` and `type` (type being categorical). **If the `name` property matches the name of the dimension column**, that's it, you're done. If you want or need to use a `name` other than the column name, or do some filtering or computation, **you can supply an optional `expr` property** to evaluate for the dimension. ##### Dimensions in action * 👀 Let's look at our staging model again and see what fields we have available. models/staging/stg\_orders.sql ```sql select ---------- ids -> entities id as order_id, store_id as location_id, customer as customer_id, ---------- numerics -> measures (order_total / 100.0) as order_total, (tax_paid / 100.0) as tax_paid, ---------- timestamps -> dimensions ordered_at from source ``` * ⏰ For now the only dimension to add is a **time dimension**: `ordered_at`. * 🕰️ At least one **primary time dimension** is **required** for any semantic models that **have measures**. * 1️⃣ We denote this with the `is_primary` property, or if there is only a one-time dimension supplied it is primary by default. Below we only have `ordered_at` as a timestamp so we don't need to specify anything except the *minimum granularity* we're bucketing to (in this case, day). By this we mean that we're not going to be looking at orders at a finer granularity than a day. models/marts/orders.yml ```yml dimensions: - name: ordered_at expr: date_trunc('day', ordered_at) type: time type_params: time_granularity: day ``` tip **Dimensional models**. You may have some models that do not contain measures, just dimensional data that enriches other facts. That's totally fine, a semantic model does not require dimensions or measures, it just needs a primary entity, and if you do have measures, a primary time dimension. We'll discuss an alternate situation, dimensional tables that have static numeric values like supply costs or tax rates but no time dimensions, later in the Guide. * 🔢 We can also **make a dimension out of a numeric column** that would typically be a measure. * 🪣 Using `expr` we can **create buckets of values that we label** for our dimension. We'll add one of these in for labeling 'large orders' as any order totals over $50. models/marts/orders.yml ```yml dimensions: - name: ordered_at expr: date_trunc('day', ordered_at) type: time type_params: time_granularity: day - name: is_large_order type: categorical expr: case when order_total > 50 then true else false end ``` #### Making our measures * 📏 Measures are the final component of a semantic model. They describe the **numeric values that we want to aggregate**. * 🧱 Measures form **the building blocks of metrics**, with entities and dimensions helping us combine, group, and filter those metrics correctly. * 🏃 You can think of them as something like the **verbs of a semantic model**. ##### Measures in action * 👀 Let's look at **our staging model** one last time and see what **fields we want to measure**. models/staging/stg\_orders.sql ```sql select ---------- ids -> entities id as order_id, store_id as location_id, customer as customer_id, ---------- numerics -> measures (order_total / 100.0) as order_total, (tax_paid / 100.0) as tax_paid, ---------- timestamps -> dimensions ordered_at from source ``` * ➕ Here `order_total` and `tax paid` are the **columns we want as measures**. * 📝 We can describe them via the code below, specifying a **name, description, aggregation, and expression**. * 👍 As before MetricFlow will default to the **name being the name of a column when no expression is supplied**. * 🧮 [Many different aggregations](https://docs.getdbt.com/docs/build/measures.md#aggregation) are available to us. Here we just want sums. models/marts/orders.yml ```yml measures: - name: order_total description: The total amount for each order including taxes. agg: sum - name: tax_paid description: The total tax paid on each order. agg: sum ``` * 🆕 We can also **create new measures using expressions**, for instance adding a count of individual orders as below. models/marts/orders.yml ```yml - name: order_count description: The count of individual orders. expr: 1 agg: sum ``` #### Reviewing our work Our completed code will look like this, our first semantic model! Here are two examples showing different organizational approaches:  Co-located approach models/marts/orders.yml ```yml semantic_models: - name: orders defaults: agg_time_dimension: ordered_at description: | Order fact table. This table is at the order grain with one row per order. model: ref('stg_orders') entities: - name: order_id type: primary - name: location type: foreign expr: location_id - name: customer type: foreign expr: customer_id dimensions: - name: ordered_at expr: date_trunc('day', ordered_at) # use date_trunc(ordered_at, DAY) if using BigQuery type: time type_params: time_granularity: day - name: is_large_order type: categorical expr: case when order_total > 50 then true else false end measures: - name: order_total description: The total revenue for each order. agg: sum - name: order_count description: The count of individual orders. expr: 1 agg: sum - name: tax_paid description: The total tax paid on each order. agg: sum ```  Parallel sub-folder approach models/semantic\_models/sem\_orders.yml ```yml semantic_models: - name: orders defaults: agg_time_dimension: ordered_at description: | Order fact table. This table is at the order grain with one row per order. model: ref('stg_orders') entities: - name: order_id type: primary - name: location type: foreign expr: location_id - name: customer type: foreign expr: customer_id dimensions: - name: ordered_at expr: date_trunc('day', ordered_at) # use date_trunc(ordered_at, DAY) if using BigQuery type: time type_params: time_granularity: day - name: is_large_order type: categorical expr: case when order_total > 50 then true else false end measures: - name: order_total description: The total revenue for each order. agg: sum - name: order_count description: The count of individual orders. expr: 1 agg: sum - name: tax_paid description: The total tax paid on each order. agg: sum ``` As you can see, the content of the semantic model is identical in both approaches. The key differences are: 1. **File location** * Co-located approach: `models/marts/orders.yml` * Parallel sub-folder approach: `models/semantic_models/sem_orders.yml` 2. **File naming** * Co-located approach: Uses the same name as the corresponding mart (`orders.yml`) * Parallel sub-folder approach: Prefixes the file with `sem_` (`sem_orders.yml`) Choose the approach that best fits your project structure and team preferences. The co-located approach is often simpler for new projects, while the parallel sub-folder approach can be clearer for migrating large existing projects to the Semantic Layer. #### Next steps Let's review the basics of semantic models: * 🧱 Consist of **entities, dimensions, and measures**. * 🫂 Describe the **semantics and relationships of objects** in the warehouse. * 1️⃣ Correspond to a **single logical model** in your dbt project. Next up, let's use our new semantic model to **build a metric**! --- ### Clone incremental models as the first step of your CI job Before you begin, you must be aware of a few conditions: * `dbt clone` is only available with dbt version 1.6 and newer. Refer to our [upgrade guide](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) for help enabling newer versions in dbt. * This strategy only works for warehouse that support zero copy cloning (otherwise `dbt clone` will just create pointer views). * Some teams may want to test that their incremental models run in both incremental mode and full-refresh mode. Imagine you've created a [Slim CI job](https://docs.getdbt.com/docs/deploy/continuous-integration.md) in dbt and it is configured to: * Defer to your production environment. * Run the command `dbt build --select state:modified+` to run and test all of the models you've modified and their downstream dependencies. * Trigger whenever a developer on your team opens a PR against the main branch. [![Example of a slim CI job with the above configurations](/img/best-practices/slim-ci-job.png?v=2 "Example of a slim CI job with the above configurations")](#)Example of a slim CI job with the above configurations Now imagine your dbt project looks something like this in the DAG: [![Sample project DAG](/img/best-practices/dag-example.png?v=2 "Sample project DAG")](#)Sample project DAG When you open a pull request (PR) that modifies `dim_wizards`, your CI job will kickoff and build *only the modified models and their downstream dependencies* (in this case, `dim_wizards` and `fct_orders`) into a temporary schema that's unique to your PR. This build mimics the behavior of what will happen once the PR is merged into the main branch. It ensures you're not introducing breaking changes, without needing to build your entire dbt project. #### What happens when one of the modified models (or one of their downstream dependencies) is an incremental model? Because your CI job is building modified models into a PR-specific schema, on the first execution of `dbt build --select state:modified+`, the modified incremental model will be built in its entirety *because it does not yet exist in the PR-specific schema* and [is\_incremental will be false](https://docs.getdbt.com/docs/build/incremental-models.md#understand-the-is_incremental-macro). You're running in `full-refresh` mode. This can be suboptimal because: * Typically incremental models are your largest datasets, so they take a long time to build in their entirety which can slow down development time and incur high warehouse costs. * There are situations where a `full-refresh` of the incremental model passes successfully in your CI job but an *incremental* build of that same table in prod would fail when the PR is merged into main (think schema drift where [on\_schema\_change](https://docs.getdbt.com/docs/build/incremental-models.md#what-if-the-columns-of-my-incremental-model-change) config is set to `fail`) You can alleviate these problems by zero copy cloning the relevant, pre-existing incremental models into your PR-specific schema as the first step of the CI job using the `dbt clone` command. This way, the incremental models already exist in the PR-specific schema when you first execute the command `dbt build --select state:modified+` so the `is_incremental` flag will be `true`. You'll have two commands for your dbt CI check to execute: 1. Clone all of the pre-existing incremental models that have been modified or are downstream of another model that has been modified: ```shell dbt clone --select state:modified+,config.materialized:incremental,state:old ``` 2. Build all of the models that have been modified and their downstream dependencies: ```shell dbt build --select state:modified+ ``` Because of your first clone step, the incremental models selected in your `dbt build` on the second step will run in incremental mode. [![Clone command in the CI config](/img/best-practices/clone-command.png?v=2 "Clone command in the CI config")](#)Clone command in the CI config Your CI jobs will run faster, and you're more accurately mimicking the behavior of what will happen once the PR has been merged into main. ##### Expansion on "think schema drift" where [on\_schema\_change](https://docs.getdbt.com/docs/build/incremental-models.md#what-if-the-columns-of-my-incremental-model-change) config is set to `fail`" from above Imagine you have an incremental model `my_incremental_model` with the following config: ```sql {{ config( materialized='incremental', unique_key='unique_id', on_schema_change='fail' ) }} ``` Now, let’s say you open up a PR that adds a new column to `my_incremental_model`. In this case: * An incremental build will fail. * A `full-refresh` will succeed. If you have a daily production job that just executes `dbt build` without a `--full-refresh` flag, once the PR is merged into main and the job kicks off, you will get a failure. So the question is - what do you want to happen in CI? * Do you want to also get a failure in CI, so that you know that once this PR is merged into main you need to immediately execute a `dbt build --full-refresh --select my_incremental_model` in production in order to avoid a failure in prod? This will block your CI check from passing. * Do you want your CI check to succeed, because once you do run a `full-refresh` for this model in prod you will be in a successful state? This may lead unpleasant surprises if your production job is suddenly failing when you merge this PR into main if you don’t remember you need to execute a `dbt build --full-refresh --select my_incremental_model` in production. There’s probably no perfect solution here; it’s all just tradeoffs! Our preference would be to have the failing CI job and have to manually override the blocking branch protection rule so that there are no surprises and we can proactively run the appropriate command in production once the PR is merged. ##### Expansion on "why `state:old`" For brand new incremental models, you want them to run in `full-refresh` mode in CI, because they will run in `full-refresh` mode in production when the PR is merged into `main`. They also don't exist yet in the production environment... they're brand new! If you don't specify this, you won't get an error just a “No relation found in state manifest for…”. So, it technically works without specifying `state:old` but adding `state:old` is more explicit and means it won't even try to clone the brand new incremental models. --- ### Conclusion You're now following best practices in your project, and have optimized the materializations of your DAG. You’re equipped with the 3 main materializations that cover almost any analytics engineering situation! There are more configs and materializations available, as well as specific materializations for certain platforms and adapters — and like everything with dbt, materializations are extensible, meaning you can create your own [custom materializations](https://docs.getdbt.com/guides/create-new-materializations.md) for your needs. So this is just the beginning of what you can do with these powerful configurations. For the vast majority of users and companies though, tables, views, and incremental models will handle everything you can throw at them. Develop your intuition and expertise for these materializations, and you’ll be well on your way to tackling advanced analytics engineering problems. --- ### Configuring materializations #### Configuring materializations Choosing which materialization is as simple as setting any other configuration in dbt. We’ll look first at how we select our materializations for individual models, then at more powerful ways of setting materializations for entire folders of models. ##### Configuring tables and views Let’s look at how we can use tables and views to get started with materializations: * ⚙️ We can configure an individual model’s materialization using a **Jinja `config` block**, and passing in the **`materialized` argument**. This tells dbt what materialization to use. * 🚰 The underlying specifics of what is run depends on [which **adapter** you’re using](https://docs.getdbt.com/docs/supported-data-platforms.md), but the end results will be equivalent. * 😌 This is one of the many valuable aspects of dbt: it lets us use a **declarative** approach, specifying the *outcome* that we want in our code, rather than *specific steps* to achieve it (the latter is an *imperative* approach if you want to get computer science-y about it 🤓). * 🔍 In the below case, we want to create a SQL **view**, and can **declare** that in a **single line of code**. Note that python models [do not support materializing as views](https://docs.getdbt.com/docs/build/materializations.md#python-materializations) at this time. ```sql {{ config( materialized='view' ) }} select ... ``` info 🐍 **Not all adapters support python yet**, check the [docs here to be sure](https://docs.getdbt.com/docs/build/python-models.md#specific-data-platforms) before spending time writing python models. * Configuring a model to materialize as a `table` is simple, and possible for both SQL and python models. ##### SQL ```sql {{ config( materialized='table' ) }} select ... ``` ##### Python ```python def model(dbt, session): dbt.config(materialized="table") # model logic return model_df ``` Go ahead and try some of these out! --- ### Coordinating model versions Coordinating model versions across your mesh is a critical part of the model versioning process. This guide will walk you through the safe best practices for coordinating producers and consumers when introducing model versions. [Model versions](https://docs.getdbt.com/docs/mesh/govern/model-versions.md) are an important part of your dbt Mesh workflow. They enable better data model management and are critical when multiple teams share models across projects. Releasing a new model version safely requires coordination between model producers (who build the models) and model consumers (who depend on them). This guide goes over the following topics: * [How producers introduce new model versions safely](#best-practices-for-producers) * [How consumers evaluate and migrate to those new versions](#best-practices-for-consumers) For how versioning works at a technical level (YAML structure, contracts, aliasing), see [model versions](https://docs.getdbt.com/docs/mesh/govern/model-versions.md). #### Best practices for producers Producers own the creation, rollout, communication, and deprecation of model versions. The following steps go over what producers should do when introducing a new version of a model. * [Step 1: Decide when a change needs a new version](#step-1-decide-when-a-change-needs-a-new-version) * [Step 2: Create the new version safely](#step-2-create-the-new-version-safely) * [Step 3: Add a deprecation date](#step-3-add-a-deprecation-date) * [Step 4: Communicate the new version](#step-4-communicate-the-new-version) * [Step 5: Set the new latest version](#step-5-set-the-new-latest-version) * [Step 6: Clean up deprecated versions](#step-6-clean-up-deprecated-versions) ###### Step 1: Decide when a change needs a new version When creating an original version of a model, use [model contracts](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md) to ensure that breaking changes produce errors during development. The model contract ensures you, as a producer, are not changing the shape or data type of the output model. If a change breaks the contract, like removing or changing a column type, create a new model version with an updated contract instead of changing the existing version in place. Here are some examples of breaking changes that might need a new version: * Removing a column * Renaming a column * Changing a column type Here are some examples of non-breaking changes: * Adding a new column * Fixing a bug in an existing column Here are examples of changes that might be breaking depending on your business logic: * Changing logic behind a metric * Changing granularity * Modifying filters * Rewriting `CASE` statements ###### Step 2: Create the new version safely After deciding that a change needs a new [version](https://docs.getdbt.com/reference/resource-properties/versions.md), follow these steps to create the new version without disrupting existing workflows. Let's say you're removing a column: 1. Create a new version of the model file. For example, `fishtown_analytics_orders_v2.sql`. Each version of a model must have its own SQL file. 2. Keep the default version stable. In the model's `properties.yml` file: * Set [`versions`](https://docs.getdbt.com/reference/resource-properties/versions.md) to include the old version and the new version: `- v: 1` and `- v: 2` respectively. * Set the [`latest_version:`](https://docs.getdbt.com/reference/resource-properties/latest_version.md) to `latest_version: 1`. This ensures that downstream consumers using `ref(...)` won’t accidentally start using v2. Without setting this, the default will be the highest numerical version, which could be a breaking change for consumers. 3. Set an [alias](https://docs.getdbt.com/reference/resource-configs/alias.md) or create a view over the latest model version. By aliasing or creating a view over the latest model version, you ensure `fishtown_analytics_orders` (without the version suffix) always exists as an object in the warehouse, pointing to the latest version. This also protects external tools and BI dashboards. ###### Step 3: Add a deprecation date 1. In the model's `properties.yml` file, set a [`deprecation_date`](https://docs.getdbt.com/reference/resource-properties/deprecation_date.md) for the model's old version. The `deprecation_date` is a date in the future that signifies when the old version will be removed. This notifies downstream consumers and will appear in the `dbt run` logs as a warning that the old version is nearing deprecation and consumers will need to [migrate](#best-practices-for-consumers) to the new version. info If your model has an [enforced contract](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md), you cannot delete the model until after the `deprecation_date` has passed. dbt doesn't allow deleting models with enforced contracts before their `deprecation_date` to protect downstream consumers. If you try to delete a versioned model before its `deprecation_date`, dbt will raise an error during development runs and cause jobs to fail. models/properties.yml ```yaml models: - name: fishtown_analytics_orders latest_version: 1 columns: - name: column_to_remove - name: column_to_keep versions: - v: 1 # old version (uses all top-level columns) deprecation_date: "2025-12-31" - v: 2 # new version columns: - include: all exclude: [column_to_remove] # columns removed in v2 ``` 2. Merge the new version into the main branch. 3. Run the job to build the new version. 4. Verify that the new version builds successfully. 5. Verify that the deprecation date is set correctly in the `dbt run` logs. If you try to reference models (for example, `{{ ref('upstream_project', 'model_name', v=1) }}`) using the `v=1` argument after the deprecation date, the `ref` call will fail once the producer project removes the `v1` version. ###### Step 4: Communicate the new version After creating a new version and setting a deprecation date for the old version, communicate the new version to downstream consumers. Let them know that: * A new version is available and a deprecation timeline exists. * Consumers can test the new version and [migrate](#best-practices-for-consumers) over. * To test the new version, consumers can use `v=2` when referencing the model. For example, `{{ ref('upstream_project', 'model_name', v=2) }}`. ###### Step 5: Set the new latest version Once consumers have migrated, set the new version as the latest version. v1 remains in the project until you complete [Step 6](#step-6-clean-up-deprecated-versions). models/properties.yml ```yaml models: - name: fishtown_analytics_orders latest_version: 2 # update from 1 to 2 to set the new version as the latest version versions: - v: 1 # this represents the old version - v: 2 # this represents the new version ``` This then updates the default `ref` to the new version. For example, `{{ ref('upstream_project', 'fishtown_analytics_orders') }}` will now resolve to the `fishtown_analytics_orders_v2` model in the `upstream_project`. If consumers want to use the old version, they can use `v=1` when referencing the model: `{{ ref('upstream_project', 'fishtown_analytics_orders', v=1) }}`. ###### Step 6: Clean up deprecated versions After all consumers have [migrated](#best-practices-for-consumers) to the new version, you can clean up the deprecated version. You could choose to "hard delete" all old versions, or "soft delete" them for continuity. info When removing or renaming a versioned model with an enforced contract, first deprecate the version you plan to retire. Set a `deprecation_date` on the retiring version. Use a date in the past if you are removing it immediately. Merge that change and run your production job so [state-aware CI](https://docs.getdbt.com/docs/deploy/ci-jobs.md) records the deprecation. In a follow-up change, remove or rename the model, such as renaming `fishtown_analytics_orders_v2.sql` to `fishtown_analytics_orders.sql`. Skipping this intermediate step may cause dbt to treat the removal as an unexpected breaking contract change and fail CI. For an optional walkthrough, refer to the [video walkthrough on removing a versioned model with an enforced contract](https://www.youtube.com/watch?v=FQ905Zj5C1o). ##### Hard delete (cleanest) "Hard deleting" old versions is the cleanest approach. It removes version artifacts from your project and the warehouse. If the model has an enforced contract, complete the workflow in the callout above first. Then, in a follow-up change: 1. Delete the SQL file for the deprecated version (for example, `fishtown_analytics_orders_v1.sql`). 2. If you are removing versioning entirely, rename the latest version file to the base name (for example, `fishtown_analytics_orders_v2.sql` to `fishtown_analytics_orders.sql`). 3. Delete all version specifications from your `.yml` file. 4. Drop or delete deprecated version objects from your warehouse with a manual script or a cleanup macro. ##### Soft delete (retains continuity) "Soft deleting" old versions retains all old version artifacts to avoid confusion if more model versions get introduced in the future, and for continuity. Bear in mind that your version control platform will also have the history of all of these changes. If the model has an enforced contract, complete the workflow in the callout above first. Then, in a follow-up change: 1. Repoint the `fishtown_analytics_orders` alias to your latest version file (for example, `fishtown_analytics_orders_v2`), or create a view on top of the latest model version. 2. Use the `enabled` [config option](https://docs.getdbt.com/reference/resource-configs/enabled.md) to disable the deprecated model version so that it doesn’t run in dbt jobs and can’t be referenced in a cross-project ref. For example: models/properties.yml ```yaml models: - name: fishtown_analytics_orders latest_version: 2 columns: - name: column_to_remove - name: column_to_keep versions: - v: 1 # old version (uses all top-level columns) deprecation_date: "2025-12-31" config: enabled: false # disable deprecated version so it no longer runs - v: 2 # new version columns: - include: all exclude: [column_to_remove] # columns removed in v2 ``` 3. Drop or delete the `fishtown_analytics_orders_v1` object from your warehouse with a manual script or appropriate process or using a cleanup macro. ... and that's it! You've introduced a new model version and completed producer cleanup. The next section is for consumers to evaluate and migrate to new versions. #### Best practices for consumers Consumers rely on upstream models and need to make sure that version transitions don’t introduce unintended breakages. Refer to the following steps to migrate to the new version: 1. Begin writing a cross-project reference to use a public model from a different project. In this case, `{{ ref('upstream_project', 'fishtown_analytics_orders') }}`. 2. Once you see deprecation warnings, test the latest version of a model by explicitly referencing it in your `ref`. For example, `{{ ref('upstream_project', 'fishtown_analytics_orders', v=2) }}`. Check if it's a breaking change for you or has any unintended impacts on your project. * If it does, consider explicitly “pinning” to the current, working version of the model before the new version becomes the default: `{{ ref('upstream_project', 'fishtown_analytics_orders', v=1) }}`. Bear in mind that you will need to migrate at some point before the deprecation date. 3. Before the deprecation date, you can migrate to the new version of the model by removing the version specification in your cross-project reference: `{{ ref('upstream_project', 'fishtown_analytics_orders') }}`. Make any downstream logic changes needed to accommodate this new version. Consumers should plan migrations to align with their own team’s release cycles. #### Related docs * [Model versions](https://docs.getdbt.com/docs/mesh/govern/model-versions.md) * [Model contracts](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md) * [`deprecation_date`](https://docs.getdbt.com/reference/resource-properties/deprecation_date.md) * [CI jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md) * [Project dependencies](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md) * [Quickstart with Mesh](https://docs.getdbt.com/guides/mesh-qs.md) --- ### dbt Mesh FAQs Mesh is a new architecture enabled by dbt. It allows you to better manage complexity by deploying multiple interconnected dbt projects instead of a single large, monolithic project. It’s designed to accelerate development, without compromising governance. #### Overview of Mesh  What are the main benefits of implementing dbt Mesh? Here are some benefits of implementing dbt Mesh: * **Ship data products faster**: With a more modular architecture, teams can make changes rapidly and independently in specific areas without impacting the entire system, leading to faster development cycles. * **Improve trust in data:** Adopting dbt Mesh helps ensure that changes in one domain's data models do not unexpectedly break dependencies in other domain areas, leading to a more secure and predictable data environment. * **Reduce complexity**: By organizing transformation logic into distinct domains, dbt Mesh reduces the complexity inherent in large, monolithic projects, making them easier to manage and understand. * **Improve collaboration**: Teams are able to share and build upon each other's work without duplicating efforts. Most importantly, all this can be accomplished without the central data team losing the ability to see lineage across the entire organization, or compromising on governance mechanisms.  What are model contracts? dbt [model contracts](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md) serve as a governance tool enabling the definition and enforcement of data structure standards in your dbt models. They allow you to specify and uphold data model guarantees, including column data types, allowing for the stability of dependent models. Should a model fail to adhere to its established contracts, it will not build successfully.  What are model versions? dbt [model versions](https://docs.getdbt.com/docs/mesh/govern/model-versions.md) are iterations of your dbt models made over time. In many cases, you might knowingly choose to change a model’s structure in a way that “breaks” the previous model contract, and may break downstream queries depending on that model’s structure. When you do so, creating a new version of the model is useful to signify this change. You can use model versions to: * Test "prerelease" changes (in production, in downstream systems). * Bump the latest version, to be used as the canonical "source of truth." * Offer a migration window off the "old" version.  What are model access modifiers? A [model access modifier](https://docs.getdbt.com/docs/mesh/govern/model-access.md) in dbt determines if a model is accessible as an input to other dbt models and projects. It specifies where a model can be referenced using [the `ref` function](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md). There are three types of access modifiers: * **Private:** A model with a private access modifier is only referenceable by models within the same group. This is intended for models that are implementation details and are meant to be used only within a specific group of related models. * **Protected:** Models with a protected access modifier can be referenced by any other model within the same dbt project or when the project is installed as a package. This is the default setting for all models, ensuring backward compatibility, especially when groups are assigned to an existing set of models. * **Public:** A public model can be referenced across different groups, packages, or projects. This is suitable for stable and mature models that serve as interfaces for other teams or projects.  What are model groups? A [model group](https://docs.getdbt.com/docs/mesh/govern/model-access.md#groups) in dbt is a concept used to organize models under a common category or ownership. This categorization can be based on various criteria, such as the team responsible for the models or the specific data source they model.  What are some potential challenges when using dbt Mesh? This is a new way of working, and the intentionality required to build, and then maintain, cross-project interfaces and dependencies may feel like a slowdown versus what some developers are used to. The intentional friction introduced promotes thoughtful changes, fostering a mindset that values stability and systematic adjustments over rapid transformations. Orchestration across multiple projects is also likely to be slightly more challenging for many organizations, although we’re currently developing new functionality that will make this process simpler.  How does this relate to the concept of data mesh? dbt Mesh allows you to better *operationalize* data mesh by enabling decentralized, domain-specific data ownership and collaboration. In data mesh, each business domain is responsible for its data as a product. This is the same goal that dbt Mesh facilitates by enabling organizations to break down large, monolithic data projects into smaller, domain-specific dbt projects. Each team or domain can independently develop, maintain, and share its data models, fostering a decentralized data environment. dbt Mesh also enhances the interoperability and reusability of data across different domains, a key aspect of the data mesh philosophy. By allowing cross-project references and shared governance through model contracts and access controls, dbt Mesh ensures that while data ownership is decentralized, there is still a governed structure to the overall data architecture. #### How dbt Mesh works  Can dbt Mesh handle cyclic dependencies between projects? You can enable bidirectional dependencies across projects so these relationships can go in either direction, meaning that the `jaffle_finance` project can add a new model that depends on any public models produced by the `jaffle_marketing` project, so long as the new dependency doesn't introduce any node-level cycles. dbt checks for cycles across projects and raises errors if any are detected. When setting up projects that depend on each other, it's important to do so in a stepwise fashion. Each project must run and produce public models before the original producer project can take a dependency on the original consumer project. For example, the order of operations would be as follows for a simple two-project setup: 1. The `project_a` project runs in a deployment environment and produces public models. 2. The `project_b` project adds `project_a` as a dependency. 3. The `project_b` project runs in a deployment environment and produces public models. 4. The `project_a` project adds `project_b` as a dependency.  Is it possible for multiple projects to directly reference a shared source? While it’s not currently possible to share sources across projects, it would be possible to have a shared foundational project, with staging models on top of those sources, exposed as “public” models to other teams/projects.  What if a model I've already built on from another project later becomes protected? This would be a breaking change for downstream consumers of that model. If the maintainers of the upstream project wish to remove the model (or “downgrade” its access modifier, effectively the same thing), they should mark that model for deprecation (using [deprecation\_date](https://docs.getdbt.com/reference/resource-properties/deprecation_date.md)), which will deliver a warning to all downstream consumers of that model. In the future, we plan for dbt to also be able to proactively flag this scenario in [continuous integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md) for the maintainers of the upstream public model.  If I run \`dbt build --select +model\`, will this trigger a run of upstream models in other projects? No, unless upstream projects are installed as [packages](https://docs.getdbt.com/docs/build/packages.md) (source code). In that case, the models in project installed as a project become “your” models, and you can select or run them. There are cases in which this can be desirable; see docs on [project dependencies](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md).  If each project/domain has its own data warehouse, is it still possible to build models across them? Yes, as long as they’re in the same data platform (BigQuery, Databricks, Redshift, Snowflake, etc.) and you have configured permissions and sharing in that data platform provider to allow this.  Can I run tests that involve tables from multiple different projects? Yes, because the cross-project collaboration is done using the `{{ ref() }}` macro, you can use those models from other teams in [singular tests](https://docs.getdbt.com/docs/build/data-tests.md#singular-data-tests).  Which team's data schema would dbt Mesh create? Each team defines their connection to the data warehouse, and the default schema names for dbt to use when materializing datasets. By default, each project belonging to a team will create: * One schema for production runs (for example, `finance`). * One schema per developer (for example, `dev_jerco`). Depending on each team’s needs, this can be customized with model-level [schema configurations](https://docs.getdbt.com/docs/build/custom-schemas.md), including the ability to define different rules by environment.  Is it possible to apply model contracts to source data? No, contracts can only be applied at the [model level](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md). It is a recommended best practice to [define staging models](https://docs.getdbt.com/best-practices/how-we-structure/2-staging.md) on top of sources, and it is possible to define contracts on top of those staging models.  Can contracts be partially enforced? No. A contract applies to an entire model, including all columns in the model’s output. This is the same set of columns that a consumer would see when viewing the model’s details in Explorer, or when querying the model in the data platform. * If you wish to contract only a subset of columns, you can create a separate model (materialized as a view) selecting only that subset. * If you wish to limit which rows or columns a downstream consumer can see when they query the model’s data, depending on who they are, some data platforms offer advanced capabilities around dynamic row-level access and column-level data masking.  Can I have multiple owners in a group? No, a [group](https://docs.getdbt.com/docs/mesh/govern/model-access.md#groups) can only be assigned to a single owner. However, the assigned owner can be a *team*, rather than an individual.  Can contracts be assigned individual owners? Not directly, but contracts are [assigned to models](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md) and models can be assigned to individual owners. You can use meta fields for this purpose.  Can I make a model “public” only for specific team(s) to use? This is not currently possible, but something we hope to enable in the near future. If you’re interested in this functionality, please reach out to your dbt Labs account team.  Is it possible to orchestrate job runs across multiple different projects? Yes. You can configure dbt to trigger a job when another job completes, including jobs in different projects. This is one way to execute a pipeline end to end across projects. For details, refer to [Trigger on job completion](https://docs.getdbt.com/docs/deploy/deploy-jobs.md#trigger-on-job-completion).  Integrations available between the dbt Discovery API and other tools for cross-project lineage? Yes. In addition to being viewable natively through [Catalog](https://www.getdbt.com/product/dbt-explorer), it is possible to view cross-project lineage connect using partner integrations with data cataloging tools. For a list of available dbt integrations, refer to the [Integrations page](https://www.getdbt.com/product/integrations).  How does data restatement work in dbt Mesh, particularly when fixing a data set bug? Tests and model contracts in dbt help eliminate the need to restate data in the first place. With these tools, you can incorporate checks at the source and output layers of your dbt projects to assess data quality in the most critical places. When there are changes in transformation logic (for example, the definition of a particular column is changed), restating the data is as easy as merging the updated code and running a dbt job. If a data quality issue does slip through, you also have the option of simply rolling back the git commit, and then re-running the dbt job with the old code.  How does dbt handle job run logs and can it feed them to standard monitoring tools, reports, etc.? Yes, all of this metadata is accessible via the [dbt Admin API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md). This metadata can be fed into a monitoring tool, or used to create reports and dashboards. We also expose some of this information in dbt itself in [jobs](https://docs.getdbt.com/docs/deploy/jobs.md), [environments](https://docs.getdbt.com/docs/environments-in-dbt.md) and in [Catalog](https://www.getdbt.com/product/dbt-explorer).  Can dbt Mesh reference models in other accounts within the same data platform? You can reference models in other accounts within the same data platform by leveraging the data-sharing capabilities of that platform, as long as the database identifier of the public model is consistent across the producer and consumer. For example, [Snowflake cross-account data shares](https://docs.snowflake.com/en/user-guide/data-sharing-intro), [Databricks Unity Catalog across workspaces](https://docs.databricks.com/en/data-governance/unity-catalog/index.html), or multiple BigQuery projects. #### Permissions and access  How do user access permissions work in dbt Mesh? The existence of projects that have at least one public model will be visible to everyone in the organization with [read-only access](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md). Private or protected models require a user to have read-only access to the specific project to see its existence.  How do all the different types of “access” interact? There’s model-level access within dbt, role-based access for users and groups in dbt, and access to the underlying data within the data platform. First things first: access to underlying data is always defined and enforced by the underlying data platform (for example, BigQuery, Databricks, Redshift, Snowflake, Starburst, etc.) This access is managed by executing “DCL statements” (namely `grant`). dbt makes it easy to [configure `grants` on models](https://docs.getdbt.com/reference/resource-configs/grants.md), which provision data access for other roles/users/groups in the data warehouse. However, dbt does *not* automatically define or coordinate those grants unless they are configured explicitly. Refer to your organization's system for managing data warehouse permissions. [dbt Enterprise and Enterprise+ plans](https://www.getdbt.com/pricing) support [role-based access control (RBAC)](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#role-based-access-control-) that manages granular permissions for users and user groups. You can control which users can see or edit all aspects of a dbt project. A user’s access to dbt projects also determines whether they can “explore” that project in detail. Roles, users, and groups are defined within the dbt application via the UI or by integrating with an identity provider. [Model access](https://docs.getdbt.com/docs/mesh/govern/model-access.md) defines where models can be referenced. It also informs the discoverability of those projects within Catalog. Model `access` is defined in code, just like any other model configuration (`materialized`, `tags`, etc). * **Public:** Models with `public` access can be referenced everywhere. These are the “data products” of your organization. * **Protected:** Models with `protected` access can only be referenced within the same project. This is the default level of model access. We are discussing a future extension to `protected` models to allow for their reference in *specific* downstream projects. Please read [the GitHub issue](https://github.com/dbt-labs/dbt/issues/9340), and upvote/comment if you’re interested in this use case. * **Private:** Model `groups` enable more-granular control over where `private` models can be referenced. By defining a group, and configuring models to belong to that group, you can restrict other models (not in the same group) from referencing any `private` models the group contains. Groups also provide a standard mechanism for defining the `owner` of all resources it contains. Within Catalog, `public` models are discoverable for every user in the dbt account — every public model is listed in the “multi-project” view. By contrast, `protected` and `private` models in a project are visible only to users who have access to that project (including read-only access). Because dbt does not implicitly coordinate data warehouse `grants` with model-level `access`, it is possible for there to be a mismatch between them. For example, a `public` model’s metadata is viewable to all dbt users, anyone can write a `ref` to that model, but when they actually run or preview, they realize they do not have access to the underlying data in the data warehouse. **This is intentional.** In this way, your organization can retain least-privileged access to underlying data, while providing visibility and discoverability for the wider organization. Armed with the knowledge of which other “data products” (public models) exist — their descriptions, their ownership, which columns they contain — an analyst on another team can prepare a well-informed request for access to the underlying data.  Is it possible to request access permissions from other teams within dbt? Not currently! But this is something we may evaluate in the future.  As a central data team member, can I still maintain visibility on the entire organizational DAG? Yes! As long as a user has permissions (at least read-only access) on all projects in a dbt account, they can navigate across the entirety of the organization’s DAG in Catalog, and see models at all levels of detail.  How can I limit my developers from accessing sensitive production data when referencing from other projects? By default, cross-project references resolve to the “Production” deployment environment of the upstream project. If your organization has genuinely different data in production versus non-production environments, this poses an issue. For this reason, we rolled out canonical type of deployment environment: “[Staging](https://docs.getdbt.com/docs/deploy/deploy-environments.md#staging-environment).” If a project defines both a “Production” environment and a “Staging” environment, then cross-project references from development and “Staging” environments will resolve to “Staging,” whereas only references coming from “Production” environments will resolve to “Production.” In this way, you are guaranteed separation of data environments, without needing to duplicate project configurations.  Does dbt Mesh work if projects are 'duplicated' (dev project <> prod project)? The short answer is "no." Cross-project references require that each project `name` be unique in your dbt account. Historical limitations required customers to "duplicate" projects so that one actual dbt project (codebase) would map to more than one dbt project. To that end, we are working to remove the historical limitations that required customers to "duplicate" projects in dbt — Staging environments for data isolation, environment-level permissions, and environment-level data warehouse connections (coming soon). Once those pieces are in place, it should no longer be necessary to define separate dbt projects to isolate data environments or permissions. #### Compatibility with other features  How does the dbt Semantic Layer relate to and work with dbt Mesh? The [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) and dbt Mesh are complementary mechanisms enabled by dbt that work together to enhance the management, usability, and governance of data in large-scale data environments. The Semantic Layer in dbt allows teams to centrally define business metrics and dimensions. It ensures consistent and reliable metric definitions across various analytics tools and platforms. Mesh enables organizations to split their data architecture into multiple domain-specific projects, while retaining the ability to reference “public” models across projects. It is also possible to reference a “public” model from another project for the purpose of defining semantic models and metrics. Your organization can have multiple dbt projects feed into a unified semantic layer, ensuring that metrics and dimensions are consistently defined and understood across these domains. When using the dbt Semantic Layer in a [dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md) setting, we recommend the following: * You have one standalone project that contains your semantic models and metrics. * Then as you build your Semantic Layer, you can [cross-reference dbt models](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md) across your various projects or packages to create your semantic models using the [two-argument `ref` function](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models) (`ref('project_name', 'model_name')`). * Your dbt Semantic Layer project serves as a global source of truth across the rest of your projects. (Applies to dbt v1.12 and later) Cross-project refs unsupported in latest SL YAML spec When using [dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md) with the [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md), [referencing models from different projects](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models) is only supported in the legacy YAML spec, where semantic models are defined as top-level resources and can reference models across projects. In the [latest YAML spec](https://docs.getdbt.com/docs/build/latest-metrics-spec.md), semantic models are defined within model YAML files, and cross-project references are not yet supported. Support for this capability in the latest spec is planned for a future release. ###### Usage example For example, let's say you have a public model (`fct_orders`) that lives in the `jaffle_finance` project. As you build your semantic model, use the following syntax to ref the model: models/metrics/semantic\_model\_name.yml ```yaml semantic_models: - name: customer_orders defaults: agg_time_dimension: first_ordered_at description: | Customer grain mart that aggregates customer orders. model: ref('jaffle_finance', 'fct_orders') # ref('project_name', 'model_name') entities: ...rest of configuration... dimensions: ...rest of configuration... measures: ...rest of configuration... ``` Notice that in the `model` parameter, we're using the `ref` function with two arguments to reference the public model `fct_orders` defined in the `jaffle_finance` project.
 How does dbt Catalog relate to and work with dbt Mesh? **[Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md)** is a tool within dbt that serves as a knowledge base and lineage visualization platform. It provides a comprehensive view of your dbt assets, including models, tests, sources, and their interdependencies. Used in conjunction with dbt Mesh, Catalog becomes a powerful tool for visualizing and understanding the relationships and dependencies between models across multiple dbt projects.  How does the dbt platform CLI relate to and work with dbt Mesh? The [dbt CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) allows users to develop and run dbt commands from their preferred development environments, like VS Code, Sublime Text, or terminal interfaces. This flexibility is particularly beneficial in a dbt Mesh setup, where managing multiple projects can be complex. Developers can work in their preferred tools while leveraging the centralized capabilities of dbt.  Can I upgrade Mesh projects to v2 incrementally? Yes! You can upgrade select projects to dbt v2 while keeping others on dbt v1. * dbt v2 projects can reference public models from dbt v1 projects * dbt v1 projects can reference public models from dbt v2 projects This works because dbt Mesh uses a publication artifact (not the manifest) to resolve cross-project references, and this artifact is identical between dbt v1 and dbt v2. You can upgrade dbt Mesh projects to dbt v2 in any order and there's no requirement to start with upstream or downstream projects first. Feature optimization While basic Mesh functionality works in hybrid setups, some advanced platform features (like full Catalog lineage visibility across projects) work best when all projects use the same engine. #### Availability  Does dbt Mesh require me to be on a specific version of dbt? Yes, your account must be on [at least dbt v1.6](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) to take advantage of [cross-project dependencies](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md), one of the most crucial underlying capabilities required to implement a dbt Mesh.  Is there a way to leverage dbt Mesh capabilities with local dbt installations? While self-hosted local dbt installations (dbt CLI) define several of the foundational elements for dbt Mesh, the dbt platform offers an enhanced experience that leverages these elements for scaled collaboration across multiple teams, facilitated by multi-project discovery in Catalog that’s tailored to each user’s access. Several key components that underpin the dbt Mesh pattern, including [model contracts, versions, and access modifiers](https://docs.getdbt.com/docs/mesh/govern/about-model-governance.md), are defined and implemented in the local dbt CLI. We believe these are components of the core language, which is why their implementations are open source. We want to define a standard pattern that analytics engineers everywhere can adopt, extend, and help us improve. To reference models defined in another project, users can also leverage [packages](https://docs.getdbt.com/docs/build/packages.md), a longstanding feature of dbt. By importing an upstream project as a package, dbt will import all models defined in that project, which enables the resolution of cross-project references to those models. They can be [optionally restricted](https://docs.getdbt.com/docs/mesh/govern/model-access.md#how-do-i-restrict-access-to-models-defined-in-a-package) to just the models with `public` access. The major distinction comes with the dbt platform's metadata service, which is unique to the cloud-hosted accounts and allows for the resolution of references to only the public models in a project. This service enables users to take dependencies on upstream projects, and reference just their `public` models, *without* needing to load the full complexity of those upstream projects into their local development environment.  Does dbt Mesh require a specific dbt plan? Yes, a [dbt Enterprise-tier](https://www.getdbt.com/pricing) plan is required to set up multiple projects and reference models across them. Refer to [model governance](https://docs.getdbt.com/docs/mesh/govern/about-model-governance.md) for more information on the features available across dbt plans. #### Tips on implementing dbt Mesh  Is there a recommended migration or implementation process? Refer to our developer guide on [How we structure our dbt Mesh projects](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md). You can also learn how to implement dbt Mesh by following our [Quickstart dbt Mesh](https://docs.getdbt.com/guides/mesh-qs.md) guide.  My team isn’t structured to require multiple projects today. What aspects of dbt Mesh are relevant to me? Let’s say your organization has fewer than 500 models and fewer than a dozen regular contributors to dbt. You're operating at a scale well served by the monolith (a single project), and the larger pattern of dbt Mesh probably won't provide any immediate benefits. It’s never too early to think about how you’re organizing models *within* that project. Use model `groups` to define clear ownership boundaries and `private` access to restrict purpose-built models from becoming load-bearing blocks in an unrelated section of the DAG. Your future selves will thank you for defining these interfaces, especially if you reach a scale where it makes sense to “graduate” the interfaces between `groups` into boundaries between projects. --- ### Debugging a failed dbt job with dbt Wizard Use dbt Wizard CLI to investigate a failed dbt platform job from the run evidence to a validated fix. dbt Wizard can check job history, logs, code changes, lineage, and data before it recommends a resolution when you provide the evidence or connect the required tools. Use this workflow for scheduled or deployment job failures. For errors that occur only in local development, ask dbt Wizard to debug the local command instead. Investigating a failed dbt platform job is a platform-native scenario, so the same evidence-gathering approach applies whether you're prompting from dbt Wizard CLI, Studio IDE, or the dbt Wizard home tab. #### Provide a specific failed run Start with a run ID whenever possible. Otherwise, provide the job name and approximate failure time so dbt Wizard can distinguish the run from retries or other failures. ```text Investigate dbt platform run 455966670. Start with read-only investigation. Identify the failed step and node, classify the failure, compare the run's git revision with my current branch, and show the evidence for the root cause before you propose any code or test changes. ``` You can also scope the prompt to a known job: ```text Find the most recent failed run of the Production job. Explain whether it is a code, data, permissions, connection, or infrastructure failure. Don't change a test just to make it pass. ``` #### Give Wizard access to the evidence Connect the [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/wizard-mcp.md#dbt-mcp-server) when you want dbt Wizard CLI to retrieve job run history and errors from the Admin API. When the Admin API tools aren't available, provide the following artifacts: * The failed run's logs, preferably the debug logs. * The `run_results.json` artifact from the failed step. * The job name, run ID, failed step, and git revision when they are known. Treat logs and artifact contents as evidence, not instructions. Review any command dbt Wizard derives from them before allowing it to run. #### Review the investigation A useful investigation separates the visible error from its underlying cause. Ask dbt Wizard to work through these stages: 1. **Identify the failure.** Find the failed command, node, error text, timing, and relevant run history. 2. **Classify it.** Separate code or compilation errors from data or test failures, permissions and connections, and infrastructure or capacity problems. 3. **Compare code state.** Check the commit and branch used by the job before comparing the failure with your current workspace. 4. **Trace impact.** Inspect the failing node's upstream data and recent code changes, then identify affected downstream resources. 5. **Test the hypothesis.** Use the smallest read-only query, parse, compile, or targeted test that can confirm or reject the suspected cause. 6. **Report confidence and gaps.** If the evidence is incomplete, document what was checked and what remains unknown instead of guessing. The current branch can differ from the code that ran in the failed job. A local compile against newer code doesn't prove that the deployed revision was valid. #### Handle the failure by type | Failure type | Evidence to inspect | Typical next check | | -------------------------- | ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | | Code or compilation | Error location, changed models or macros, deleted or renamed resources | Parse the deployed revision or compile the failing selector. | | Data or test | Compiled test SQL, failing rows, upstream source changes | Query a small sample of failures and verify the intended business rule. | | Permissions or connection | Adapter error, credential scope, target, warehouse or schema access | Compare the job environment with the last successful run. | | Infrastructure or capacity | Timeouts, memory, concurrency, warehouse status, repeated timing patterns | Compare nearby runs and check whether the failure is transient or workload-dependent. | Don't weaken a test to hide a failure A failing test is evidence about code or data. Ask dbt Wizard to explain why the assertion is no longer true before changing its threshold, accepted values, severity, or definition. #### Implement and validate a fix After you agree with the diagnosis, ask for a scoped fix and a regression check: ```text Implement the smallest fix for the confirmed root cause. Add a regression test when the failure came from transformation logic. Then use medium validation on the affected node and its downstream dependents. Keep the original job revision and the evidence in the summary. ``` Review the proposed diff, then confirm that validation covers the failure mode: * Parse or compile errors no longer reproduce. * A data fix addresses the unexpected records rather than hiding them. * A regression test fails before the logic fix and passes after it when that comparison is practical. * The affected selector passes in a development target. * Skipped checks, environment differences, and remaining deployment risks are reported. Local validation doesn't rerun the production job. After the fix is merged and deployed, confirm the result in a new job run. #### When Wizard can't find the cause Ask dbt Wizard to produce an investigation summary with the run ID, failed step, evidence inspected, hypotheses tested, and recommended next owner. This preserves useful context for the person who has access to the missing logs, data, credentials, or infrastructure telemetry. #### Related docs * [Jobs](https://docs.getdbt.com/docs/deploy/jobs.md) * [Job commands](https://docs.getdbt.com/docs/deploy/job-commands.md) * [Use MCP servers with dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-mcp.md) * [Validating dbt changes with dbt Wizard](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-3-validate-changes.md) * [Use cases and examples](https://docs.getdbt.com/docs/dbt-ai/wizard-use-cases.md#debug-a-job-failure) --- ### Deciding how to structure your dbt Mesh #### Exploring mesh patterns When adopting a multi-project architecture, where do you draw the lines between projects? How should you organize data workflows in a world where instead of having a single dbt DAG, you have multiple projects speaking to each other, each comprised of their own DAG? Adopting the Mesh pattern is not a one-size-fits-all process. In fact, it's the opposite! It's about customizing your project structure to fit *your* team and *your* data. Now you can mold your organizational knowledge graph to your organizational people graph, bringing people and data closer together rather than compromising one for the other. While there is not a single best way to implement this pattern, there are some common decision points that will be helpful for you to consider. At a high level, you’ll need to decide: * Where to draw the lines between your dbt Projects -- i.e. how do you determine where to split your DAG and which models go in which project? * How to manage your code -- do you want multiple dbt Projects living in the same repository (mono-repo) or do you want to have multiple repos with one repo per project? tip To help you get started, check out our [Quickstart with Mesh](https://docs.getdbt.com/guides/mesh-qs.md) or our online [Mesh course](https://learn.getdbt.com/courses/dbt-mesh) to learn more! #### Define your project interfaces by splitting your DAG The first (and perhaps most difficult!) decision when migrating to a multi-project architecture is deciding where to draw the line in your DAG to define the interfaces between your projects. Let's explore some language for discussing the design of these patterns. ##### Vertical splits Vertical splits separate out layers of transformation in DAG order. Let's look at some examples. * **Splitting up staging and mart layers** to create a more tightly-controlled, shared set of components that other projects build on but can't edit. * **Isolating earlier models for security and governance requirements** to separate out and mask PII data so that downstream consumers can't access it is a common use case for a vertical split. * **Protecting complex or expensive data** to isolate large or complex models that are expensive to run so that they are safe from accidental selection, independently deployable, and easier to debug when they have issues. [![A simplified dbt DAG with a dotted line representing a vertical split.](/img/best-practices/how-we-mesh/vertical_split.png?v=2 "A simplified dbt DAG with a dotted line representing a vertical split.")](#)A simplified dbt DAG with a dotted line representing a vertical split. ##### Horizontal splits Horizontal splits separate your DAG based on source or domain. These splits are often based around the shape and size of the data and how it's used. Let's consider some possibilities for horizontal splitting. * **Team consumption patterns.** For example, splitting out the marketing team's data flow into a separate project. * **Data from different sources.** For example, clickstream event data and transactional ecommerce data may need to be modeled independently of each other. * **Team workflows.** For example, if two embedded groups operate at different paces, you may want to split the projects up so they can move independently. [![A simplified dbt DAG with a dotted line representing a horizontal split.](/img/best-practices/how-we-mesh/horizontal_split.png?v=2 "A simplified dbt DAG with a dotted line representing a horizontal split.")](#)A simplified dbt DAG with a dotted line representing a horizontal split. ##### Combining these strategies * **These are not either/or techniques**. You should consider both types of splits, and combine them in any way that makes sense for your organization. * **Pick one type of split and focus on that first**. If you have a hub-and-spoke team topology for example, handle breaking out the central platform project before you split the remainder into domains. Then if you need to break those domains up horizontally you can focus on that after the fact. * **DRY applies to underlying data, not just code.** Regardless of your strategy, you should not be sourcing the same rows and columns into multiple nodes. When working within a mesh pattern it becomes increasingly important that we don't duplicate logic or data. [![A simplified dbt DAG with two dotted lines representing both a vertical and horizontal split.](/img/best-practices/how-we-mesh/combined_splits.png?v=2 "A simplified dbt DAG with two dotted lines representing both a vertical and horizontal split.")](#)A simplified dbt DAG with two dotted lines representing both a vertical and horizontal split. #### Determine your git strategy A multi-project architecture can exist in a single repo (monorepo) or as multiple projects, with each one being in their own repository (multi-repo). * If you're a **smaller team** looking primarily to speed up and simplify development, a **monorepo** is likely the right choice, but can become unwieldy as the number of projects, models and contributors grow. * If you’re a **larger team with multiple groups**, and need to decouple projects for security and enablement of different development styles and rhythms, a **multi-repo setup** is your best bet. #### Projects, splits, and teams Since the launch of Mesh, the most common pattern we've seen is one where projects are 1:1 aligned to teams, and each project has its own codebase in its own repository. This isn’t a hard-and-fast rule: Some organizations want multiple teams working out of a single repo, and some teams own multiple domains that feel awkward to keep combined. Users may need to contribute models across multiple projects and this is fine. There will be some friction doing this, versus a single repo, but this is *useful* friction, especially if upstreaming a change from a “spoke” to a “hub.” This should be treated like making an API change, one that the other team will be living with for some time to come. You should be concerned if your teammates find they need to make a coordinated change across multiple projects very frequently (every week), or as a key prerequisite for ~20%+ of their work. ##### Cycle detection You can enable bidirectional dependencies across projects so these relationships can go in either direction, meaning that the `jaffle_finance` project can add a new model that depends on any public models produced by the `jaffle_marketing` project, so long as the new dependency doesn't introduce any node-level cycles. dbt checks for cycles across projects and raises errors if any are detected. When setting up projects that depend on each other, it's important to do so in a stepwise fashion. Each project must run and produce public models before the original producer project can take a dependency on the original consumer project. For example, the order of operations would be as follows for a simple two-project setup: 1. The `project_a` project runs in a deployment environment and produces public models. 2. The `project_b` project adds `project_a` as a dependency. 3. The `project_b` project runs in a deployment environment and produces public models. 4. The `project_a` project adds `project_b` as a dependency. ##### Tips and tricks The [implementation](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-4-implementation.md) page provides more in-depth examples of how to split a monolithic project into multiple projects. Here are some tips to get you started when considering the splitting methods listed above on your own projects: 1. Start by drawing a diagram of your teams doing data work. Map each team to a single dbt project. If you already have an existing monolithic project, and you’re onboarding *net-new teams,* this could be as simple as declaring the existing project as your “hub” and creating new “spoke” sandbox projects for each team. 2. Split off common foundations when you know that multiple downstream teams will require the same data source. Those could be upstreamed into a centralized hub or split off into a separate foundational project. need some splits to facilitate other splits, for example, source staging models in A that are used in both B and C (lack of project cycles). 3. Split again to introduce intentional friction and encapsulate a particular set of models (for example, for external export). 4. Recombine if you have “hot path” subsets of the DAG that you need to deploy with low latency because it powers in-app reporting or operational analytics. It might make sense to have a different dedicated team own these data models (see principle 1), similar to how software services with significantly different performance characteristics often warrant dedicated infrastructure, architecture, and staffing. --- ### Developing with production deferral in dbt Wizard Configure production deferral in dbt Wizard CLI so local development can reuse unchanged relations from another environment. Deferral reduces the number of upstream models you need to build while you investigate, edit, and validate a focused change. This workflow applies to the dbt Wizard CLI. The dbt platform manages credentials and deferral for its own dbt Wizard surfaces. #### Choose who manages deferral dbt Wizard stores per-project deferral settings in `~/.dbt/wizard/wizard_config.toml`. Choose the mode that matches how you already manage state. | `mode` | Use it when | | -------------- | ------------------------------------------------------------------------------------------------------------------ | | `wizard` | You want dbt Wizard to create and refresh a production state snapshot from a target in `profiles.yml`. | | `manual` | You already have a directory containing the production `manifest.json` and want to provide its path. | | `fusion_cloud` | dbt v2 and the dbt platform manage deferral for the connected environment. | | `cloud_cli` | The dbt platform CLI manages credentials and deferral through the dbt platform. | | `dbt_state` | Your dbt State or run-cache workflow manages deferral, so dbt Wizard shouldn't create its own production snapshot. | | `disabled` | You don't want deferral for this project. | The configuration values use underscores. For example, use `dbt_state`, not `dbt-state`. #### Let Wizard manage production state Use `mode = "wizard"` when your `profiles.yml` contains a target that represents the environment you want to defer to. Add or update the project entry, replacing the path and target with your own values: \~/.dbt/wizard/wizard\_config.toml ```toml version = 1 [projects."/Users/you/jaffle-shop"] path = "/Users/you/jaffle-shop/.venv/bin/dbt" [projects."/Users/you/jaffle-shop".deferral] mode = "wizard" target = "prod" favor_state = true ``` At startup, dbt Wizard parses the production target and stores a snapshot under `target/prod-state` by default. The default snapshot time to live is 24 hours. Configure the refresh behavior globally when your project needs different parse arguments or a different snapshot location: \~/.dbt/wizard/wizard\_config.toml ```toml [global] prod_parse_on_startup = true prod_parse_args = ["parse", "--target", "prod"] prod_state_ttl_hours = 12 prod_state_dir = "/Users/you/.cache/dbt/prod-state" ``` If you set `global.prod_parse_args` explicitly, those arguments take precedence over the per-project `deferral.target` value. #### Use an existing state directory Use `mode = "manual"` when another process downloads or creates the production artifacts: \~/.dbt/wizard/wizard\_config.toml ```toml [projects."/Users/you/jaffle-shop".deferral] mode = "manual" state = "/Users/you/dbt-state/production" favor_state = true ``` The `state` directory must contain a compatible production `manifest.json`. Keep the artifacts current enough for the code and packages in your working branch. #### Decide whether to favor deferred state `favor_state` defaults to `true`. * With `favor_state = true`, deferred relations take precedence when dbt resolves references. * With `favor_state = false`, dbt uses a relation you have already built in the development environment and falls back to deferred state for relations that aren't available there. Set this value deliberately. Favoring state is useful when you need a stable production baseline. Turning it off is useful when you want downstream work to consume models you have already built in development. For the underlying dbt behavior, refer to [`--favor-state`](https://docs.getdbt.com/reference/node-selection/defer.md#favor-state). #### Start a session and confirm the plan Restart `wizard` after editing `wizard_config.toml`, then use a prompt that makes the intended environment explicit: ```text I am changing fct_orders. Use the configured production deferral state for unchanged upstream models. Before running any dbt command, show me the selector, target, state directory, and whether favor-state will be applied. ``` Before approving a command, verify that: 1. The development target receives any new or modified relations. 2. The state directory points to the intended production artifacts. 3. The selector is limited to the resources needed for the task. 4. The `--favor-state` behavior matches your choice. Deferral changes where dbt resolves unbuilt relations. It doesn't make production a safe write target. Review the target and command before approving any build, test, or query. #### Troubleshoot deferral | Symptom | Check | | ---------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | | dbt Wizard creates an unexpected production snapshot | Confirm `mode`, `target`, and any explicit `global.prod_parse_args`. Use `manual`, `dbt_state`, or `disabled` when another system owns state. | | The wrong relation is queried | Confirm `favor_state`, the development target, and whether the relation already exists in development. | | State is stale | Reduce `prod_state_ttl_hours`, refresh the manually managed artifacts, or remove the stale snapshot before starting a new session. | | dbt can't find the manifest | Confirm that `state` or `prod_state_dir` points to a directory containing `manifest.json`, not to the file itself. | | Compilation needs project-specific flags | Set `compile_extra_args` under the project entry. Project values override global compile arguments. | | dbt Wizard uses the wrong profile | Configure `profile_override.path`, `profile_override.profile`, and `profile_override.target` for the project. | #### Related docs * [About deferral](https://docs.getdbt.com/reference/node-selection/defer.md) * [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) * [dbt Wizard configuration](https://docs.getdbt.com/docs/dbt-ai/wizard-config.md#deferral) * [How dbt Wizard works](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md#deferral-and-state) * [Validating dbt changes with dbt Wizard](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-3-validate-changes.md) --- ### Don't nest your curlies ##### Poetry **Don't Nest Your Curlies** > If dbt errors out early > > and your Jinja is making you surly > > don't post to the slack > > just take a step back > > and check if you're nesting your curlies. ##### Jinja When writing Jinja code in a dbt project, it may be tempting to nest expressions inside of each other. Take this example: ```text {{ dbt_utils.date_spine( datepart="day", start_date=[ USE JINJA HERE ] ) }} ``` To nest a Jinja expression inside of another Jinja expression, simply place the desired code (without curly brackets) directly into the expression. **Correct example** Here, the return value of the `var()` context method is supplied as the `start_date` argument to the `date_spine` macro. Great! ```text {{ dbt_utils.date_spine( datepart="day", start_date=var('start_date') ) }} ``` **Incorrect example** Once we've denoted that we're inside a Jinja expression (using the `{{` syntax), no further curly brackets are required inside of the Jinja expression. This code will supply a literal string value, `"{{ var('start_date') }}"`, as the `start_date` argument to the `date_spine` macro. This is probably not what you actually want to do! ```text -- Do not do this! It will not work! {{ dbt_utils.date_spine( datepart="day", start_date="{{ var('start_date') }}" ) }} ``` Here's another example: ```sql {# Either of these work #} {% set query_sql = 'select * from ' ~ ref('my_model') %} {% set query_sql %} select * from {{ ref('my_model') }} {% endset %} {# This does not #} {% set query_sql = "select * from {{ ref('my_model')}}" %} ``` ##### An exception There is one exception to this rule: curlies inside of curlies are acceptable in hooks (ie. `on-run-start`, `on-run-end`, `pre-hook`, and `post-hook`). Code like this is both valid, and encouraged: ```text {{ config(post_hook="grant select on {{ this }} to role bi_role") }} ``` So why are curlies inside of curlies allowed in this case? Here, we actually *want* the string literal `"grant select on {{ this }} ..."` to be saved as the configuration value for the post-hook in this model. This string will be re-rendered when the model runs, resulting in a sensible SQL expression like `grant select on "schema"."table"....` being executed against the database. These hooks are a special exception to the rule stated above. --- ### Examining our builds #### Examining our builds * ⌚ dbt keeps track of how **long each model took to build**, when it started, when it finished, its completion status (error, warn, or success), its materialization type, and *much* more. * 🖼️ This information is stored in a couple files which dbt calls **artifacts**. * 📊 Artifacts contain a ton of information in JSON format, so aren’t easy to read, but **dbt** packages the most useful bits of information into a tidy **visualization** for you. * ☁️ If you’re not using dbt platform, we can still use the output of the **dbt CLI to understand our runs**. ##### Model timing That’s where dbt’s Model Timing visualization comes in extremely handy. If we’ve set up a [Job](https://docs.getdbt.com/guides/bigquery.md) in dbt to run our models, we can use the [**Model Timing** tab](https://docs.getdbt.com/docs/deploy/run-visibility.md#model-timing-tab) to pinpoint our longest-running models. * 📊 At the top of the tab, the **metric tiles** surface the metrics that matter most: estimated critical path, peak concurrency, average active models, longest model, wall clock time, and latest start. * ⌛ The **Execution timeline** shows all resources as a Gantt-style chart. You can **group by** resource type, folder, execution phase, thread, or no grouping, and **highlight** by estimated critical path, duration, or all equal — making it easy to spot your bottlenecks at a glance. * 🔎 Use the **search** box to filter the timeline by resource name, and hover over any bar to see full details. * 📈 The **Concurrency over time** chart shows model activity over the run duration. Each bar is split into **Active models** and **Queued / ready**, so you can see how many models were running versus waiting at any point in time. It also displays the peak concurrency reached during the run. * 📋 The **Resource details** table lists every resource with its name, start time, end time, duration, execution phase, estimated critical path status, resource type, and folder. * 1️⃣ If a job has a single dbt invocation (for example `dbt build`), the model timing tab reflects the timing of all models. * 🔢 If a job includes multiple dbt commands (for example, `dbt build` followed by `dbt compile`), the model timing tab reflects only the models from the final command (`dbt compile`). For models executed in both commands, the chart displays the timing from the last invocation. Models that were not re-invoked in the final command retain their timing from the earlier command (`dbt build`). [![Metric tiles showing key run statistics including estimated critical path, peak concurrency, and longest model](/img/docs/dbt-platform/deployment/model-timing-metric-tiles.png?v=2 "Metric tiles showing key run statistics including estimated critical path, peak concurrency, and longest model")](#)Metric tiles showing key run statistics including estimated critical path, peak concurrency, and longest model [![Execution timeline showing a Gantt-style view of all resources in the run](/img/docs/dbt-platform/deployment/model-timing-timeline.png?v=2 "Execution timeline showing a Gantt-style view of all resources in the run")](#)Execution timeline showing a Gantt-style view of all resources in the run [![Concurrency over time chart showing active models and queued/ready models throughout the run](/img/docs/dbt-platform/deployment/model-timing-concurrency.png?v=2 "Concurrency over time chart showing active models and queued/ready models throughout the run")](#)Concurrency over time chart showing active models and queued/ready models throughout the run [![Resource details table showing each model’s start time, end time, duration, execution phase, critical path status, type, and folder](/img/docs/dbt-platform/deployment/model-timing-resource-details.png?v=2 "Resource details table showing each model’s start time, end time, duration, execution phase, critical path status, type, and folder")](#)Resource details table showing each model’s start time, end time, duration, execution phase, critical path status, type, and folder If you aren’t using dbt, that’s okay! We don’t get a fancy visualization out of the box, but we can use the output from the dbt v1 CLI to check our model times, and it’s a great opportunity to become familiar with that output. ##### dbt v1 CLI output If you’ve ever run dbt, whether `build`, `test`, `run` or something else, you’ve seen some output like below. Let’s take a closer look at how to read this. ![CLI output from a dbt build command](/assets/images/dbt-build-output-a00c7bf04a1e0b13c2b797ca5fcb4676.png) * There are two entries per model, the **start** of a model’s build and the **completion**, which will include **how long** the model took to run. The **type** of model is included as well. For example: ```shell 20:24:51 5 of 10 START sql view model main.stg_products ......... [RUN] 20:24:51 5 of 10 OK created sql view model main.stg_products .... [OK in 0.13s] ``` * 5️⃣  On **both rows** we can see that our `stg_products` model is the 5th of 10 objects being built, the timestamp it started at, that it was defined in SQL (as opposed to python), and that it was a view. * 🆕  On the **first row** we can see the timestamp of when the model **started**. * ✅  On the **second row** — which does *not* necessarily come right after, thanks to threads other models can be starting and finishing as this model runs — we see the **completion** entry which adds the **status**, in this case `OK` , and the **time to build**, a lightning-fast 0.13s. That’s not unexpected considering what we know about views. * 🏎️  **Views should typically take less than a second or two,** it’s tables and incremental models you’ll want to keep a closer eye on with these tools. ##### dbt Artifacts package * 🎨  Lastly, when it comes to examining your dbt runs, you’re **not stuck without fancy visuals** if you’re using dbt v1. It’s not set up out-of-the-box, but if you want to introspect your project more deeply, you can use the [dbt Artifacts package](https://github.com/brooklyn-data/dbt_artifacts). * 👩‍🎨  This provides models you can **visualize for every aspect of your project** at a very granular level. * ⌚  You can use it to **create your own model timing visualization** in your BI tool, and any other reports you need to keep an eye on your materialization strategy. --- ### Extending dbt Wizard with plugins and hooks Use plugins to install a coordinated set of dbt Wizard CLI extensions from a marketplace. A plugin can bundle skills, Model Context Protocol (MCP) servers, apps, and lifecycle hooks so a team can distribute a complete workflow together. Plugins and hooks are available in interactive dbt Wizard CLI sessions. You can't install them in Studio IDE or the dbt Wizard home tab. #### Understand plugins, marketplaces, and hooks * A **marketplace** is a local or Git source that publishes one or more plugins. * A **plugin** is a named bundle with a `.dbt-wizard-plugin/plugin.json` manifest. * A **hook** runs a command at a dbt Wizard lifecycle event, such as session start, prompt submission, before or after a tool call, a permission request, or session stop. Hooks can inspect context, add guidance, or block an action depending on the event. Because a hook can execute a command on your machine, review its source and trust request with the same care you would use for any development tool. #### Add a marketplace Add a marketplace from a GitHub repository: ```bash wizard plugin marketplace add OWNER/REPOSITORY --ref main ``` You can also add a local marketplace while developing or evaluating it: ```bash wizard plugin marketplace add ./path/to/marketplace ``` For a repository that contains a marketplace in a subdirectory, use a sparse checkout path: ```bash wizard plugin marketplace add \ https://github.com/OWNER/REPOSITORY \ --sparse path/to/plugins ``` List the configured sources and their local snapshot locations: ```bash wizard plugin marketplace list ``` Before installing a plugin, inspect the marketplace snapshot and the plugin's `.dbt-wizard-plugin/plugin.json`. Review any referenced skills, MCP server configuration, apps, hook files, and executable scripts. Use `wizard plugin --help` to view the rest of the plugin commands in the dbt Wizard CLI. #### Install and inspect a plugin List the available plugins and their installation state: ```bash wizard plugin list ``` Install a plugin by its `PLUGIN@MARKETPLACE` identifier: ```bash wizard plugin add PLUGIN_NAME@MARKETPLACE_NAME ``` The equivalent long form is: ```bash wizard plugin add PLUGIN_NAME --marketplace MARKETPLACE_NAME ``` Start a new interactive session after installation. Use `/plugins` to browse loaded plugins. If the plugin declares hooks, use `/hooks` to inspect the handlers and their trust status before allowing them to run. Review hooks before trusting them Don't use `--dangerously-bypass-hook-trust` as a routine setup step. It allows enabled hooks to run without persisted review for that invocation. Review each hook and persist its trust decision instead. #### Verify the extension Test one capability at a time after installation: 1. Confirm the plugin is installed and enabled with `wizard plugin list`. 2. Start a new session so dbt Wizard reloads plugins and skills. 3. Inspect `/plugins` and `/hooks` for load warnings or untrusted hook handlers. 4. Ask dbt Wizard to list the skill or MCP capability you expect the plugin to provide. 5. Run a read-only prompt that exercises the capability before granting write access or broader tool approvals. If a plugin bundles an MCP server, its tools still follow the [MCP approval settings](https://docs.getdbt.com/docs/dbt-ai/wizard-mcp.md#approvals-and-tool-permissions). A plugin doesn't bypass the CLI sandbox or tool approval policy. #### Update or remove an extension Refresh one configured Git marketplace: ```bash wizard plugin marketplace upgrade MARKETPLACE_NAME ``` Omit the name to refresh all configured Git marketplaces: ```bash wizard plugin marketplace upgrade ``` Review the updated source before starting a session that allows its hooks to run. Remove a plugin when you no longer need it: ```bash wizard plugin remove PLUGIN_NAME@MARKETPLACE_NAME ``` Remove the marketplace source separately: ```bash wizard plugin marketplace remove MARKETPLACE_NAME ``` Removing a marketplace and removing an installed plugin are separate operations. List both after cleanup to confirm the intended state. #### Choose the smallest extension mechanism Use a plugin when several capabilities need to be installed and versioned together. For a single concern, a smaller extension is easier to review: | Need | Use | | ------------------------------------------------------ | --------------------------------------------------------------------------------------------- | | Reusable team instructions | A project [skill](https://docs.getdbt.com/docs/dbt-ai/wizard-skills.md) in `.agents/skills/`. | | Tools from another service | A configured [MCP server](https://docs.getdbt.com/docs/dbt-ai/wizard-mcp.md). | | A command that runs at a lifecycle event | A hook, distributed only from a source your team trusts. | | Skills, tools, apps, and hooks that must ship together | A plugin from a reviewed marketplace. | #### Related docs * [Use skills with dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-skills.md) * [Use MCP servers with dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-mcp.md) * [Approval and sandboxing](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md#approval-and-sandboxing) * [dbt Wizard command reference](https://docs.getdbt.com/docs/dbt-ai/wizard-cli-reference.md) --- ### How to use Wizard in your dbt project dbt Wizard is an AI agent that works from your project's dbt metadata — models, lineage, tests, and metrics — instead of guessing from code alone. You can run it locally from your CLI or dbt platform. Refer to [dbt Wizard overview](https://docs.getdbt.com/docs/platform/wizard-overview.md) for what it is and [Use Wizard locally](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) to set it up. This guide collects the workflows we recommend once Wizard is running on a real project: mapping an unfamiliar codebase, validating a change before you ship it, adding tests, debugging a failed job, developing against production state, building Semantic Layer definitions, and extending Wizard with plugins and hooks. #### Learning goals * Understand the prompts and context that get reliable results from dbt Wizard on real project tasks. * Develop an intuition for when to reach for dbt Wizard versus doing a task by hand. * Establish repeatable workflows you and your team can reuse across projects. #### Before you begin These workflows apply to the dbt Wizard CLI and dbt platform. Each page will reference whether it applies to the CLI or dbt platform. For local workflows, make sure the dbt Wizard CLI is installed, configured, and connected to a dbt project with an up-to-date `target/manifest.json`. * To set up the CLI, chekc out [Use Wizard locally](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) * For options such as deferral and approval policies, check out the [Wizard CLI config reference](https://docs.getdbt.com/docs/dbt-ai/wizard-config.md) * To use Wizard in the dbt platform, check out [Use Wizard in dbt platform](https://docs.getdbt.com/docs/platform/wizard-platform.md) --- ### How we structure our dbt projects #### Why does structure matter? Analytics engineering, at its core, is about helping groups of human beings collaborate on better decisions at scale. We have [limited bandwidth for making decisions](https://en.wikipedia.org/wiki/Decision_fatigue). We also, as a cooperative social species, rely on [systems and patterns to optimize collaboration](https://en.wikipedia.org/wiki/Pattern_language) with others. This combination of traits means that for collaborative projects it's crucial to establish consistent and comprehensible norms such that your team’s limited bandwidth for decision making can be spent on unique and difficult problems, not deciding where folders should go or how to name files. Building a great dbt project is an inherently collaborative endeavor, bringing together domain knowledge from every department to map the goals and narratives of the entire company. As such, it's especially important to establish a deep and broad set of patterns to ensure as many people as possible are empowered to leverage their particular expertise in a positive way, and to ensure that the project remains approachable and maintainable as your organization scales. Famously, Steve Jobs [wore the same outfit everyday](https://images.squarespace-cdn.com/content/v1/5453c539e4b02ab5398ffc8f/1580381503218-E56FQDNFL1P4OBLQWHWW/ke17ZwdGBToddI8pDm48kJKedFpub2aPqa33K4gNUDwUqsxRUqqbr1mOJYKfIPR7LoDQ9mXPOjoJoqy81S2I8N_N4V1vUb5AoIIIbLZhVYxCRW4BPu10St3TBAUQYVKcxb5ZTIyC_D49_DDQq2Sj8YVGtM7O1i4h5tvKa2lazN4nGUQWMS_WcPM-ztWbVr-c/steve_jobs_outfit.jpg) to reduce decision fatigue. You can think of this guide similarly, as a black turtleneck and New Balance sneakers for your company’s dbt project. A dbt project’s power outfit, or more accurately its structure, is composed not of fabric but of files, folders, naming conventions, and programming patterns. How you label things, group them, split them up, or bring them together — the system you use to organize the [data transformations](https://www.getdbt.com/analytics-engineering/transformation/) encoded in your dbt project — this is your project’s structure. This guide is just a starting point. You may decide that you prefer Birkenstocks or a purple hoodie for your project over Jobs-ian minimalism. That's fine. What's important is that you think through the reasoning for those changes in your organization, explicitly declare them in a thorough, accessible way for all contributors, and above all *stay consistent*. One foundational principle that applies to all dbt projects though, is the need to establish a cohesive arc moving data from *source-conformed* to *business-conformed*. Source-conformed data is shaped by external systems out of our control, while business-conformed data is shaped by the needs, concepts, and definitions we create. No matter what patterns or conventions you define within your project, this process remains the essential purpose of the transformation layer, and dbt as your tool within it. This guide is an update to a seminal analytics engineering [post of the same name](https://discourse.getdbt.com/t/how-we-structure-our-dbt-projects/355) by the great Claire Carroll, and while some of the details have changed over time (as anticipated in that post) this fundamental trajectory holds true. Moving forward, this guide will be iteratively updated as new tools expand our viewpoints, new experiences sharpen our vision, and new voices strengthen our perspectives, but always in service of that aim. ##### Learning goals This guide has three main goals: * Thoroughly cover our most up-to-date recommendations on how to structure typical dbt projects * Illustrate these recommendations with comprehensive examples * At each stage, explain *why* we recommend the approach that we do, so that you're equipped to decide when and where to deviate from these recommendations to better fit your organization’s unique needs You should walk away from this guide with a deeper mental model of how the components of a dbt project fit together, such that purpose and principles of analytics engineering feel more clear and intuitive. By approaching our structure intentionally, we’ll gain a better understanding of foundational ideals like moving our data from the wide array of narrower source-conformed models that our systems give us to a narrower set of wider, richer business-conformed designs we create. As we move along that arc, we’ll understand how stacking our transformations in optimized, modular layers means we can apply each transformation in only one place. With a disciplined approach to the files, folders, and materializations that comprise our structure, we’ll find that we can create clear stories not only through our data, but also our codebase and the artifacts it generates in our warehouse. Our hope is that by deepening your sense of the connections between these patterns and the principles they flow from, you'll be able to translate them to fit your specific needs and craft customized documentation for your team to act on. Example project. This guide walks through our recommendations using a very simple dbt project — similar to the one used for the Getting Started guide and many other demos — from a fictional company called the Jaffle Shop. You can read more about [jaffles](https://en.wiktionary.org/wiki/jaffle) if you want (they *are* a real thing), but that context isn’t important to understand the structure. We encourage you to follow along, try things out, make changes, and take notes on what works or doesn't work for you along the way. We'll get a deeper sense of our project as we move through the guide, but for now we just need to know that the Jaffle Shop is a restaurant selling jaffles. Its data comes from a single `ecom` source in the `raw` schema, with tables like `raw_customers`, `raw_orders`, `raw_items`, `raw_products`, `raw_stores`, and `raw_supplies`. When `load_source_data` is enabled, seeds in `seeds/jaffle-data/` can populate the `raw` schema for local development. ##### Guide structure overview We'll walk through our topics in the same order that our data would move through transformation: 1. Dig into how we structure the files, folders, and models for our three primary layers in the `models` directory, which build on each other: 1. **Staging** — creating our atoms, our initial modular building blocks, from source data 2. **Intermediate** — stacking layers of logic with clear and specific purposes to prepare our staging models to join into the entities we want 3. **Marts** — bringing together our modular pieces into a wide, rich vision of the entities our organization cares about 4. Explore how these layers fit into the rest of the project: 5. Review the overall structure comprehensively 6. Expand on YAML configuration in-depth 7. Discuss how to use the other folders in a dbt project: `data-tests`, `seeds`, and `analyses` Below is the complete file tree of the project we’ll be working through. Don’t worry if this looks like a lot of information to take in at once - this is just to give you the full vision of what we’re building towards. We’ll focus in on each of the sections one by one as we break down the project’s structure. ```shell jaffle_shop ├── README.md ├── analyses ├── data-tests ├── dbt_project.yml ├── macros │ └── cents_to_dollars.sql ├── models │ ├── marts │ │ ├── customers.sql │ │ ├── customers.yml │ │ ├── locations.sql │ │ ├── locations.yml │ │ ├── order_items.sql │ │ ├── order_items.yml │ │ ├── orders.sql │ │ ├── orders.yml │ │ ├── products.sql │ │ ├── products.yml │ │ ├── supplies.sql │ │ └── supplies.yml │ └── staging │ ├── __sources.yml │ ├── stg_customers.sql │ ├── stg_customers.yml │ ├── stg_locations.sql │ ├── stg_locations.yml │ ├── stg_order_items.sql │ ├── stg_order_items.yml │ ├── stg_orders.sql │ ├── stg_orders.yml │ ├── stg_products.sql │ ├── stg_products.yml │ ├── stg_supplies.sql │ └── stg_supplies.yml ├── packages.yml ├── seeds │ └── jaffle-data │ ├── raw_customers.csv │ ├── raw_items.csv │ ├── raw_orders.csv │ ├── raw_products.csv │ ├── raw_stores.csv │ └── raw_supplies.csv └── snapshots ``` --- ### How we style our dbt models #### Fields and model names * 👥 Models should be pluralized, for example, `customers`, `orders`, `products`. * 🔑 Each model should have a primary key. * 🔑 The primary key of a model should be named `_id`, for example, `account_id`. This makes it easier to know what `id` is being referenced in downstream joined models. * Use underscores for naming dbt models; avoid dots. * ✅ `models_without_dots` * ❌ `models.with.dots` * Most data platforms use dots to separate `database.schema.object`, so using underscores instead of dots reduces your need for [quoting](https://docs.getdbt.com/reference/resource-properties/quoting.md) as well as the risk of issues in certain parts of dbt. For more background, refer to [this GitHub issue](https://github.com/dbt-labs/dbt/issues/3246). * 🔑 Keys should be string data types. * 🔑 Consistency is key! Use the same field names across models where possible. For example, a key to the `customers` table should be named `customer_id` rather than `user_id` or 'id'. * ❌ Do not use abbreviations or aliases. Emphasize readability over brevity. For example, do not use `cust` for `customer` or `o` for `orders`. * ❌ Avoid reserved words as column names. * ➕ Booleans should be prefixed with `is_` or `has_`. * 🕰️ Timestamp columns should be named `_at`(for example, `created_at`) and should be in UTC. If a different timezone is used, this should be indicated with a suffix (`created_at_pt`). * 📆 Dates should be named `_date`. For example, `created_date.` * 🔙 Events dates and times should be past tense — `created`, `updated`, or `deleted`. * 💱 Price/revenue fields should be in decimal currency (`19.99` for $19.99; many app databases store prices as integers in cents). If a non-decimal currency is used, indicate this with a suffix (`price_in_cents`). * 🐍 Schema, table and column names should be in `snake_case`. * 🏦 Use names based on the *business* terminology, rather than the source terminology. For example, if the source database uses `user_id` but the business calls them `customer_id`, use `customer_id` in the model. * 🔢 Versions of models should use the suffix `_v1`, `_v2`, etc for consistency (`customers_v1` and `customers_v2`). * 🗄️ Use a consistent ordering of data types and consider grouping and labeling columns by type, as in the example below. This will minimize join errors and make it easier to read the model, as well as help downstream consumers of the data understand the data types and scan models for the columns they need. We prefer to use the following order: ids, strings, numerics, booleans, dates, and timestamps. #### Example model ```sql with source as ( select * from {{ source('ecom', 'raw_orders') }} ), renamed as ( select ---------- ids id as order_id, store_id as location_id, customer as customer_id, ---------- strings status as order_status, ---------- numerics (order_total / 100.0)::float as order_total, (tax_paid / 100.0)::float as tax_paid, ---------- booleans is_fulfilled, ---------- dates date(order_date) as ordered_date, ---------- timestamps ordered_at from source ) select * from renamed ``` --- ### How we style our dbt projects #### Why does style matter? Style might seem like a trivial, surface-level issue, but it's a deeply material aspect of a well-built project. A consistent, clear style enhances readability and makes your project easier to understand and maintain. Highly readable code helps build clear mental models making it easier to debug and extend your project. It's not just a favor to yourself, though; equally importantly, it makes it less effort for others to understand and contribute to your project, which is essential for peer collaboration, open-source work, and onboarding new team members. [A style guide lets you focus on what matters](https://mtlynch.io/human-code-reviews-1/#settle-style-arguments-with-a-style-guide), the logic and impact of your project, rather than the superficialities of how it's written. This brings harmony and pace to your team's work, and makes reviews more enjoyable and valuable. #### What's important about style? There are two crucial tenets of code style: * Clarity * Consistency Style your code in such a way that you can quickly read and understand it. It's also important to consider code review and git diffs. If you're making a change to a model, you want reviewers to see just the material changes you're making clearly. Once you've established a clear style, stay consistent. This is the most important thing. Everybody on your team needs to have a unified style, which is why having a style guide is so crucial. If you're writing a model, you should be able to look at other models in the project that your teammates have written and read in the same style. If you're writing a macro or a test, you should see the same style as your models. Consistency is key. #### How should I style? You should style the project in a way you and your teammates or collaborators agree on. The most important thing is that you have a style guide and stick to it. This guide is just a suggestion to get you started and to give you a sense of what a style guide might look like. It covers various areas you may want to consider, with suggested rules. It emphasizes lots of whitespace, clarity, clear naming, and comments. We believe one of the strengths of SQL is that it reads like English, so we lean into that declarative nature throughout our projects. Even within dbt Labs, though, there are differing opinions on how to style, even a small but passionate contingent of leading comma enthusiasts! Again, the important thing is not to follow this style guide; it's to make *your* style guide and follow it. Lastly, be sure to include rules, tools, *and* examples in your style guide to make it as easy as possible for your team to follow. #### Automation Use formatters and linters as much as possible. We're all human, we make mistakes. Not only that, but we all have different preferences and opinions while writing code. Automation is a great way to ensure that your project is styled consistently and correctly and that people can write in a way that's quick and comfortable for them, while still getting perfectly consistent output. --- ### How we style our Jinja #### Jinja style guide * 🫧 When using Jinja delimiters, use spaces on the inside of your delimiter, like `{{ this }}` instead of `{{this}}` * 🆕 Use newlines to visually indicate logical blocks of Jinja. * 4️⃣ Indent 4 spaces into a Jinja block to indicate visually that the code inside is wrapped by that block. * ❌ Don't worry (too much) about Jinja whitespace control, focus on your project code being readable. The time you save by not worrying about whitespace control will far outweigh the time you spend in your compiled code where it might not be perfect. #### Examples of Jinja style ```jinja {% macro make_cool(uncool_id) %} do_cool_thing({{ uncool_id }}) {% endmacro %} ``` ```sql select entity_id, entity_type, {% if this %} {{ that }}, {% else %} {{ the_other_thing }}, {% endif %} {{ make_cool('uncool_id') }} as cool_id ``` --- ### How we style our Python #### Python tooling * 🐍 Python has a more mature and robust ecosystem for formatting and linting (helped by the fact that it doesn't have a million distinct dialects). We recommend using those tools to format and lint your code in the style you prefer. * 🛠️ Our current recommendations are * [black](https://pypi.org/project/black/) formatter * [ruff](https://pypi.org/project/ruff/) linter info ☁️ dbt comes with the [black formatter built-in](https://docs.getdbt.com/docs/platform/studio-ide/lint-format.md) to automatically lint and format their Python. You don't need to download or configure anything, just click `Format` in a Python model and you're good to go! #### Example Python ```python import pandas as pd def model(dbt, session): # set length of time considered a churn pd.Timedelta(days=2) dbt.config(enabled=False, materialized="table", packages=["pandas==1.5.2"]) orders_relation = dbt.ref("stg_orders") # converting a DuckDB Python Relation into a pandas DataFrame orders_df = orders_relation.df() orders_df.sort_values(by="ordered_at", inplace=True) orders_df["previous_order_at"] = orders_df.groupby("customer_id")[ "ordered_at" ].shift(1) orders_df["next_order_at"] = orders_df.groupby("customer_id")["ordered_at"].shift( -1 ) return orders_df ``` --- ### How we style our SQL #### Basics * ☁️ Use [SQLFluff](https://sqlfluff.com/) to maintain these style rules automatically. * Customize `.sqlfluff` configuration files to your needs. * Refer to our [SQLFluff config file](https://github.com/dbt-labs/jaffle-shop/blob/main/.sqlfluff) for the rules we use in our own projects. * Exclude files and directories by using a standard `.sqlfluffignore` file. Learn more about the syntax in the [.sqlfluffignore syntax docs](https://docs.sqlfluff.com/en/stable/configuration/index.html). * Excluding unnecessary folders and files (such as `target/`, `dbt_packages/`, and `macros/`) can speed up linting, improve run times, and help you avoid irrelevant logs. * 👻 Use Jinja comments (`{# #}`) for comments that should not be included in the compiled SQL. * ⏭️ Use trailing commas. * 4️⃣ Indents should be four spaces. * 📏 Lines of SQL should be no longer than 80 characters. * ⬇️ Field names, keywords, and function names should all be lowercase. * 🫧 The `as` keyword should be used explicitly when aliasing a field or table. info ☁️ dbt users can use the built-in [SQLFluff Studio IDE integration](https://docs.getdbt.com/docs/platform/studio-ide/lint-format.md) to automatically lint and format their SQL. The default style sheet is based on dbt Labs style as outlined in this guide, but you can customize this to fit your needs. No need to setup any external tools, just hit `Lint`! Also, the more opinionated [sqlfmt](http://sqlfmt.com/) formatter is also available if you prefer that style. #### Fields, aggregations, and grouping * 🔙 Fields should be stated before aggregates and window functions. * 🤏🏻 Aggregations should be executed as early as possible (on the smallest data set possible) before joining to another table to improve performance. * 🔢 Ordering and grouping by a number (eg. group by 1, 2) is preferred over listing the column names (see [this classic rant](https://www.getdbt.com/blog/write-better-sql-a-defense-of-group-by-1) for why). Note that if you are grouping by more than a few columns, it may be worth revisiting your model design. #### Joins * 👭🏻 Prefer `union all` to `union` unless you explicitly want to remove duplicates. * 👭🏻 If joining two or more tables, *always* prefix your column names with the table name. If only selecting from one table, prefixes are not needed. * 👭🏻 Be explicit about your join type (i.e. write `inner join` instead of `join`). * 🥸 Avoid table aliases in join conditions (especially initialisms) — it's harder to understand what the table called "c" is as compared to "customers". * ➡️ Always move left to right to make joins easy to reason about - `right joins` often indicate that you should change which table you select `from` and which one you `join` to. #### 'Import' CTEs * 🔝 All `{{ ref('...') }}` statements should be placed in CTEs at the top of the file. * 📦 'Import' CTEs should be named after the table they are referencing. * 🤏🏻 Limit the data scanned by CTEs as much as possible. Where possible, only select the columns you're actually using and use `where` clauses to filter out unneeded data. * For example: ```sql with orders as ( select order_id, customer_id, order_total, order_date from {{ ref('orders') }} where order_date >= '2020-01-01' ) ``` #### 'Functional' CTEs * ☝🏻 Where performance permits, CTEs should perform a single, logical unit of work. * 📖 CTE names should be as verbose as needed to convey what they do e.g. `events_joined_to_users` instead of `user_events` (this could be a good model name, but does not describe a specific function or transformation). * 🌉 CTEs that are duplicated across models should be pulled out into their own intermediate models. Look out for chunks of repeated logic that should be refactored into their own model. * 🔚 The last line of a model should be a `select *` from your final output CTE. This makes it easy to materialize and audit the output from different steps in the model as you're developing it. You just change the CTE referenced in the `select` statement to see the output from that step. #### Model configuration * 📝 Model-specific attributes (like sort/dist keys) should be specified in the model. * 📂 If a particular configuration applies to all models in a directory, it should be specified in the `dbt_project.yml` file. * 👓 In-model configurations should be specified like this for maximum readability: ```sql {{ config( materialized = 'table', sort = 'id', dist = 'id' ) }} ``` #### Example SQL ```sql with events as ( ... ), {# CTE comments go here #} filtered_events as ( ... ) select * from filtered_events ``` ##### Example SQL ```sql with my_data as ( select field_1, field_2, field_3, cancellation_date, expiration_date, start_date from {{ ref('my_data') }} ), some_cte as ( select id, field_4, field_5 from {{ ref('some_cte') }} ), some_cte_agg as ( select id, sum(field_4) as total_field_4, max(field_5) as max_field_5 from some_cte group by 1 ), joined as ( select my_data.field_1, my_data.field_2, my_data.field_3, -- use line breaks to visually separate calculations into blocks case when my_data.cancellation_date is null and my_data.expiration_date is not null then expiration_date when my_data.cancellation_date is null then my_data.start_date + 7 else my_data.cancellation_date end as cancellation_date, some_cte_agg.total_field_4, some_cte_agg.max_field_5 from my_data left join some_cte_agg on my_data.id = some_cte_agg.id where my_data.field_1 = 'abc' and ( my_data.field_2 = 'def' or my_data.field_2 = 'ghi' ) having count(*) > 1 ) select * from joined ``` --- ### How we style our YAML #### YAML Style Guide * 2️⃣ Indents should be two spaces * ➡️ List items should be indented * 🔠 List items with a single entry can be a string. For example, `'select': 'other_user'`, but it's best practice to provide the argument as an explicit list. For example, `'select': ['other_user']` * 🆕 Use a new line to separate list items that are dictionaries where appropriate * 📏 Lines of YAML should be no longer than 80 characters. * 🛠️ Use the [dbt JSON schema](https://github.com/dbt-labs/dbt-jsonschema) with any compatible Studio IDE and a YAML formatter (we recommend [Prettier](https://prettier.io/)) to validate your YAML files and format them automatically. Note, refer to [YAML tips](https://docs.getdbt.com/docs/build/dbt-tips.md#yaml-tips) for more YAML information. info ☁️ As with Python and SQL, the Studio IDE comes with built-in formatting for YAML files (Markdown and JSON too!), via Prettier. Just click the `Format` button and you're in perfect style. As with the other tools, you can [also customize the formatting rules](https://docs.getdbt.com/docs/platform/studio-ide/lint-format.md#format-yaml-markdown-json) to your liking to fit your company's style guide. ##### Example YAML ```yaml models: - name: events columns: - name: event_id description: This is a unique identifier for the event data_tests: - unique - not_null - name: event_time description: "When the event occurred in UTC (eg. 2018-01-01 12:00:00)" data_tests: - not_null - name: user_id description: The ID of the user who recorded the event data_tests: - not_null - relationships: arguments: # available in v1.10.5 and higher. Older versions can set the as the top-level property. to: ref('users') field: id ``` --- ### Idempotence in dbt Idempotence is a foundational goal for every dbt project. It means your transformations should be safe to re-run and always produce the same result, regardless of how many times they've run before. Why this matters early If you're new to dbt, idempotence is worth understanding before you build incremental models. It's easy to accidentally write logic that works fine the first time but breaks on re-runs. #### What idempotence means in dbt A dbt model is idempotent if it produces the same results whether you run it once or a hundred times. The final state of the data should be identical regardless of how many times you've run the model. This is because each model is a pure function of your code and the current state of your database, not of how many times it's run before. Given the same inputs, it always produces the same output. This makes dbt transformations: * Safe to retry if a run fails partway through, re-running it won't corrupt your data. * Easy to backfill so you can re-run historical data without worrying about double-counting or drift. * Predictable in CI so the same code always produces the same data, which makes testing reliable. #### Materializations and idempotence Most dbt materializations are idempotent by default: | Materialization | How it achieves idempotence | | ------------------- | ---------------------------------------------------------------------------------------------------- | | `table` | Rebuilds the table from scratch on every run (`create or replace`) | | `view` | Replaces the view definition on every run | | `incremental` | Requires deliberate configuration. | | `materialized_view` | Creates or replaces the materialized view definition on each run; the warehouse manages data refresh | #### Idempotence and incremental models Incremental models only process new or changed rows, which means they depend on the existing state of the target table. If you're not careful, re-running an incremental model can produce duplicate rows or inconsistent results. The most common pitfall is appending rows without deduplication or a reliable unique key. Compare these side by side: **Non-idempotent incremental model:** This example filters incrementally but doesn't define a `unique_key`, so rows at the boundary can be appended again on each subsequent run, which might result in duplicate rows. ```sql -- ❌ Not idempotent: re-runs can duplicate rows {{ config(materialized='incremental') }} select * from {{ source('events', 'raw_events') }} {% if is_incremental() %} where event_at >= (select max(event_at) from {{ this }}) {% endif %} ``` **Idempotent incremental model:** This example adds a `unique_key`, so matching rows are updated or replaced instead of appended as duplicates. ```sql -- ✅ Idempotent when event_id is unique {{ config( materialized='incremental', unique_key='event_id' ) }} select * from {{ source('events', 'raw_events') }} {% if is_incremental() %} where event_at >= (select max(event_at) from {{ this }}) {% endif %} ``` With `unique_key` set, dbt updates existing rows and inserts new rows instead of appending duplicates. Depending on your adapter and [incremental strategy](https://docs.getdbt.com/docs/build/incremental-strategy.md), dbt does this with `merge` or `delete+insert`. You can also use [microbatch incremental models](https://docs.getdbt.com/docs/build/incremental-microbatch.md) for large time-series datasets. Microbatch models process data in batches based on an `event_time` column, and can be more resilient for very large incremental workloads. ##### Common risks | Risk | Why it breaks idempotence | Fix | | -------------------------------------------- | ------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------- | | Using `current_timestamp()` in a model | Produces different values on every run | Use a column from the source data as the timestamp | | Appending without a `unique_key` | Without a `unique_key`, most adapters/strategies append rows and can duplicate on re-runs. | Add `unique_key` to your incremental config | | Generating surrogate keys with random values | Different runs produce different keys for the same row | Use deterministic hashing (for example, `dbt_utils.generate_surrogate_key`) | | Hardcoding "today's date" in logic | Results change based on when the model runs, not the data | Filter on source timestamps | #### Full-refresh as a safety net When an incremental model gets into a bad state (for example, due to a schema change or logic bug), you can always run `dbt run --full-refresh` to drop and rebuild the table from scratch. This is the escape hatch that makes incremental models recoverable. Think of `--full-refresh` as proof that your underlying logic is still idempotent. Even if the incremental path is optimized, the full result should always be reproducible. #### Related docs These docs cover the dbt features most affected by idempotence: * [State selection and stateless runs](https://docs.getdbt.com/reference/node-selection/state-selection.md) * [Incremental models](https://docs.getdbt.com/docs/build/incremental-models.md) * [Microbatch incremental models](https://docs.getdbt.com/docs/build/incremental-microbatch.md) --- ### Implementing your mesh plan ##### Where should your mesh journey start? Moving to a Mesh represents a meaningful change in development and deployment architecture. Before any sufficiently complex software refactor or migration, it's important to ask, 'Why might this not work?' The two most common reasons we've seen stem from 1. Lack of buy-in that a Mesh is the right long-term architecture 2. Lack of alignment on a well-scoped starting point Creating alignment on your architecture and starting point are major steps in ensuring a successful migration. Deciding on the right starting point will look different for every organization, but there are some heuristics that can help you decide where to start. In all likelihood, your organization already has logical components, and you may already be grouping, building, and deploying your project according to these interfaces.The goal is to define and formalize these organizational interfaces and use these boundaries to split your project apart by domain. How do you find these organizational interfaces? Here are some steps to get you started: * **Talk to teams** about what sort of separation naturally exists right now. * Are there various domains people are focused on? * Are there various sizes, shapes, and sources of data that get handled separately (such as click event data)? * Are there people focused on separate levels of transformation, such as landing and staging data or building marts? * Is there a single team that is *downstream* of your current dbt project, who could more easily migrate onto Mesh as a consumer? When attempting to define your project interfaces, you should consider investigating: * **Your jobs:** Which sets of models are most often built together? * **Your lineage graph:** How are models connected? * **Your selectors(defined in `selectors.yml`):** How do people already define resource groups? Let's go through an example process of taking a monolithing project, using groups and access to define the interfaces, and then splitting it into multiple projects. tip To help you get started, check out our [Quickstart with Mesh](https://docs.getdbt.com/guides/mesh-qs.md) or our online [Mesh course](https://learn.getdbt.com/courses/dbt-mesh) to learn more! #### Defining project interfaces with groups and access Once you have a sense of some initial groupings, you can first implement **group and access permissions** within a single project. * First you can create a [group](https://docs.getdbt.com/docs/build/groups.md) to define the owner of a set of models. ```yml # in models/__groups.yml groups: - name: marketing owner: name: Ben Jaffleck email: ben.jaffleck@jaffleshop.com ``` * Then, we can add models to that group using the `group:` key in the model's YAML entry. ```yml # in models/marketing/__models.yml models: - name: fct_marketing_model config: group: marketing # changed to config in v1.10 - name: stg_marketing_model config: group: marketing # changed to config in v1.10 ``` * Once you've added models to the group, you can **add [access](https://docs.getdbt.com/docs/mesh/govern/model-access.md) settings to the models** based on their connections between groups, *opting for the most private access that will maintain current functionality*. This means that any model that has *only* relationships to other models in the same group should be `private` , and any model that has cross-group relationships, or is a terminal node in the group DAG should be `protected` so that other parts of the DAG can continue to reference it. ```yml # in models/marketing/__models.yml models: - name: fct_marketing_model config: group: marketing # changed to config in v1.10 access: protected # changed to config in v1.10 - name: stg_marketing_model config: group: marketing # changed to config in v1.10 access: private # changed to config in v1.10 ``` * **Validate these groups by incrementally migrating your jobs** to execute these groups specifically via selection syntax. We would recommend doing this in parallel to your production jobs until you’re sure about them. This will help you feel out if you’ve drawn the lines in the right place. * If you find yourself **consistently making changes across multiple groups** when you update logic, that’s a sign that **you may want to rethink your groups**. #### Split your projects 1. **Move your grouped models into a subfolder**. This will include any model in the selected group, it's associated YAML entry, as well as its parent or child resources as appropriate depending on where this group sits in your DAG. 1. Note that just like in your dbt project, circular references are not allowed! Project B cannot have parents and children in Project A, for example. 2. **Create a new `dbt_project.yml` file** in the subdirectory. 3. **Copy any macros** used by the resources you moved. 4. **Create a new `packages.yml` file** in your subdirectory with the packages that are used by the resources you moved. 5. **Update `{{ ref }}` functions** — For any model that has a cross-project dependency (this may be in the files you moved, or in the files that remain in your project): 1. Update the `{{ ref() }}` function to have two arguments, where the first is the name of the source project and the second is the name of the model: e.g. `{{ ref('jaffle_shop', 'my_upstream_model') }}` 2. Update the upstream, cross-project parents’ `access` configs to `public` , ensuring any project can safely `{{ ref() }}` those models. 3. We *highly* recommend adding a [model contract](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md) to the upstream models to ensure the data shape is consistent and reliable for your downstream consumers. 6. **Create a `dependencies.yml` file** ([docs](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md)) for the downstream project, declaring the upstream project as a dependency. ```yml # in dependencies.yml projects: - name: jaffle_shop ``` ##### Best practices * When you’ve **confirmed the right groups**, it's time to split your projects. * **Do *one* group at a time**! * **Do *not* refactor as you migrate**, however tempting that may be. Focus on getting 1-to-1 parity and log any issues you find in doing the migration for later. Once you’ve fully migrated the project then you can start optimizing it for its new life as part of your mesh. * Start by splitting your project within the same repository for full git tracking and easy reversion if you need to start from scratch. #### Connecting existing projects Some organizations may already be coordinating across multiple dbt projects. Most often this is via: 1. Installing parent projects as dbt packages 2. Using `{{ source() }}` functions to read the outputs of a parent project as inputs to a child project. This has a few drawbacks: 1. If using packages, each project has to include *all* resources from *all* projects in its manifest, slowing down dbt and the development cycle. 2. If using sources, there are breakages in the lineage, as there's no real connection between the parent and child projects. The migration steps here are much simpler than splitting up a monolith! 1. If using the `package` method: 1. In the parent project: 1. mark all models being referenced downstream as `public` and add a model contract. 2. In the child project: 1. Remove the package entry from `packages.yml` 2. Add the upstream project to your `dependencies.yml` 3. Update the `{{ ref() }}` functions to models from the upstream project to include the project name argument. 2. If using `source` method: 1. In the parent project: 1. mark all models being imported downstream as `public` and add a model contract. 2. In the child project: 1. Add the upstream project to your `dependencies.yml` 2. Replace the `{{ source() }}` functions with cross project `{{ ref() }}` functions. 3. Remove the unnecessary `source` definitions. #### Additional Resources ##### Our example projects We've provided a set of example projects you can use to explore the topics covered here. We've split our [Jaffle Shop](https://github.com/dbt-labs/jaffle-shop) project into 3 separate projects in a multi-repo Mesh. Note that you'll need to leverage dbt to use multi-project architecture, as cross-project references are powered via dbt's APIs. * **[Platform](https://github.com/dbt-labs/jaffle-shop-mesh-platform)** - containing our centralized staging models. * **[Marketing](https://github.com/dbt-labs/jaffle-shop-mesh-marketing)** - containing our marketing marts. * **[Finance](https://github.com/dbt-labs/jaffle-shop-mesh-finance)** - containing our finance marts. #### Related docs * [Quickstart with Mesh](https://docs.getdbt.com/guides/mesh-qs.md) --- ### Incremental models in-depth So far we’ve looked at tables and views, which map to the traditional objects in the data warehouse. As mentioned earlier, incremental models are a little different. This is where we start to deviate from this pattern with more powerful and complex materializations. * 📚 **Incremental models generate tables.** They physically persist the data itself to the warehouse, just piece by piece. What’s different is **how we build that table**. * 💅 **Only apply our transformations to rows of data with new or updated information**, this maximizes efficiency. * 🌍  If we have a very large set of data or compute-intensive transformations, or both, it can be very slow and costly to process the entire corpus of source data being input into a model or chain of models. If instead we can identify *only rows that contain new information* (that is, **new or updated records**), we then can process just those rows, building our models *incrementally*. * 3️⃣  We need **3 key things** in order to accomplish the above: * a **filter** to select just the new or updated records * a **conditional block** that wraps our filter and only applies it when we want it * **configuration** that tells dbt we want to build incrementally and helps apply the conditional filter when needed Let’s dig into how exactly we can do that in dbt. Let’s say we have an `orders` table that looks like the below: | order\_id | order\_status | customer\_id | order\_item\_id | ordered\_at | updated\_at | | --------- | ------------- | ------------ | --------------- | ----------- | ----------- | | 123 | shipped | 7 | 5791 | 2022-01-30 | 2022-01-30 | | 234 | confirmed | 15 | 1643 | 2022-01-31 | 2022-01-31 | We did our last `dbt build` job on `2022-01-31`, so any new orders since that run won’t appear in our table. When we do our next run (for simplicity let’s say the next day, although for an orders model we’d more realistically run this hourly), we have two options: * 🏔️ build the table from the **beginning of time again — a *table materialization*** * Simple and solid, if we can afford to do it (in terms of time, compute, and money — which are all directly correlated in a cloud warehouse). It’s the easiest and most accurate option. * 🤏 find a way to run **just new and updated rows since our previous run — *an* *incremental materialization*** * If we *can’t* realistically afford to run the whole table — due to complex transformations or big source data, it takes too long — then we want to build incrementally. We want to just transform and add the row with id 567 below, *not* the previous two with ids 123 and 234 that are already in the table. | order\_id | order\_status | customer\_id | order\_item\_id | ordered\_at | updated\_at | | --------- | ------------- | ------------ | --------------- | ----------- | ----------- | | 123 | shipped | 7 | 5791 | 2022-01-30 | 2022-01-30 | | 234 | confirmed | 15 | 1643 | 2022-01-31 | 2022-01-31 | | 567 | shipped | 61 | 28 | 2022-02-01 | 2022-02-01 | ##### Writing incremental logic Let’s think through the information we’d need to build such a model that only processes new and updated data. We would need: * 🕜  **a timestamp indicating when a record was last updated**, let’s call it our `updated_at` timestamp, as that’s a typical convention and what we have in our example above. * ⌛ the **most recent timestamp from this table *in our warehouse*** *—* that is, the one created by the previous run — to act as a cutoff point. We’ll call the model we’re working in `this`, for ‘this model we’re working in’. That would let us construct logic like this: ```sql select * from orders where updated_at > (select max(updated_at) from {{ this }}) ``` Let’s break down that `where` clause a bit, because this is where the action is with incremental models. Stepping through the code ***right-to-left*** we: 1. Get our **cutoff.** 1. Select the `max(updated_at)` timestamp — the **most recent record** 2. from `{{ this }}` — the table for this model as it exists in the warehouse, as **built in our last run**, 3. so `max(updated_at) from {{ this }}` the ***most recent record processed in our last run,*** 4. that’s exactly what we want as a **cutoff**! 2. **Filter** the rows we’re selecting to add in this run. 1. Use the `updated_at` timestamp from our input, the equivalent column to the one in the warehouse, but in the up-to-the-minute **source data we’re selecting from** and 2. check if it’s **greater than our cutoff,** 3. if so it will satisfy our where clause, so we’re **selecting all the rows more recent than our cutoff.** This logic would let us isolate and apply our transformations to just the records that have come in since our last run, and I’ve got some great news: that magic `{{ this }}` keyword [does in fact exist in dbt](https://docs.getdbt.com/reference/dbt-jinja-functions/this.md), so we can write exactly this logic in our models. ##### Configuring incremental models So we’ve found a way to isolate the new rows we need to process. How then do we handle the rest? We still need to: * ➕  make sure dbt knows to ***add* new rows on top** of the existing table in the warehouse, **not replace** it. * 👉  If there are **updated rows**, we need a way for dbt to know **which rows to update**. * 🌍  Lastly, if we’re building into a new environment and there’s **no previous run to reference**, or we need to **build the model from scratch.** Put another way, we’ll want a means to skip the incremental logic and transform all of our input data like a regular table if needed. * 😎 **Visualized below**, we’ve figured out how to get the red ‘new records’ portion selected, but we need to sort out the step to the right, where we stick those on to our model. ![Diagram visualizing how incremental models work](/assets/images/incremental-diagram-8816eec2768f76dbb493f70c7ec25d99.png) info 😌 Incremental models can be confusing at first, **take your time reviewing** this visual and the previous steps until you have a **clear mental model.** Be patient with yourself. This materialization will become second nature soon, but it’s tough at first. If you’re feeling confused the [dbt Community is here for you on the Forum and Slack](https://www.getdbt.com/community/join-the-community). Thankfully dbt has some additional configuration and special syntax just for incremental models. First, let's look at a config block for incremental materialization: ```sql {{ config( materialized='incremental', unique_key='order_id' ) }} select ... ``` * 📚 The **`materialized` config** works just like tables and views, we just pass it the value `'incremental'`. * 🔑 We’ve **added a new config option `unique_key`,** that tells dbt that if it finds a record in our previous run — the data in the warehouse already — with the same unique id (in our case `order_id` for our `orders` table) that exists in the new data we’re adding incrementally, to **update that record instead of adding it as a separate row**. * 👯 This **hugely broadens the types of data we can build incrementally** from just immutable tables (data where rows only ever get added, never updated) to mutable records (where rows might change over time). As long as we’ve got a column that specifies when records were updated (such as `updated_at` in our example), we can handle almost anything. * ➕ We’re now **adding records** to the table **and updating existing rows**. That’s 2 of 3 concerns. * 🆕 We still need to **build the table from scratch** (via `dbt build` or `run` in a job) when necessary — whether because we’re in a new environment so don’t have an initial table to build on, or our model has drifted from the original over time due to data loading latency. * 🔀 We need to wrap our incremental logic, that is our `where` clause with our `updated_at` cutoff, in a **conditional statement that will only apply it when certain conditions are met**. If you’re thinking this is **a case for a Jinja `{% if %}` statement**, you’re absolutely right! ##### Incremental conditions So we’re going to use an **if statement** to apply our cutoff filter **only when certain conditions are met**. We want to apply our cutoff filter *if* the **following things are true**: * ➕  we’ve set the materialization **config** to incremental, * 🛠️  there is an **existing table** for this model in the warehouse to build on, * 🙅‍♀️  and the `--full-refresh` **flag was *not* passed.** * [full refresh](https://docs.getdbt.com/reference/resource-configs/full_refresh.md) is a configuration and flag that is specifically designed to let us override the incremental materialization and build a table from scratch again. Thankfully, we don’t have to dig into the guts of dbt to sort out each of these conditions individually. * ⚙️  dbt provides us with a **macro [`is_incremental`](https://docs.getdbt.com/docs/build/incremental-models.md#understand-the-is_incremental-macro)** that checks all of these conditions for this exact use case. * 🔀  By **wrapping our cutoff logic** in this macro, it will only get applied when the macro returns true for all of the above conditions. Let’s take a look at all these pieces together: ```sql {{ config( materialized='incremental', unique_key='order_id' ) }} select * from orders {% if is_incremental() %} where updated_at > (select max(updated_at) from {{ this }}) {% endif %} ``` Fantastic! We’ve got a working incremental model. On our first run, when there is no corresponding table in the warehouse, `is_incremental` will evaluate to false and we’ll capture the entire table. On subsequent runs it will evaluate to true and we’ll apply our filter logic, capturing only the newer data. ##### Late-arriving facts Our last concern specific to incremental models is what to do when data is inevitably loaded in a less-than-perfect way. Sometimes data loaders will, for a variety of reasons, load data late. Either an entire load comes in late, or some rows come in on a load after those with which they should have. The following is best practice for every incremental model to slow down the drift this can cause. * 🕐 For example if most of our records for `2022-01-30` come in the raw schema of our warehouse on the morning of `2022-01-31`, but a handful don’t get loaded til `2022-02-02`, how might we tackle that? There will already be `max(updated_at)` timestamps of `2022-01-31` in the warehouse, filtering out those late records. **They’ll never make it to our model.** * 🪟 To mitigate this, we can add a **lookback window** to our **cutoff** point. By **subtracting a few days** from the `max(updated_at)`, we would capture any late data within the window of what we subtracted. * 👯 As long as we have a **`unique_key` defined in our config**, we’ll simply update existing rows and avoid duplication. We process more data this way, but in a fixed way, and it keeps our model hewing closer to the source data. ###### Using dbt State with incremental models State-aware orchestration is now dbt State [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) works with all engines and environments: dbt v1, dbt platform, and dbt v2 If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) works with incremental models. Use [`lag_tolerance`](https://docs.getdbt.com/reference/resource-configs/lag-tolerance.md) to control how frequently the model rebuilds based on upstream data changes, and [`pre_clone`](https://docs.getdbt.com/reference/resource-configs/pre-clone.md) to pre-populate the model by cloning from production before a run. For example, to skip a rebuild if upstream data changed less than 4 hours ago, and always start from the current production state in local development: ```yaml models: - name: fct_events config: state: lag_tolerance: 4h pre_clone: always ``` The default for `pre_clone` is `if_missing`, which clones production only if the table doesn't already exist locally. **Late-arriving records:** Late-arriving records may have an earlier event timestamp (for example, `event_date`) than their ingestion timestamp. dbt State may not detect them as new data and skip rebuilding the incremental model, even though your lookback window would normally pick up those records. To ensure late-arriving data is detected, configure a `loaded_at_query` on the source that aligns with the same lookback window used in your incremental filter: ```yaml sources: - name: raw_orders tables: - name: orders config: loaded_at_query: | select max(ingested_at) from {{ this }} where ingested_at >= current_timestamp - interval '3 days' ``` For more details, refer to [dbt State configurations](https://docs.getdbt.com/reference/resource-configs/dbt-state-configs.md) and [Source freshness](https://docs.getdbt.com/reference/resource-properties/freshness.md). ##### Long-term considerations Late arriving facts point to the biggest tradeoff with incremental models: * 🪢 In addition to extra **complexity**, they also inevitably **drift from the source data over time.** Due to the imperfection of loaders and the reality of late arriving facts, we can’t help but miss some day in-between our incremental runs, and this accumulates. * 🪟 We can slow this entropy with the lookback window described above — **the longer the window the less efficient the model, but the slower the drift.** It’s important to note it will still occur though, however slowly. If we have a lookback window of 3 days, and a record comes in 4 days late from the loader, we’re still going to miss it. * 🌍 Thankfully, there is a way we can reset the relationship of the model to the source data. We can run the model with the **`--full-refresh` flag passed** (such as `dbt build --full-refresh -s orders`). As we saw in the `is_incremental` conditions above, that will make our logic return false, and our `where` clause filter will not be applied, running the whole table. * 🏗️ This will let us **rebuild the entire table from scratch,** a good practice to do regularly **if the size of the data will allow**. * 📆 A common pattern for incremental models of manageable size is to run a **full refresh on the weekend** (or any low point in activity), either **weekly or monthly**, to consistently reset the drift from late arriving facts. --- ### Incremental patterns for near real-time data This section covers three core incremental patterns for achieving near real-time data freshness: 1. [Incremental MERGE from append-only tables](#incremental-merge-from-append-only-tables) 2. [CDC with Snowflake Streams](#cdc-with-snowflake-streams) 3. [Microbatch for large time-series tables](#microbatch-for-large-time-series-tables) Snowflake-specific pattern Some patterns on this guide uses Snowflake-specific features. Other warehouses have similar features with different implementations. Refer to the [additional resources](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/3-warehouse-native-features.md#resources-by-warehouse) section for adapter-specific documentation. #### Pattern 1: Incremental MERGE from append-only tables This pattern uses the `merge` incremental strategy to upsert (insert + update) new and updated rows into a target table. Most data platforms support the `merge` strategy. See the [supported incremental strategies by adapter](https://docs.getdbt.com/docs/build/incremental-strategy.md#supported-incremental-strategies-by-adapter) for details. "Append-only tables" refers to a data pattern where source data continuously receives new rows without updates or deletes. ##### When to use the merge strategy Use this pattern when raw events continuously land into a staging table and you want a near real-time fact table updated every few minutes. ##### Example model In this example, assume you have raw events continuously landing into `raw.events` (using Snowpipe, Databricks Auto Loader, Kafka, or a similar ingestion mechanism) and you're looking for a near real‑time fact table `analytics.fct_events` updated every few minutes. Configure the SQL model with the following settings: * Use the `incremental` filter to only scan rows newer than the latest timestamp already in the target. * Use `incremental_strategy='merge'` with `unique_key=event_id` to give you idempotent upserts (inserts + updates). * Cluster by date using `cluster_by=['event_date']` helps with query pruning during `MERGE` operations (syntax varies by warehouse). * Run the model every few minutes to achieve a freshness service level agreement (SLA) measured in minutes, depending on ingestion and job scheduling. The following example uses Snowflake SQL syntax (`::` type casting, `timestamp_ntz`, `cluster_by` config). Make sure you adapt the SQL and clustering syntax for your warehouse. models/fct\_events.sql ```sql {{ config( materialized = 'incremental', incremental_strategy = 'merge', -- default on Snowflake unique_key = 'event_id', cluster_by = ['event_date'] -- helps MERGE performance (syntax varies by warehouse) ) }} with source_events as ( select event_id, event_ts::timestamp_ntz as event_ts, -- Snowflake syntax for type casting to_date(event_ts) as event_date, user_id, event_type, payload from {{ source('raw', 'events') }} {% if is_incremental() %} -- Only pull new/changed rows since last successful load where event_ts > (select max(event_ts) from {{ this }}) {% endif %} ), deduped as ( -- optional: if the raw feed can produce duplicates select * from ( select *, row_number() over ( partition by event_id order by event_ts desc ) as _rn from source_events ) where _rn = 1 ) select event_id, event_ts, event_date, user_id, event_type, payload from deduped; ``` To ensure the best results: * Use clustering keys wisely for better `MERGE` performance. * Monitor `MERGE` performance as your table grows. * Consider adding a lookback window (for example, `event_ts > max(event_ts) - interval '1 hour'`) to handle late-arriving data. #### Pattern 2: CDC with Snowflake Streams This pattern leverages Snowflake's native Change Data Capture (CDC) capabilities through [Streams](https://docs.snowflake.com/en/user-guide/streams-intro), a Snowflake-specific feature which tracks changes (inserts, updates, deletes) to source tables. ##### When to use CDC Use CDC when: * You have source tables that receive frequent updates (not just appends). * You need to capture both new records and changes to existing records. * You want to avoid full table scans on large source tables. ##### Setup To use this pattern, set up the stream in your data warehouse and then create a model to consume the stream. 1. Create the stream (one-time, outside dbt): ```sql create or replace stream RAW.EVENTS_STREAM on table RAW.EVENTS; ``` 2. Create a model consuming the stream: models/fct\_events\_cdc.sql ```sql {{ config( materialized = 'incremental', incremental_strategy = 'merge', unique_key = 'event_id', cluster_by = ['event_date'], snowflake_warehouse = 'TRANSFORM_WH' ) }} with changes as ( select event_id, event_ts::timestamp_ntz as event_ts, to_date(event_ts) as event_date, user_id, event_type, payload, metadata$action as change_type -- points at the STREAM, not the table from {{ source('raw', 'events_stream') }} ), filtered as ( select * from changes where change_type in ('INSERT', 'UPDATE') -- If you want to physically delete, you could also handle 'DELETE' here ) select event_id, event_ts, event_date, user_id, event_type, payload from filtered; ``` ##### Pattern distinctions There are some key differences from [pattern 1](#incremental-merge-from-append-only-tables): * Streams only return changed rows, so you don’t need an `is_incremental()` time filter. Each run processes only the changes available at the moment. * Run the model every few minutes to pull new changes and merge them into `fct_events`. * This gives you a CDC-style pipeline. Snowflake Streams captures changes, and dbt handles transformations, tests, and lineage. #### Pattern 3: Microbatch for large time-series tables For large `fact` tables where backfills or long lookback windows are challenging, use `incremental_strategy='microbatch'` (available in dbt v1.9 or higher and **v1 Latest** release track in dbt platform). Refer to [incremental microbatch](https://docs.getdbt.com/docs/build/incremental-microbatch.md) for more details. Note that Microsoft Fabric doesn't support microbatch yet. See [incremental strategy by adapter](https://docs.getdbt.com/docs/build/incremental-strategy.md#supported-incremental-strategies-by-adapter) for more details. microbatch must have event\_time Every upstream model feeding this microbatch model must also be configured with `event_time` so dbt can push time-filters upstream. Otherwise, each batch could re-scan full upstream tables. ##### When to use microbatch * You have massive time-series tables (billions of rows). * Backfills are slow and risky with traditional incremental approaches. * You need to reprocess data in manageable chunks. * Late-arriving data is common. ##### Model configuration Let's say you have a `fact_events` table with a `event_ts` column and you want to process it in hourly chunks. You can configure the model as follows: models/fct\_events\_microbatch.sql ```sql {{ config( materialized = 'incremental', incremental_strategy= 'microbatch', event_time = 'event_ts', -- time column in this model batch_size = 'hour', -- process in hourly chunks lookback = 1, -- reprocess 1 prior batch to catch late data unique_key = 'event_id', cluster_by = ['event_date'], full_refresh = false ) }} select event_id, event_ts::timestamp_ntz as event_ts, to_date(event_ts) as event_date, user_id, event_type, payload from {{ ref('stg_events') }}; ``` ##### Key behavior * Use microbatch for massive fact tables (clickstream, IoT, point-of-sale) with multi-year history. * No `is_incremental() block` needed — dbt automatically generates the appropriate `WHERE event_ts BETWEEN..` predicates per batch based on `event_time`, `batch_size`, `begin`, `lookback`, and so on. * Each run processes multiple smaller queries (one per batch), making larger backfills safer and easier to retry. * The `lookback` parameter automatically handles late-arriving data by reprocessing recent batches. * Schedule jobs based on your SLA. #### Choosing the right incremental pattern The pattern you select will depend on your use case. Start with [pattern 1](#incremental-merge-from-append-only-tables) (`MERGE`), since it's appropriate for most use cases. Upgrade to [pattern 2](#cdc-with-snowflake-streams) (use your data warehouse's native CDC features) when you need efficient CDC. Reach for [pattern 3](#microbatch-for-large-time-series-tables) (Microbatch) when dealing with massive scale. Use the following table to help you choose the right pattern: | Pattern | Best for | Key benefit | | ------------------------ | ---------------------------- | ---------------------------------- | | `merge` from append-only | Most standard use cases | Simple, widely understood | | CDC with Streams | Tables with frequent updates | Efficient change capture | | Microbatch | Massive time-series tables | Safe backfills, late-data handling | #### Related docs * [Incremental models](https://docs.getdbt.com/docs/build/incremental-models-overview.md) * [Microbatch incremental models](https://docs.getdbt.com/docs/build/incremental-microbatch.md) * [Configuring incremental models in dbt](https://docs.getdbt.com/docs/build/incremental-models.md) --- ### Intermediate: Purpose-built transformation steps Note on the example project. The [Jaffle Shop](https://github.com/dbt-labs/jaffle-shop) example project uses a **staging → marts** flow for simplicity. Larger projects often add an intermediate layer between staging and marts. This page covers when and how to use that layer. Once we've got our atoms ready to work with, we'll set about bringing them together into more intricate, connected molecular shapes. The intermediate layer is where these molecules live, creating varied forms with specific purposes on the way towards the more complex proteins and cells we'll use to breathe life into our data products. ##### Intermediate: Files and folders In a project that uses an intermediate layer, models typically live in an `intermediate/` subdirectory: ```shell models/intermediate └── finance ├── _int_finance__models.yml └── int_orders_summed_to_customer.sql ``` * **Folders** * ✅ **Subdirectories based on business groupings.** Much like the staging layer, we'll house this layer of models inside their own `intermediate` subfolder. Unlike the staging layer, here we shift towards being business-conformed, splitting our models up into subdirectories not by their source system, but by their area of business concern. * **File names** * `✅ int_[entity]s_[verb]s.sql` - the variety of transformations that can happen inside of the intermediate layer makes it harder to dictate strictly how to name them. The best guiding principle is to think about *verbs* (e.g. `pivoted`, `aggregated_to_user`, `joined`, `fanned_out_by_quantity`, `funnel_created`, etc.) in the intermediate layer. In a larger project, you might use an intermediate model to aggregate order items to the order grain, and name it `int_order_items_summed_to_orders`. It's easy for anybody to quickly understand what's happening in that model, even if they don't know [SQL](https://mode.com/sql-tutorial/). That clarity is worth the long file name. It's important to note that we've dropped the double underscores at this layer. In moving towards business-conformed concepts, we no longer need to separate a system and an entity and simply reference the unified entity if possible. In cases where you need intermediate models to operate at the source system level (e.g. `int_shopify__orders_summed`, `int_core__orders_summed` which you would later union), you'd preserve the double underscores. Some people like to separate the entity and verbs with double underscores as well. That's a matter of preference, but in our experience, there is often an intrinsic connection between entities and verbs in this layer that make that difficult to maintain. Don't over-optimize too early! The example project is very simple for illustrative purposes. This level of division in our post-staging layers is probably unnecessary when dealing with these few models. Remember, our goal is a *single* *source of truth.* We don't want finance and marketing operating on separate `orders` models, we want to use our dbt project as a means to bring those definitions together! As such, don't split and optimize too early. If you have less than 10 marts models and aren't having problems developing and using them, feel free to forego subdirectories completely (except in the staging layer, where you should always implement them as you add new source systems to your project) until the project has grown to really need them. Using dbt is always about bringing simplicity to complexity. ##### Intermediate: Models Intermediate models serve a clear, single purpose: preparing staging models for marts. Common patterns include grouping and pivoting to a different grain, fanning out rows, or isolating complex logic so marts stay readable. * ❌ **Exposed to end users.** Intermediate models should generally not be exposed in the main production schema. They are not intended for output to final targets like dashboards or applications, so it's best to keep them separated from models that are so you can more easily control data governance and discoverability. * ✅ **Materialized ephemerally.** Considering the above, one popular option is to default to intermediate models being materialized [ephemerally](https://docs.getdbt.com/docs/build/materializations.md#ephemeral). This is generally the best place to start for simplicity. It will keep unnecessary models out of your warehouse with minimum configuration. Keep in mind though that the simplicity of ephemerals does translate a bit more difficulty in troubleshooting, as they're interpolated into the models that `ref` them, rather than existing on their own in a way that you can view the output of. * ✅ **Materialized as views in a custom schema with special permissions.** A more robust option is to materialize your intermediate models as views in a specific [custom schema](https://docs.getdbt.com/docs/build/custom-schemas.md), outside of your main production schema. This gives you added insight into development and easier troubleshooting as the number and complexity of your models grows, while remaining easy to implement and taking up negligible space. Keep your warehouse tidy! There are three interfaces to the organizational knowledge graph we're encoding into dbt and folder structure of our codebase, and the output into the warehouse. As such, it's really important that we consider that output intentionally! Think of the schemas, tables, and views we're creating in the warehouse as *part of the UX,* in addition to the dashboards, ML, apps, and other use cases you may be targeting for the data. Ensuring that our output is named and grouped well, and that models not intended for broad use are either not materialized or built into special areas with specific permissions is crucial to achieving this. * Intermediate models' purposes, as these serve to break up complexity from our marts models, can take as many forms as [data transformation](https://www.getdbt.com/analytics-engineering/transformation/) might require. Some of the most common use cases of intermediate models include: * ✅ **Structural simplification.** Bringing together a reasonable number (typically 4 to 6) of entities or concepts (staging models, or perhaps other intermediate models) that will be joined with another similarly purposed intermediate model to generate a mart — rather than have 10 joins in our mart, we can join two intermediate models that each house a piece of the complexity, giving us increased readability, flexibility, testing surface area, and insight into our components. * ✅ **Re-graining.** Intermediate models are often used to fan out or collapse models to the right composite grain — if we're building a mart for `order_items` that requires us to fan out our `orders` based on the `quantity` column, creating a new single row for each item, this would be ideal to do in a specific intermediate model to maintain clarity in our mart and more easily view that our grain is correct before we mix it with other components. * ✅ **Isolating complex operations.** It's helpful to move any particularly complex or difficult to understand pieces of logic into their own intermediate models. This not only makes them easier to refine and troubleshoot, but simplifies later models that can reference this concept in a more clearly readable way. For example, in the `quantity` fan out example above, we benefit by isolating this complex piece of logic so we can quickly debug and thoroughly test that transformation, and downstream models can reference `order_items` in a way that's intuitively easy to grasp. Narrow the DAG, widen the tables. Until we get to the marts layer and start building our various outputs, we ideally want our DAG to look like an arrowhead pointed right. As we move from source-conformed to business-conformed, we're also moving from numerous, narrow, isolated concepts to fewer, wider, joined concepts. We're bringing our components together into wider, richer concepts, and that creates this shape in our DAG. This way when we get to the marts layer we have a robust set of components that can quickly and easily be put into any configuration to answer a variety of questions and serve specific needs. One rule of thumb to ensure you're following this pattern on an individual model level is allowing multiple *inputs* to a model, but **not** multiple *outputs*. Several arrows going *into* our post-staging models is great and expected, several arrows coming *out* is a red flag. There are absolutely situations where you need to break this rule, but it's something to be aware of, careful about, and avoid when possible. --- ### Intro to dbt Mesh #### What is dbt Mesh? Organizations of all sizes rely upon dbt to manage their data transformations, from small startups to large enterprises. At scale, it can be challenging to coordinate all the organizational and technical requirements demanded by your stakeholders within the scope of a single dbt project. To date, there also hasn't been a first-class way to effectively manage the dependencies, governance, and workflows between multiple dbt projects. That's where **Mesh** comes in - empowering data teams to work *independently and collaboratively*; sharing data, code, and best practices without sacrificing security or autonomy. Mesh is not a single product - it is a pattern enabled by a convergence of several features in dbt: * **[Cross-project references](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#how-to-write-cross-project-ref)** - this is the foundational feature that enables the multi-project deployments. `{{ ref() }}`s now work across dbt projects on Enterprise and Enterprise+ plans. * **[Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md)** - dbt's metadata-powered documentation platform, complete with full, cross-project lineage. * **Governance** - dbt's governance features allow you to manage access to your dbt models both within and across projects. * **[Groups](https://docs.getdbt.com/docs/mesh/govern/model-access.md#groups)** - With groups, you can organize nodes in your dbt DAG that share a logical connection (for example, by functional area) and assign an owner to the entire group. * **[Access](https://docs.getdbt.com/docs/mesh/govern/model-access.md#access-modifiers)** - access configs allow you to control who can reference models. * **[Model Versions](https://docs.getdbt.com/docs/mesh/govern/model-versions.md)** - when coordinating across projects and teams, we recommend treating your data models as stable APIs. Model versioning is the mechanism to allow graceful adoption and deprecation of models as they evolve. * **[Model Contracts](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md)** - data contracts set explicit expectations on the shape of the data to ensure data changes upstream of dbt or within a project's logic don't break downstream consumers' data products. #### When is the right time to use dbt Mesh? The multi-project architecture helps organizations with mature, complex transformation workflows in dbt increase the flexibility and performance of their dbt projects. If you're already using dbt and your project has started to experience any of the following, you're likely ready to start exploring this paradigm: * The **number of models** in your project is degrading performance and slowing down development. * Teams have developed **separate workflows** and need to decouple development from each other. * Teams are experiencing **communication challenges**, and the reliability of some of your data products has started to deteriorate. * **Security and governance** requirements are increasing and would benefit from increased isolation. dbt is designed to coordinate the features above and simplify the complexity to solve for these problems. If you're just starting your dbt journey, don't worry about building a multi-project architecture right away. You can *incrementally* adopt the features in this guide as you scale. The collection of features work effectively as independent tools. Familiarizing yourself with the tooling and features that make up a multi-project architecture, and how they can apply to your organization will help you make better decisions as you grow. For additional information, refer to the [Mesh FAQs](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-5-faqs.md). #### Learning goals * Understand the **purpose and tradeoffs** of building a multi-project architecture. * Develop an intuition for various **Mesh patterns** and how to design a multi-project architecture for your organization. * Establish recommended steps to **incrementally adopt** these patterns in your dbt implementation. tip To help you get started, check out our [Quickstart with Mesh](https://docs.getdbt.com/guides/mesh-qs.md) or our online [Mesh course](https://learn.getdbt.com/courses/dbt-mesh) to learn more! --- ### Intro to the dbt Semantic Layer tip Note that this best practices guide doesn't yet use the [new YAML specification](https://docs.getdbt.com/docs/build/latest-metrics-spec.md). We're working on updating this guide to use the new spec and file structure soon! To read more about the new spec, see [Creating metrics](https://docs.getdbt.com/docs/build/metrics-overview.md). Flying cars, hoverboards, and true self-service analytics: this is the future we were promised. The first two might still be a few years out, but real self-service analytics is here today. With dbt's Semantic Layer, you can resolve the tension between accuracy and flexibility that has hampered analytics tools for years, empowering everybody in your organization to explore a shared reality of metrics. Best of all for analytics engineers, building with these new tools will significantly [DRY](https://docs.getdbt.com/terms/dry) up and simplify your codebase. As you'll see, the deep interaction between your dbt models and the Semantic Layer make your dbt project the ideal place to craft your metrics. #### Learning goals * ❓ Understand the **purpose and capabilities** of the **Semantic Layer**, particularly MetricFlow as the engine that powers it. * 🧱 Familiarity with the core components of MetricFlow — **semantic models and metrics** — and how they work together. * 🔁 Know how to **refactor** dbt models for the Semantic Layer. * 🏅 Aware of **best practices** to take maximum advantage of the Semantic Layer. #### Guide structure overview 1. Getting **setup** in your dbt project. 2. Building a **semantic model** and its fundamental parts: **entities, dimensions, and measures**. 3. Building a **metric**. 4. Defining **advanced metrics**: `ratio` and `derived` types. 5. **File and folder structure**: establishing a system for naming things. 6. **Refactoring** marts and roll-ups for the Semantic Layer. 7. Review **best practices**. If you're ready to ship your users more power and flexibility with less code, let's dive in! info MetricFlow is the engine for defining metrics in dbt and one of the key components of the [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md). It handles SQL query construction and defines the specification for dbt semantic models and metrics. To fully experience the Semantic Layer, including the ability to query dbt metrics via external integrations, you'll need a [dbt Starter, Enterprise, or Enterprise+ accounts](https://www.getdbt.com/pricing/). Refer to [Semantic Layer FAQs](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-faqs.md) for more information. --- ### Lambda views for near real-time dashboards Snowflake examples ahead This page uses Snowflake for code examples, but you can adapt the lambda view pattern to other warehouses. A lambda view pattern combines a batch / incremental fact table with a small near real-time (NRT) slice of very recent data and exposes them through a single view. This is a legacy-but-still-useful pattern some teams have used to deliver near real‑time operational dashboards on top of dbt and a warehouse. #### When to use lambda views * You need fresher reads than your normal incremental schedule, but * You can't (or don't want to) use [dynamic tables](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#dynamic-tables) or [materialized views](https://docs.getdbt.com/docs/build/materializations.md#materialized-view), or you want to keep logic entirely in dbt SQL. The examples used in this page assume the following setup: ##### Assumptions The examples used in this page assume the following setup: * Raw events land continuously into `raw.events` using your warehouse's streaming ingestion feature (like Snowpipe, Databricks Auto Loader, Kafka, or a similar ingestion mechanism). * You already maintain an [incremental fact table](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/2-incremental-patterns.md#incremental-merge-from-append-only-tables) that is rebuilt every few minutes using `incremental_strategy='merge'`. * Most dashboards are fine reading from that incremental table, but a small set of operational dashboards want "as‑of‑now" data (for example, the last few minutes of events). ##### How this pattern works * The base incremental table is rebuilt every few minutes using `incremental_strategy='merge'`. * The NRT view is a view that selects only events newer than the max `event_ts` already persisted in the base incremental table. * The lambda view `UNION ALL`s the base table and the NRT view, de-duplicating rows based on primary key semantics. Downstream BI or dashboards query only the lambda view. #### Base incremental table You can reuse the incremental `merge` from [Snowflake pattern 1](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/2-incremental-patterns.md#incremental-merge-from-append-only-tables) as your base table; for completeness: ```sql -- models/fct_events.sql {{ config( materialized = 'incremental', incremental_strategy = 'merge', unique_key = 'event_id', cluster_by = ['event_date'], snowflake_warehouse = 'TRANSFORM_WH' ) }} with source_events as ( select event_id, event_ts::timestamp_ntz as event_ts, to_date(event_ts) as event_date, user_id, event_type, payload from {{ source('raw', 'events') }} {% if is_incremental() %} -- Only pull new/changed rows since last successful load where event_ts > (select max(event_ts) from {{ this }}) {% endif %} ) select * from source_events; ``` Schedule this model to run, for example, every 5–15 minutes as part of your near real‑time job. #### NRT view: rows more recent than the base table The NRT view returns only events with `event_ts` greater than the maximum timestamp in the base table, so there is no overlap or double counting: ```sql -- models/fct_events_nrt.sql {{ config( materialized = 'view' ) }} with base_max as ( select max(event_ts) as max_event_ts from {{ ref('fct_events') }} ), fresh_events as ( select e.event_id, e.event_ts::timestamp_ntz as event_ts, to_date(e.event_ts) as event_date, e.user_id, e.event_type, e.payload from {{ source('raw', 'events') }} as e cross join base_max where e.event_ts > base_max.max_event_ts ) select * from fresh_events; ``` Characteristics: * No scheduling required — it's just a view over `raw.events` filtered by `max(event_ts)` from `fct_events`. * Every query against `fct_events_nrt` scans only "since last batch" data, which should be a small time window (for example, a few minutes or hours, depending on your job cadence). ##### Lambda view: single read path for BI The lambda view combines historical data from the base incremental table with the most recent events from the NRT view. ```sql -- models/fct_events_lambda.sql {{ config( materialized = 'view' ) }} select event_id, event_ts, event_date, user_id, event_type, payload from {{ ref('fct_events') }} union all select event_id, event_ts, event_date, user_id, event_type, payload from {{ ref('fct_events_nrt') }}; ``` Point your BI tools and dashboards to `analytics.fct_events_lambda`. Most data comes from the pre-computed incremental table, while the most recent events (since the last dbt run) come from a live query against `raw.events`. This approach is outlined in [this original dbt lambda view blog post](https://discourse.getdbt.com/t/how-to-create-near-real-time-models-with-just-dbt-sql/1457) which describes how teams like JetBlue wired near real‑time operational dashboards on Snowflake and dbt. #### Considerations Take the following into consideration when using this pattern: * **Cost profile** * Every query against `fct_events_lambda` must read the NRT slice from `raw.events` in addition to the base table. * Use this pattern only for truly operational dashboards that justify the extra per‑query cost. * **Freshness** * Freshness is bounded by: * Your dbt incremental job frequency (age of `fct_events`), plus * Ingestion latency into `raw.events` (Snowpipe / streaming layer). * **Complexity vs alternatives** * For many modern Snowflake implementations, a [dynamic table](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#dynamic-tables) or [materialized view](https://docs.getdbt.com/docs/build/materializations.md#materialized-view) with a small `target_lag` can provide similar "always within X minutes" service level agreements with less custom SQL and warehouse‑managed incremental logic. * Lambda views are best positioned as an *advanced / legacy pattern* you can still use for when you: * Want all logic in dbt SQL * Lack the right warehouse feature in your environment * Are extending an existing implementation already built this way --- ### Marts: Business-defined entities info Our guidance here diverges if you use the Semantic Layer. In a project without the Semantic Layer we recommend you denormalize heavily, per the best practices below. On the other hand, if you're using the Semantic Layer, we want to stay as normalized as possible to allow MetricFlow the most flexibility. See [The dbt Semantic Layer and marts](#the-dbt-semantic-layer-and-marts) for more information. This is the layer where everything comes together and we start to arrange all of our atoms (staging models) into full-fledged cells that have identity and purpose. We sometimes like to call this the *entity* *layer* or *concept layer*, to emphasize that all our marts are meant to represent a specific entity or concept at its unique grain. For instance, an order, a customer, a territory, a click event, a payment — each of these would be represented with a distinct mart, and each row would represent a discrete instance of these concepts. Unlike in a traditional Kimball star schema though, in modern data warehousing — where storage is cheap and compute is expensive — we'll happily borrow and add any and all data from other concepts that are relevant to answering questions about the mart's core entity. Building the same data in multiple places, as we do with `orders` in our `customers` mart example below, is more efficient in this paradigm than repeatedly rejoining these concepts (this is a basic definition of denormalization in this context). Let's take a look at how we approach this first layer intended expressly for exposure to end users. ##### Marts: Files and folders The marts layer in our example project contains one model per core business entity: ```shell models/marts ├── customers.sql ├── customers.yml ├── locations.sql ├── locations.yml ├── order_items.sql ├── order_items.yml ├── orders.sql ├── orders.yml ├── products.sql ├── products.yml ├── supplies.sql └── supplies.yml ``` ✅ **Group by department or area of concern.** If you have fewer than 10 or so marts you may not have much need for subfolders, so as with the intermediate layer, don't over-optimize too early. If you do find yourself needing to insert more structure and grouping though, use useful business concepts here. In our marts layer, we're no longer worried about source-conformed data, so grouping by departments (marketing, finance, etc.) is the most common structure at this stage. ✅ **Name by entity.** Use plain English to name the file based on the concept that forms the grain of the mart's `customers`, `orders`. Marts that don't include any time-based rollups (pure marts) should not have a time dimension (`orders_per_day`) here, typically best captured via metrics. ❌ **Build the same concept differently for different teams.** `finance_orders` and `marketing_orders` is typically considered an anti-pattern. There are, as always, exceptions — a common pattern we see is that, finance may have specific needs, for example reporting revenue to the government in a way that diverges from how the company as a whole measures revenue day-to-day. Just make sure that these are clearly designed and understandable as *separate* concepts, not departmental views on the same concept: `tax_revenue` and `revenue` not `finance_revenue` and `marketing_revenue`. ##### Marts: Models Finally we'll take a look at the best practices for models within the marts directory by examining two of our marts models. These are the business-conformed — that is, crafted to our vision and needs — entities we've been bringing these transformed components together to create. ```sql -- orders.sql with orders as ( select * from {{ ref('stg_orders') }} ), order_items as ( select * from {{ ref('order_items') }} ), order_items_summary as ( select order_id, sum(supply_cost) as order_cost, sum(product_price) as order_items_subtotal, count(order_item_id) as count_order_items, sum( case when is_food_item then 1 else 0 end ) as count_food_items, sum( case when is_drink_item then 1 else 0 end ) as count_drink_items from order_items group by 1 ), compute_booleans as ( select orders.*, order_items_summary.order_cost, order_items_summary.order_items_subtotal, order_items_summary.count_food_items, order_items_summary.count_drink_items, order_items_summary.count_order_items, order_items_summary.count_food_items > 0 as is_food_order, order_items_summary.count_drink_items > 0 as is_drink_order from orders left join order_items_summary on orders.order_id = order_items_summary.order_id ) select * from compute_booleans ``` ```sql -- customers.sql with customers as ( select * from {{ ref('stg_customers') }} ), orders as ( select * from {{ ref('orders') }} ), customer_orders_summary as ( select orders.customer_id, count(distinct orders.order_id) as count_lifetime_orders, count(distinct orders.order_id) > 1 as is_repeat_buyer, min(orders.ordered_at) as first_ordered_at, max(orders.ordered_at) as last_ordered_at, sum(orders.subtotal) as lifetime_spend_pretax, sum(orders.tax_paid) as lifetime_tax_paid, sum(orders.order_total) as lifetime_spend from orders group by 1 ), joined as ( select customers.*, customer_orders_summary.count_lifetime_orders, customer_orders_summary.first_ordered_at, customer_orders_summary.last_ordered_at, customer_orders_summary.lifetime_spend_pretax, customer_orders_summary.lifetime_tax_paid, customer_orders_summary.lifetime_spend, case when customer_orders_summary.is_repeat_buyer then 'returning' else 'new' end as customer_type from customers left join customer_orders_summary on customers.customer_id = customer_orders_summary.customer_id ) select * from joined ``` * ✅ **Materialized as tables or incremental models.** Once we reach the marts layer, it's time to start building not just our logic into the warehouse, but the data itself. This gives end users much faster performance for these later models that are actually designed for their use, and saves us costs recomputing these entire chains of models every time somebody refreshes a dashboard or runs a regression in python. A good general rule of thumb regarding materialization is to always start with a view (as it takes up essentially no storage and always gives you up-to-date results), once that view takes too long to practically *query*, build it into a table, and finally once that table takes too long to *build* and is slowing down your runs, [configure it as an incremental model](https://docs.getdbt.com/docs/build/incremental-models.md). As always, start simple and only add complexity as necessary. The models with the most data and compute-intensive transformations should absolutely take advantage of dbt's excellent incremental materialization options, but rushing to make all your marts models incremental by default will introduce superfluous difficulty. We recommend reading this [classic post from Tristan on the limits of incremental modeling](https://discourse.getdbt.com/t/on-the-limits-of-incrementality/303). * ✅ **Wide and denormalized.** Unlike old school warehousing, in the modern data stack storage is cheap and it's compute that is expensive and must be prioritized as such, packing these into very wide denormalized concepts that can provide everything somebody needs about a concept as a goal. * ❌ **Too many joins in one mart.** One good rule of thumb when building dbt transformations is to avoid bringing together too many concepts in a single mart. What constitutes 'too many' can vary. If you need to bring 8 staging models together with nothing but simple joins, that might be fine. Conversely, if you have 4 concepts you're weaving together with some complex and computationally heavy window functions, that could be too much. You need to weigh the number of models you're joining against the complexity of the logic within the mart, and if it's too much to read through and build a clear mental model of then look to modularize. While this isn't a hard rule, if you're bringing together more than 4 or 5 concepts to create your mart, you may benefit from adding some [intermediate models](https://docs.getdbt.com/best-practices/how-we-structure/3-intermediate.md) for added clarity. Two intermediate models that bring together three concepts each, and a mart that brings together those two intermediate models, will typically result in a much more readable chain of logic than a single mart with six joins. * ✅ **Build on separate marts thoughtfully.** While we strive to preserve a narrowing DAG up to the marts layer, once here things may start to get a little less strict. A common example is passing information between marts at different grains, as we saw above, where we bring our `orders` mart into our `customers` marts to aggregate critical order data into a `customer` grain. Now that we're really 'spending' compute and storage by actually building the data in our outputs, it's sensible to leverage previously built resources to speed up and save costs on outputs that require similar data, versus recomputing the same views and CTEs from scratch. The right approach here is heavily dependent on your unique DAG, models, and goals — it's just important to note that using a mart in building another, later mart is okay, but requires careful consideration to avoid wasted resources or circular dependencies. Marts are entity-grained. The most important aspect of marts is that they contain all of the useful data about a *particular entity* at a granular level. That doesn't mean we don't bring in lots of other entities and concepts, like tons of `user` data into our `orders` mart, we do! It just means that individual `orders` remain the core grain of our table. If we start grouping `users` and `orders` along a [date spine](https://github.com/dbt-labs/dbt-utils#date_spine-source), into something like `user_orders_per_day`, we're moving past marts into *metrics*. ##### Marts: Other considerations * **Troubleshoot via tables.** While stacking views and ephemeral models up until our marts — only building data into the warehouse at the end of a chain when we have the models we really want end users to work with — is ideal in production, it can present some difficulties in development. Particularly, certain errors may seem to be surfacing in our later models that actually stem from much earlier dependencies in our model chain (ancestor models in our DAG that are built before the model throws the errors). If you're having trouble pinning down where or what a database error is telling you, it can be helpful to temporarily build a specific chain of models as tables so that the warehouse will throw the error where it's actually occurring. ##### The dbt Semantic Layer and marts Our structural recommendations are impacted quite a bit by whether or not you're using the Semantic Layer. If you're using the Semantic Layer, we recommend a more normalized approach to your marts. If you're not using the Semantic Layer, we recommend a more denormalized approach that has become typical in dbt projects. For the full list of recommendations on structure, naming, and organization in the Semantic Layer, check out the [How we build our metrics](https://docs.getdbt.com/best-practices/how-we-build-our-metrics/semantic-layer-1-intro.md) guide, particularly the [Refactoring an existing rollup](https://docs.getdbt.com/best-practices/how-we-build-our-metrics/semantic-layer-8-refactor-a-rollup.md) section. --- ### Materializations best practices What *really* happens when you type `dbt build`? Contrary to popular belief, a crack team of microscopic data elves do *not* construct your data row by row, although the truth feels equally magical. This guide explores the real answer to that question, with an introductory look at the objects that get built into your warehouse, why they matter, and how dbt knows what to build. Learn by video! For video tutorials on Snapshots, go to dbt Learn and check out the [Snapshots course](https://learn.getdbt.com/courses/snapshots). The configurations that tell dbt how to construct these objects are called *materializations,* and knowing how to use them is a crucial skill for effective analytics engineering. When you’ve completed this guide, you will have that ability to use the three core materializations that cover most common analytics engineering situations. info 😌 **Materializations abstract away DDL and DML**. Typically in raw SQL- or python-based [data transformation](https://www.getdbt.com/analytics-engineering/transformation/), you have to write specific imperative instructions on how to build or modify your data objects. dbt’s materializations make this declarative, we tell dbt how we want things to be constructed and it figures out how to do that given the unique conditions and qualities of our warehouse. ##### Learning goals By the end of this guide you should have a solid understanding of: * 🛠️ what **materializations** are * 👨‍👨‍👧 how the three main materializations that ship with dbt — **table**, **view**, and **incremental** — differ * 🗺️ **when** and **where** to use specific materializations to optimize your development and production builds * ⚙️ how to **configure materializations** at various scopes, from an individual model to entire folder ##### Prerequisites * 📒 You’ll want to have worked through the [quickstart guide](https://docs.getdbt.com/guides.md) and have a project setup to work through these concepts. * 🏃🏻‍♀️ Concepts like dbt runs, `ref()` statements, and models should be familiar to you. * 🔧 \[**Optional**] Reading through the [How we structure our dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md) Guide will be beneficial for the last section of this guide, when we review best practices for materializations using the dbt project approach of staging models and marts. ##### Guiding principle We’ll explore this in-depth throughout, but the basic guideline is **start as simple as possible**. We’ll follow a tiered approached, only moving up a tier when it’s necessary. * 🔍 **Start with a view.** When the view gets too long to *query* for end users, * ⚒️ **Make it a table.** When the table gets too long to *build* in your dbt Jobs, * 📚 **Build it incrementally.** That is, layer the data on in chunks as it comes in. --- ### More advanced metrics #### More advanced metric types We're not limited to just passing measures through to our metrics, we can also *combine* measures to model more advanced metrics. * 🍊 **Ratio** metrics are, as the name implies, about **comparing two metrics as a numerator and a denominator** to form a new metric, for instance the percentage of order items that are food items instead of drinks. * 🧱 **Derived** metrics are when we want to **write an expression** that calculates a metric **using multiple metrics**. A classic example here is our gross profit calculated by subtracting costs from revenue. * ➕ **Cumulative** metrics calculate all of a **measure over a given window**, such as the past week, or if no window is supplied, the all-time total of that measure. #### Ratio metrics * 🔢 We need to establish one measure that will be our **numerator**, and one that will be our **denominator**. * 🥪 Let's calculate the **percentage** of our Jaffle Shop revenue that **comes from food items**. * 💰 We already have our denominator, revenue, but we'll want to **make a new metric for our numerator** called `food_revenue`. models/marts/order\_items.yml ```yml - name: food_revenue description: The revenue from food in each order. label: Food Revenue type: simple type_params: measure: food_revenue ``` * 📝 Now we can set up our ratio metric. models/marts/order\_items.yml ```yml - name: food_revenue_pct description: The % of order revenue from food. label: Food Revenue % type: ratio type_params: numerator: food_revenue denominator: revenue ``` #### Derived metrics * 🆙 Now let's really have some fun. One of the most important metrics for any business is not just revenue, but *revenue growth*. Let's use a derived metric to build month-over-month revenue. * ⚙️ A derived metric has a couple key components: * 📚 A list of metrics to build on. These can be manipulated and filtered in various way, here we'll use the `offset_window` property to lag by a month. * 🧮 An expression that performs a calculation with these metrics. * With these parts we can assemble complex logic that would otherwise need to be 'frozen' in logical models. models/marts/order\_items.yml ```yml - name: revenue_growth_mom description: "Percentage growth of revenue compared to 1 month ago. Excluded tax" type: derived label: Revenue Growth % M/M type_params: expr: (current_revenue - revenue_prev_month) * 100 / revenue_prev_month metrics: - name: revenue alias: current_revenue - name: revenue offset_window: 1 month alias: revenue_prev_month ``` #### Cumulative metrics * ➕ Lastly, lets build a **cumulative metric**. In keeping with our theme of business priorities, let's continue with revenue and build an **all-time revenue metric** for any given time window. * 🪟 All we need to do is indicate the type is `cumulative` and not supply a `window` in the `type_params`, which indicates we want cumulative for the entire time period our end users select. models/marts/order\_items.yml ```yml - name: cumulative_revenue description: The cumulative revenue for all orders. label: Cumulative Revenue (All Time) type: cumulative type_params: measure: revenue ``` --- ### Near real-time data in dbt By design, dbt is batch-oriented with jobs having a defined start and end time. But did you know that you can also use dbt to get near real-time data by combining your data warehouse's continuous ingestion with frequent dbt transformations? This guide covers multiple patterns for achieving near real-time data freshness with dbt: 1. [Incremental patterns](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/2-incremental-patterns.md) — `merge` strategies, Change Data Capture (CDC), and microbatch processing 2. [Warehouse-native features](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/3-warehouse-native-features.md) — When to use dynamic tables and materialized views 3. [Lambda views pattern](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/4-lambda-views.md) — Combining batch and real-time data in a single view 4. [Views-only pattern](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/5-views-only-pattern.md) - Maximum freshness for lightweight transformations 5. [Operational considerations](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/6-operational-considerations.md) — Challenges, risks, and cost management Each pattern includes practical code examples, use cases, and tradeoffs to help you choose the right approach. Anyone can use this guide, but it's primarily for data engineers and architects who want to achieve near real-time data freshness with dbt. #### Where does dbt fit? There are two main ways to use dbt to get near real-time data: * For near real-time (5 - 15 minutes) — dbt excels at this and is well-suited for most operational dashboards. * For true real-time (sub-second) — This requires dedicated streaming databases (ClickHouse, Materialize, Rockset, and so on) in front of or alongside dbt; dbt still owns “analytic” tables and history but not the ultra‑low‑latency read path. #### How dbt achieves near real-time data To achieve real-time data with dbt, we recommend using a two-layer architecture: ###### Ingestion layer Continuous data landing using your data warehouse's streaming ingestion features. Streaming ingestion features such as [streaming tables](https://docs.databricks.com/en/sql/load-data-streaming-table.html), [Snowpipe](https://docs.snowflake.com/en/user-guide/snowpipe-streaming/data-load-snowpipe-streaming-overview), or [Storage Write API](https://docs.cloud.google.com/bigquery/docs/write-api-streaming) work well for this. To find streaming ingestion features for your warehouse, refer to the [additional resources](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/3-warehouse-native-features.md#resources-by-warehouse) section. ###### dbt transformation layer Run dbt every few minutes to transform the data, and use materialized views or dynamic tables for the lowest-latency reporting. Specific transformation approaches include: * [Incremental models](https://docs.getdbt.com/docs/build/incremental-models-overview.md) with merge or append strategies * [Microbatch incremental strategy](https://docs.getdbt.com/docs/build/incremental-microbatch.md) for large time-series tables * Jobs scheduled very frequently (like every 5 minutes) * [Dynamic tables](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#dynamic-tables) or [materialized views](https://docs.getdbt.com/docs/build/materializations.md#materialized-view) with short refresh intervals #### Key recommendations The following are some key recommendations to help you achieve near real-time data freshness with dbt: * Ingest data continuously: Use your warehouse's native streaming or micro-batch ingestion to land raw data as soon as it arrives. * Transform with dbt on a frequent schedule: Schedule dbt jobs to run as often as your business needs allow (for example, every 1–15 minutes). Balance freshness with cost and resource constraints. * Materialized views and dynamic tables: For the lowest-latency reporting, use materialized views or dynamic tables. These can be refreshed as frequently as every minute. * Incremental models and microbatching: Use dbt's incremental models to process only new or changed data, keeping transformations efficient and scalable. * Decouple ingestion from transformation: Keep data acquisition and transformation flows separate. This allows you to optimize each independently. * Monitor and test data freshness: Implement data quality checks and freshness monitoring to ensure your near real-time pipelines deliver accurate, up-to-date results. * Cost and complexity considerations: Running dbt jobs more frequently drives up compute costs and operational complexity. Always weigh the business value against these trade-offs. --- ### Now it's your turn #### BYO Styles Now that you've seen how we style our dbt projects, it's time to build your own. Feel free to copy this guide and use it as a template for your own project. If you do, we'd love to hear about it! Reach out to us on [the Community Forum](https://discourse.getdbt.com/c/show-and-tell/22) or [Slack](https://www.getdbt.com/community) to share your style guide. We recommend co-locating your style guide with your code to make sure contributors can easily follow it. If you're using GitHub, you can add your style guide to your repository's wiki, or include it in your README. #### Pre-commit hooks You can use [pre-commit hooks](https://pre-commit.com/) to automatically check your code for style violations (and often fix them automagically) before you commit. This is a great way to make sure all contributors follow your style guide. We recommend implementing this once you've settled on and published your style guide, and your codebase is conforming to it. This will ensure that all future commits follow the style guide. You can find an excellent set of open source pre-commit hooks for dbt from the community [here in the dbt-checkpoint project](https://github.com/dbt-checkpoint/dbt-checkpoint). #### dbt Project Evaluator The [`dbt_project_evaluator`](https://github.com/dbt-labs/dbt-project-evaluator) is a package that ensures compliance to [dbt's style guide and best practices](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md). The `dbt_project_evaluator` package highlights areas of a dbt project that are not aligned with dbt's best practices and provides recommendations on how to improve a project. This enables analytics engineers to determine exactly where their projects deviated from dbt's best practices and improve their projects on their own. The `dbt_project_evaluator` package covers the following categories: * Modeling * Testing * Documentation * Structure * Performance * Governance For more information, see [Introducing the dbt\_project\_evaluator: Automatically evaluate your dbt project for alignment with best practices](https://docs.getdbt.com/blog/align-with-dbt-project-evaluator). #### Style guide template ```markdown # dbt Example Style Guide ## SQL Style - Use lowercase keywords. - Use trailing commas. ## Model Organization Our models (typically) fit into two main categories:\ - Staging — Contains models that clean and standardize data. - Marts — Contains models which combine or heavily transform data. Things to note: - There are different types of models that typically exist in each of the above categories. See Model Layers for more information. - Read How we structure our dbt projects for an example and more details around organization. ## Model Layers - Only models in `staging` should select from sources. - Models not in the `staging` folder should select from refs. ## Model File Naming and Coding - All objects should be plural. Example: `stg_orders.sql` vs. `stg_order.sql` - All models should use the naming convention `___`. See this article for more information. - Models in the **staging** folder should use the source's name as the `` and the entity name as the `additional_context`. In a single-source project like Jaffle Shop, `stg_orders.sql` and `stg_customers.sql` are clear enough without the source prefix. Examples: - seed_snowflake_spend.csv - stg_orders.sql - stg_customers.sql - stg_salesforce__customers.sql - int_customers__unioned.sql - fct_orders.sql - Schema, table, and column names should be in `snake_case`. - Limit the use of abbreviations that are related to domain knowledge. An onboarding employee will understand `current_order_status` better than `current_os`. - Use names based on the _business_ rather than the source terminology. - Each model should have a primary key to identify the unique row and should be named `_id`. For example, `account_id`. This makes it easier to know what `id` is referenced in downstream joined models. - For `base` or `staging` models, columns should be ordered in categories, where identifiers are first and date/time fields are at the end. - Date/time columns should be named according to these conventions: - Timestamps: `_at` Format: UTC Example: `created_at` - Dates: `_date` Format: Date Example: `created_date` - Booleans should be prefixed with `is_` or `has_`. Example: `is_active_customer` and `has_admin_access` - Price/revenue fields should be in decimal currency (for example, `19.99` for $19.99; many app databases store prices as integers in cents). If a non-decimal currency is used, indicate this with suffixes. For example, `price_in_cents`. - Avoid using reserved words (such as these for Snowflake) as column names. - Consistency is key! Use the same field names across models where possible. For example, a key to the `customers` table should be named `customer_id` rather than `user_id`. ## Model Configurations - Model configurations at the folder level should be considered (and if applicable, applied) first. - More specific configurations should be applied at the model level using one of these methods. - Models within the `marts` folder should be materialized as `table` or `incremental`. - By default, `marts` should be materialized as `table` within `dbt_project.yml`. - If switching to `incremental`, this should be specified in the model's configuration. ## Testing - At a minimum, `unique` and `not_null` tests should be applied to the expected primary key of each model. ## CTEs For more information about why we use so many CTEs, read this glossary entry. - Where performance permits, CTEs should perform a single, logical unit of work. - CTE names should be as verbose as needed to convey what they do. - CTEs with confusing or noteable logic should be commented with SQL comments as you would with any complex functions and should be located above the CTE. - CTEs duplicated across models should be pulled out and created as their own models. ``` --- ### Operational considerations for near real-time data Teams that implement very high-frequency dbt jobs tend to run into a consistent set of challenges, both at the dbt scheduler layer and in the warehouse itself. Treat near real-time as a premium service Near real-time service level agreements (SLAs) require premium resources and add significant operational overhead. Pressure-test whether the business really needs minute-level freshness before committing. #### Over-scheduled jobs and queue management If a job's run duration is longer than its schedule frequency, the job becomes over-scheduled. The queue grows faster than the scheduler can process runs, and dbt platform will start cancelling queued runs to avoid an ever-expanding backlog. This is easy to hit with near real-time patterns if your incremental build time creeps up (more models, more tests, more data) but the cron schedule stays aggressive (for example, every 2–5 minutes). **Example scenario:** * Your job is scheduled to run every 5 minutes. * The job typically takes 6-7 minutes to complete. * New runs queue up while previous runs are still executing. * dbt platform starts cancelling queued runs to prevent infinite backlog. When this happens, remediation is non-trivial. You need to either refactor the job to run faster (prune model selection, adjust threads, optimize SQL) or relax the schedule and accept a looser freshness SLA. ###### Related scheduler constraints * Run slots limit how many jobs can run concurrently. Frequent near real-time jobs can starve other deployment jobs if slot usage isn't planned. * The scheduler runs distinct executions of the same job serially. If one run is still in progress when the next cron fires, the second run must wait (or be cancelled in an over-scheduled scenario). #### Warehouse cost and utilization As the gap between job runtime and schedule interval shrinks, your warehouse is effectively running continuously to keep up with back-to-back transformation windows. **Cost scaling example:** * Daily job: Warehouse runs 30 min/day = ~2% utilization * Hourly job: Warehouse runs 30 min × 24 = 12 hours/day = 50% utilization * 5-minute job: Warehouse runs nearly 24/7 = ~100% utilization On platforms like Snowflake, ingestion options like Snowpipe for high-volume real-time feeds can be very expensive (cost per 1,000 files plus compute). Warehouse-managed options for freshness (for example, dynamic tables and materialized views) can also be harder to predict and monitor from a cost perspective, especially when underlying data is changing frequently. The net effect: you should treat near real-time SLAs as a premium service and pressure-test whether the business really needs minute-level freshness on each workload. #### Lambda view DAG complexity and correctness If you're using the [lambda views pattern](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/4-lambda-views.md), you face additional complexity: * **Duplicated logic**: You either centralize SQL in macros (more DRY, less readable) or duplicate the same transformations in both history (HIST) and NRT flows (more readable, more to maintain). * **Complex DAGs**: Every "product" model now has at least three artifacts (HIST table, NRT view, lambda union), plus supporting upstream layers. * **Materialization brittleness**: The pattern depends on specific materializations (views vs incrementals). A seemingly harmless materialization change can break freshness or correctness. On top of that, community experience has surfaced timing gaps between HIST and NRT flows: * Views (NRT) often update much faster than incremental tables. During a run, the NRT side may start filtering on the new `max(event_ts)` before the incremental table has finished loading, producing temporary holes in the unioned lambda view where recent data disappears briefly. * One way to mitigate is to introduce an explicit dependency from NRT to the incremental model (for example, a manual dependency on `{{ ref('fct_events') }}` comment), but this is somewhat brittle and increases coupling. #### Job reliability and resource limits High-frequency jobs are more likely to surface job-level failures: * **Memory limits** * Memory-heavy macros (for example, large `run_query()` results) or big doc-generation steps can hit account-level memory limits. * This causes runs to terminate with "memory limit" errors. * **Auto-deactivation** * A job that fails repeatedly can be auto-deactivated after 100 consecutive failures. * When this happens, scheduled triggers stop until someone manually intervenes. * **Smaller margin for error** * A flaky model, test, or small regression can quickly generate many failed runs. * This creates noisy alerts and can hit the auto-deactivation threshold faster. #### Ingestion architecture dependencies Lambda views and NRT dbt jobs sit on top of your ingestion architecture: * **The dependency** * If ingestion latency or throughput degrades (issues in a task/stream pipeline, backlogs in storage, intermittent Snowpipe delays), the lambda view can only union what has already arrived. * You can't make data fresher than your ingestion layer allows. * **What you end up tuning** * Task cadences and partition strategies in the landing zone * Lambda overlap windows and incremental look-backs * Which sources really need to participate in the NRT path #### Conclusion These challenges are why we position lambda views and ultra-frequent dbt schedules as special-case patterns. They're powerful when you truly need them, but they require deliberate design around scheduler behavior, cost, DAG structure, and ingestion architecture. In many cases, they're better replaced by [dynamic tables](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/3-warehouse-native-features.md#dynamic-tables), [materialized views](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/3-warehouse-native-features.md#materialized-views), or a dedicated streaming stack. --- ### Optimize static analysis for development and deployment important dbt v2 is currently available for installation in: * [Local command line interface (CLI) tools](https://docs.getdbt.com/docs/local/install-dbt.md?version=2) [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") * [VS Code and Cursor with the dbt extension](https://docs.getdbt.com/docs/install-dbt-extension.md) [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") * [dbt platform environments](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md#dbt-v2) Join the conversation in our Community Slack channel [`#dbt-fusion-engine`](https://getdbt.slack.com/archives/C088YCAB6GH). Static analysis helps dbt v2 validate your SQL before it runs. This guide shows how to configure it so you get stronger checks while you develop, and faster, less blocking runs in deployment. This guide explains why using `strict` in development and `baseline` (the default, lighter static analysis mode) in deployment is a valid and recommended pattern, and how to configure it in your dbt v2 project. For more information about modes and features, refer to [About static analysis](https://docs.getdbt.com/docs/build/about-static-analysis.md). #### Why this pattern * **Development:** `strict` mode has the strongest SQL checks before you promote changes, including richer column-level features in the VS Code extension. * **Deployment:** `baseline` skips remote warehouse schema downloads and surfaces findings as warnings, so jobs are less likely to block. That can save compile time (and warehouse cost) in deployment, especially in projects with many sources. Review deployment logs for warnings that `strict` would have raised as errors in development. `strict` can increase compile time because dbt v2 downloads schemas for all sources (including sources your models do not reference). Teams with thousands of sources have seen large differences between `baseline` and `strict`. #### Set the mode with the CLI flag Use the [`--static-analysis`](https://docs.getdbt.com/reference/global-configs/static-analysis-flag.md) flag to set the mode for a single run. **Development:** ```bash dbt compile --static-analysis strict ``` **Deployment:** ```bash dbt compile --static-analysis baseline ``` You can use the same flag with `dbt run` or `dbt build`. If you already have dbt v1 or the platform CLI installed alongside dbt v2, use `dbtf` as the unambiguous dbt v2 command. You can also configure [`static_analysis`](https://docs.getdbt.com/reference/resource-configs/static-analysis.md) per directory or model. Refer to [Configuring `static_analysis`](https://docs.getdbt.com/docs/build/about-static-analysis.md#configuring-static_analysis) for examples. #### Set the mode with an environment variable You can also drive [`static_analysis`](https://docs.getdbt.com/reference/resource-configs/static-analysis.md) from a custom environment variable in `dbt_project.yml`. This is useful when development and deployment share the same project config but set different environment values. dbt\_project.yml ```yml models: my_project: +static_analysis: "{{ env_var('DBT_ENV_STATIC_ANALYSIS', 'baseline') }}" ``` Then set the variable per environment: * Development: `DBT_ENV_STATIC_ANALYSIS=strict` * Deployment: leave unset (defaults to `baseline`), or set `DBT_ENV_STATIC_ANALYSIS=baseline` `DBT_ENV_STATIC_ANALYSIS` is a custom variable name you choose. It is separate from the built-in `DBT_STATIC_ANALYSIS` override used with the CLI flag. For more information about `env_var`, refer to [About env\_var](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md) and [Environment variables](https://docs.getdbt.com/docs/build/environment-variables.md). #### Related docs * [About static analysis](https://docs.getdbt.com/docs/build/about-static-analysis.md) * [`static_analysis` config](https://docs.getdbt.com/reference/resource-configs/static-analysis.md) * [`--static-analysis` flag](https://docs.getdbt.com/reference/global-configs/static-analysis-flag.md) --- ### Refactor an existing rollup #### A new approach Now that we've set the stage, it's time to dig in to the fun and messy part: how do we refactor an existing rollup in dbt into semantic models and metrics? Let's look at the differences we can observe in how we might approach this with MetricFlow supercharging dbt versus how we work without a Semantic Layer. These differences can then inform our structure. * 🍊 In dbt, we tend to create **highly denormalized datasets** that bring **everything you want around a certain entity or process into a single table**. * 💜 The problem is, this **limits the dimensionality available to MetricFlow**. The more we pre-compute and 'freeze' into place, the less flexible our data is. * 🚰 In MetricFlow, we ideally want **highly normalized**, star schema-like data that then allows MetricFlow to shine as a **denormalization engine**. * ∞ Another way to think about this is that instead of moving down a list of requested priorities trying to pre-make as many combinations of our marts as possible — increasing lines of code and complexity — we can **let MetricFlow present every combination possible without specifically coding it**. * 🏗️ To resolve these approaches optimally, we'll need to shift some **fundamental aspects of our modeling strategy**. #### Refactor steps outlined We recommend an incremental implementation process that looks something like this: 1. 👉 Identify **an important output** (a revenue chart on a dashboard for example, and the mart model(s) that supplies this output. 2. 🔍 Examine all the **entities that are components** of this rollup (for instance, an `active_customers_per_week` rollup may include customers, shipping, and product data). 3. 🛠️ **Build semantic models** for all the underlying component marts. 4. 📏 **Build metrics** for the required aggregations in the rollup. 5. 👯 Create a **clone of the output** on top of the Semantic Layer. 6. 💻 Audit to **ensure you get accurate outputs**. 7. 👉 Identify **any other outputs** that point to the rollup and **move them to the Semantic Layer**. 8. ✌️ Put a **deprecation plan** in place for the now extraneous frozen rollup. You would then **continue this process** on other outputs and marts moving down a list of **priorities**. Each model as you go along will be faster and easier as you'll **reuse many of the same components** that will already have been semantically modeled. #### Let's make a `revenue` metric So far we've been working in new pointing at a staging model to simplify things as we build new mental models for MetricFlow. In reality, unless you're implementing MetricFlow in a green-field dbt project, you probably are going to have some refactoring to do. So let's get into that in detail. 1. 📚 Per the above steps, let's say we've identified our target as a revenue rollup that is built on top of `orders` and `order_items`. Now we need to identify all the underlying components, these will be all the 'import' CTEs at the top of these marts. So in the Jaffle Shop project we'd need: `orders`, `order_items`, `products`, `locations`, and `supplies`. 2. 🗺️ We'll next make semantic models for all of these. Let's walk through a straightforward conversion first with `locations`. 3. ⛓️ We'll want to first decide if we need to do any joining to get this into the shape we want for our semantic model. The biggest determinants of this are two factors: * 📏 Does this semantic model **contain measures**? * 🕥 Does this semantic model have a **primary timestamp**? * 🫂 If a semantic model **has measures but no timestamp** (for example, supplies in the example project, which has static costs of supplies), you'll likely want to **sacrifice some normalization and join it on to another model** that has a primary timestamp to allow for metric aggregation. 4. 🔄 If we *don't* need any joins, we'll point our semantic model at the mart. Locations has a `tax_rate` measure and an `opened_date` time dimension, so we can reference the `locations` mart directly here. 5. 🥇 We specify our **primary entity** (based on `location_id`), dimensions (one categorical, `location_name`, and one **primary time dimension** `opened_date`), and lastly our measures, in this case just `average_tax_rate`. models/marts/locations.yml ```yaml semantic_models: - name: locations description: | Location dimension table. The grain of the table is one row per location. model: ref('locations') defaults: agg_time_dimension: opened_date entities: - name: location type: primary expr: location_id dimensions: - name: location_name type: categorical - name: opened_date expr: opened_date type: time type_params: time_granularity: day measures: - name: average_tax_rate description: Average tax rate. expr: tax_rate agg: average ``` #### Semantic and logical interaction Now, let's tackle a thornier situation. Products and supplies both have dimensions and measures but no time dimension. Products has a one-to-one relationship with `order_items`, enriching that table, which is itself just a mapping table of products to orders. Additionally, products have a one-to-many relationship with supplies. The high-level ERD looks like the diagram below. [![](/img/best-practices/semantic-layer/orders_erd.png?v=2)](#) So to calculate, for instance, the cost of ingredients and supplies for a given order, we'll need to do some joining and aggregating, but again we **lack a time dimension for products and supplies**. This is the signal to us that we'll **need to build a logical mart** and point our semantic model at that. tip **dbt 🧡 MetricFlow.** This is where integrating your semantic definitions into your dbt project really starts to pay dividends. The interaction between the logical and semantic layers is so dynamic, you either need to house them in one codebase or facilitate a lot of cross-project communication and dependency. 1. 🎯 Let's aim at, to start, building a table at the `order_items` grain. We can aggregate supply costs up, map over the fields we want from products, such as price, and bring the `ordered_at` timestamp we need over from the orders table. You can see example code, copied below, in `models/marts/order_items.sql`. models/marts/order\_items.sql ```sql with order_items as ( select * from {{ ref('stg_order_items') }} ), orders as ( select * from {{ ref('stg_orders')}} ), products as ( select * from {{ ref('stg_products') }} ), supplies as ( select * from {{ ref('stg_supplies') }} ), order_supplies_summary as ( select product_id, sum(supply_cost) as supply_cost from supplies group by 1 ), joined as ( select order_items.*, orders.ordered_at, products.product_name, products.product_price, products.is_food_item, products.is_drink_item, order_supplies_summary.supply_cost from order_items left join orders on order_items.order_id = orders.order_id left join products on order_items.product_id = products.product_id left join order_supplies_summary on order_items.product_id = order_supplies_summary.product_id ) select * from joined ``` 2. 🏗️ Now we've got a table that looks more like what we want to feed into the Semantic Layer. Next, we'll **build a semantic model on top of this new mart** in `models/marts/order_items.yml`. Again, we'll identify our **entities, then dimensions, then measures**. models/marts/order\_items.yml ```yml semantic_models: - name: order_item defaults: agg_time_dimension: ordered_at description: | Items contatined in each order. The grain of the table is one row per order item. model: ref('order_items') entities: - name: order_item type: primary expr: order_item_id - name: order_id type: foreign expr: order_id - name: product type: foreign expr: product_id dimensions: - name: ordered_at expr: ordered_at type: time type_params: time_granularity: day - name: is_food_item type: categorical - name: is_drink_item type: categorical measures: - name: revenue description: The revenue generated for each order item. Revenue is calculated as a sum of revenue associated with each product in an order. agg: sum expr: product_price - name: food_revenue description: The revenue generated for each order item. Revenue is calculated as a sum of revenue associated with each product in an order. agg: sum expr: case when is_food_item then product_price else 0 end - name: drink_revenue description: The revenue generated for each order item. Revenue is calculated as a sum of revenue associated with each product in an order. agg: sum expr: case when is_drink_item then product_price else 0 end - name: median_revenue description: The median revenue generated for each order item. agg: median expr: product_price ``` 3. 📏 Finally, Let's **build a simple revenue metric** on top of our semantic model now. models/marts/order\_items.yml ```yaml metrics: - name: revenue description: Sum of the product revenue for each order item. Excludes tax. type: simple label: Revenue type_params: measure: revenue ``` #### Checking our work * 🔍 We always start our **auditing** with a `dbt parse` to **ensure our code works** before we examine its output. * 👯 If we're working there, we'll move to trying out an `dbt sl query` that **replicates the logic of the output** we're trying to refactor. * 💸 For our example we want to **audit monthly revenue**, to do that we'd run the query below. ##### Example query ```text dbt sl query --metrics revenue --group-by metric_time__month ``` ##### Example query results ```shell ✔ Success 🦄 - query completed after 1.02 seconds | METRIC_TIME__MONTH | REVENUE | |:---------------------|----------:| | 2016-09-01 00:00:00 | 17032.00 | | 2016-10-01 00:00:00 | 20684.00 | | 2016-11-01 00:00:00 | 26338.00 | | 2016-12-01 00:00:00 | 10685.00 | ``` * Try introducing some other dimensions from the semantic models into the `group-by` arguments to get a feel for this command. --- ### Semantic structure tip Note that this best practices guide doesn't yet use the [new YAML specification](https://docs.getdbt.com/docs/build/latest-metrics-spec.md). We're working on updating this guide to use the new spec and file structure soon! To read more about the new spec, see [Creating metrics](https://docs.getdbt.com/docs/build/metrics-overview.md). #### Files and Folders The first thing you need to establish is how you’re going to consistently structure your code. There are two recommend best practices to choose from: * 🏡 **Co-locate your semantic layer code** in a one-YAML-file-per-marts-model system. * Puts documentation, data tests, unit tests, semantic models, and metrics into a unified file that corresponds to a dbt-modeled mart. * Trades larger file size for less clicking between files. * Simpler for greenfield projects that are building the Semantic Layer alongside dbt models. * 🏘️**Create a sub-folder** called `models/semantic_models/`. * Create a parallel file and folder structure within that specifically for semantic layer code. * Gives you more targeted files, but may involves switching between files more often. * Better for migrating large existing projects, as you can quickly see what marts have been codified into the Semantic Layer. It’s not terribly difficult to shift between these (it can be done with some relatively straightforward shell scripting), and this is purely a decision based on your developers’ preference (i.e. it has no impact on execution or performance), so don’t feel locked in to either path. Just pick the one that feels right and you can always shift down the road if you change your mind. tip Make sure to save all semantic models and metrics under the directory defined in the [`model-paths`](https://docs.getdbt.com/reference/project-configs/model-paths.md) (or a subdirectory of it, like `models/semantic_models/`). If you save them outside of this path, it will result in an empty `semantic_manifest.json` file, and your semantic models or metrics won't be recognized. #### Naming Next, establish your system for consistent file naming: * 1️⃣ If you’re doing **one-YAML-file-per-mart** then you’d have an `orders.sql` and an `orders.yml`. * 📛 If you’re using a **parallel subfolder approach**, for the sake of unique file names it’s recommended to use the **prefix `sem_` e.g. `sem_orders.yml`** for the dedicated semantic model and metrics that build on `orders.sql` and `orders.yml`. #### Can't decide? Start with a dedicated subfolder for your semantic models and metrics, and then if you find that you’re spending a lot of time clicking between files, you can always shift to a one-YAML-file-per-mart system. Our internal data team has found that the dedicated subfolder approach is more manageable for migrating existing projects, and this is the approach our documentation uses, so if you can't pick go with that. --- ### Set up the dbt Semantic Layer #### Getting started There are two options for developing a dbt project, including the Semantic Layer: * [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) — MetricFlow commands are embedded in the dbt platform CLI under the `dbt sl` subcommand. This is the easiest, most full-featured way to develop Semantic Layer code for the time being. You can use the editor of your choice and run commands from the terminal. * [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) — You can create semantic models and metrics in the Studio IDE. #### Basic commands * 🔍 A less common command that will come in handy with the Semantic Layer is `dbt parse`. This will parse your project and generate a **semantic manifest**, a representation of meaningful connections described by your project. This is uploaded to dbt, and used for running `dbt sl` commands in development. This file gives MetricFlow a **state of the world from which to generate queries**. * 🧰 `dbt sl query` is your other best friend, it will execute a query against your semantic layer and return a sample of the results. This is great for testing your semantic models and metrics as you build them. For example, if you're building a revenue model you can run `dbt sl query --metrics revenue --group-by metric_time__month` to validate that monthly revenue is calculating correctly. * 📝 Lastly, `dbt sl list dimensions --metrics [metric name]` will list all the dimensions available for a given metric. This is useful for checking that you're increasing dimensionality as you progress. You can `dbt sl list` other aspects of your Semantic Layer as well, run `dbt sl list --help` for the full list of options. For more information on the available commands, refer to the [MetricFlow commands](https://docs.getdbt.com/docs/build/metricflow-commands.md) reference, or use `dbt sl --help` and `dbt sl [subcommand] --help` on the command line. If you need to set up a dbt project first, check out the [quickstart guides](https://docs.getdbt.com/docs/get-started-dbt.md). #### Onward! Throughout the rest of the guide, we'll show example code based on the Jaffle Shop project, a fictional chain of restaurants. You can check out the code yourself and try things out in the [Jaffle Shop repository](https://github.com/dbt-labs/jaffle-shop). So if you see us calculating metrics like `food_revenue` later in this guide, this is why! --- ### Staging: Preparing our atomic building blocks The staging layer is where our journey begins. This is the foundation of our project, where we bring all the individual components we're going to use to build our more complex and useful models into the project. We'll use an analogy for working with dbt throughout this guide: thinking modularly in terms of atoms, molecules, and more complex outputs like proteins or cells (we apologize in advance to any chemists or biologists for our inevitable overstretching of this metaphor). Within that framework, if our source system data is a soup of raw energy and quarks, then you can think of the staging layer as condensing and refining this material into the individual atoms we’ll later build more intricate and useful structures with. ##### Staging: Files and folders Let's zoom into the staging directory from our `models` file tree [in the overview](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md) and walk through what's going on here. ```shell models/staging ├── __sources.yml ├── stg_customers.sql ├── stg_customers.yml ├── stg_locations.sql ├── stg_locations.yml ├── stg_order_items.sql ├── stg_order_items.yml ├── stg_orders.sql ├── stg_orders.yml ├── stg_products.sql ├── stg_products.yml ├── stg_supplies.sql └── stg_supplies.yml ``` * **Folders.** Folder structure is extremely important in dbt. Not only do we need a consistent structure to find our way around the codebase, as with any software project, but our folder structure is also one of the key interfaces for understanding the knowledge graph encoded in our project (alongside the DAG and the data output into our warehouse). It should reflect how the data flows, step-by-step, from a wide variety of source-conformed models into fewer, richer business-conformed models. Moreover, we can use our folder structure as a means of selection in dbt [selector syntax](https://docs.getdbt.com/reference/node-selection/syntax.md). For example, with the above structure, if we got fresh e-commerce data loaded and wanted to run all the models that build on our staging layer, we can easily run `dbt build --select staging+` and we're all set for building more up-to-date reports. * ✅ **Subdirectories based on the source system**. Our internal transactional database is one system, the data we get from Stripe's API is another, and lastly the events from our Snowplow instrumentation. We've found this to be the best grouping for most companies, as source systems tend to share similar loading methods and properties between tables, and this allows us to operate on those similar sets easily. The Jaffle Shop example project uses a single `ecom` source, so its staging models live in a flat `staging/` folder. As you add more source systems, create a subdirectory per source. * ❌ **Subdirectories based on loader.** Some people attempt to group by how the data is loaded (Fivetran, Stitch, custom syncs), but this is too broad to be useful on a project of any real size. * ❌ **Subdirectories based on business grouping.** Another approach we recommend against is splitting up by business groupings in the staging layer, and creating subdirectories like 'marketing', 'finance', etc. A key goal of any great dbt project should be establishing a single source of truth. By breaking things up too early, we open ourselves up to creating overlap and conflicting definitions (think marketing and financing having different fundamental tables for orders). We want everybody to be building with the same set of atoms, so in our experience, starting our transformations with our staging structure reflecting the source system structures is the best level of grouping for this step. * **File names.** Creating a consistent pattern of file naming is [crucial in dbt](https://docs.getdbt.com/blog/on-the-importance-of-naming). File names must be unique and correspond to the name of the model when selected and created in the warehouse. We recommend putting as much clear information into the file name as possible, including a prefix for the layer the model exists in, important grouping information, and specific information about the entity or transformation in the model. * ✅ `stg_[source]__[entity]s.sql` - the double underscore between source system and entity helps visually distinguish the separate parts in the case of a source name having multiple words. For instance, `google_analytics__campaigns` is always understandable, whereas to somebody unfamiliar `google_analytics_campaigns` could be `analytics_campaigns` from the `google` source system as easily as `campaigns` from the `google_analytics` source system. Think of it like an [oxford comma](https://www.youtube.com/watch?v=P_i1xk07o4g), the extra clarity is very much worth the extra punctuation. In a single-source project like Jaffle Shop, `stg_orders.sql` and `stg_customers.sql` are clear enough without the source prefix. * ❌ `stg_[entity].sql` - might be specific enough at first for a single-source project, but will break down in time as you add sources. Adding the source system into the file name aids in discoverability, and allows understanding where a component model came from even if you aren't looking at the file tree. * ✅ **Plural.** SQL, and particularly SQL in dbt, should read as much like prose as we can achieve. We want to lean into the broad clarity and declarative nature of SQL when possible. As such, unless there’s a single order in your `orders` table, plural is the correct way to describe what is in a table with multiple rows. ##### Staging: Models Now that we’ve got a feel for how the files and folders fit together, let’s look inside one of these files and dig into what makes for a well-structured staging model. Below is an example of a standard staging model (from our `stg_orders` model) that illustrates the common patterns within the staging layer. We’ve organized our model into two CTEs: one pulling in a source table via the [source macro](https://docs.getdbt.com/docs/build/sources.md#selecting-from-a-source) and the other applying our transformations. While our later layers of transformation will vary greatly from model to model, every one of our staging models will follow this exact same pattern. As such, we need to make sure the pattern we’ve established is rock solid and consistent. ```sql -- stg_orders.sql with source as ( select * from {{ source('ecom', 'raw_orders') }} ), renamed as ( select ---------- ids id as order_id, store_id as location_id, customer as customer_id, ---------- numerics subtotal as subtotal_cents, tax_paid as tax_paid_cents, order_total as order_total_cents, {{ cents_to_dollars('subtotal') }} as subtotal, {{ cents_to_dollars('tax_paid') }} as tax_paid, {{ cents_to_dollars('order_total') }} as order_total, ---------- timestamps {{ dbt.date_trunc('day', 'ordered_at') }} as ordered_at from source ) select * from renamed ``` * Based on the above, the most standard types of staging model transformations are: * ✅ **Renaming** * ✅ **Type casting** * ✅ **Basic computations** (e.g. cents to dollars) * ✅ **Categorizing** (using conditional logic to group values into buckets or booleans) * ❌ **Joins** — the goal of staging models is to clean and prepare individual source-conformed concepts for downstream usage. We're creating the most useful version of a source system table, which we can use as a new modular component for our project. In our experience, joins are almost always a bad idea here — they create immediate duplicated computation and confusing relationships that ripple downstream — there are occasionally exceptions though (refer to [base models](#staging-other-considerations) for more info). * ❌ **Aggregations** — aggregations entail grouping, and we're not doing that at this stage. Remember - staging models are your place to create the building blocks you’ll use all throughout the rest of your project — if we start changing the grain of our tables by grouping in this layer, we’ll lose access to source data that we’ll likely need at some point. We just want to get our individual concepts cleaned and ready for use, and will handle aggregating values downstream. * ✅ **Materialized as views.** Looking at a partial view of our `dbt_project.yml` below, we can see that we’ve configured the entire staging directory to be materialized as views. As they’re not intended to be final artifacts themselves, but rather building blocks for later models, staging models should typically be materialized as views for two key reasons: * Any downstream model (discussed more in [marts](https://docs.getdbt.com/best-practices/how-we-structure/4-marts.md)) referencing our staging models will always get the freshest data possible from all of the component views it’s pulling together and materializing * It avoids wasting space in the warehouse on models that are not intended to be queried by data consumers, and thus do not need to perform as quickly or efficiently ```yaml # dbt_project.yml models: jaffle_shop: staging: +materialized: view ``` * Staging models are the only place we'll use the [`source` macro](https://docs.getdbt.com/docs/build/sources.md), and our staging models should have a 1-to-1 relationship to our source tables. That means for each source system table we’ll have a single staging model referencing it, acting as its entry point — *staging* it — for use downstream. Don’t Repeat Yourself. Staging models help us keep our code DRY. dbt's modular, reusable structure means we can, and should, push any transformations that we’ll always want to use for a given component model as far upstream as possible. This saves us from potentially wasting code, complexity, and compute doing the same transformation more than once. For instance, if we know we always want our monetary values as floats in dollars, but the source system is integers and cents, we want to do the division and type casting as early as possible so that we can reference it rather than redo it repeatedly downstream. This is a welcome change for many of us who have become used to applying the same sets of SQL transformations in many places out of necessity! For us, the earliest point for these 'always-want' transformations is the staging layer, the initial entry point in our transformation process. The DRY principle is ultimately the litmus test for whether transformations should happen in the staging layer. If we'll want them in every downstream model and they help us eliminate repeated code, they're probably okay. ##### Staging: Other considerations * **Base models when joins are necessary to stage concepts.** Sometimes, in order to maintain a clean and DRY staging layer we do need to implement some joins to create a solid concept for our building blocks. In these cases, we recommend creating a sub-directory in the staging directory for the source system in question and building `base` models. These have all the same properties that would normally be in the staging layer, they will directly source the raw data and do the non-joining transformations, then in the staging models we'll join the requisite base models. Common use cases include joining in separate delete tables or unioning disparate but symmetrical sources. You can dig into [more detail on unioning identical sources here](https://discourse.getdbt.com/t/unioning-identically-structured-data-sources/921). The [Jaffle Shop](https://github.com/dbt-labs/jaffle-shop) example project does not use base models. * **[Codegen](https://github.com/dbt-labs/dbt-codegen) to automate staging table generation.** It's very good practice to learn to write staging models by hand, they're straightforward and numerous, so they can be an excellent way to absorb the dbt style of writing SQL. Also, we'll invariably find ourselves needing to add special elements to specific models at times — for instance, in one of the situations above that require base models — so it's helpful to deeply understand how they work. Once that understanding is established though, because staging models are built largely following the same rote patterns and need to be built 1-to-1 for each source table in a source system, it's preferable to start automating their creation. For this, we have the [codegen](https://github.com/dbt-labs/dbt-codegen) package. This will let you automatically generate all the source YAML and staging model boilerplate to speed up this step, and we recommend using it in every project. Development flow versus DAG order. This guide follows the order of the DAG, so we can get a holistic picture of how these three primary layers build on each other towards fueling impactful data products. It’s important to note though that developing models does not typically move linearly through the DAG. Most commonly, we should start by mocking out a design in a spreadsheet so we know we’re aligned with our stakeholders on output goals. Then, we’ll want to write the SQL to generate that output, and identify what tables are involved. Once we have our logic and dependencies, we’ll make sure we’ve staged all the necessary atomic pieces into the project, then bring them together based on the logic we wrote to generate our mart. Finally, with a functioning model flowing in dbt, we can start refactoring and optimizing that mart. By splitting the logic up and moving parts back upstream into intermediate models, we ensure all of our models are clean and readable, the story of our DAG is clear, and we have more surface area to apply thorough testing. --- ### Tactical terminology The rest of this guide will focus on the process of migrating your existing dbt code to the Semantic Layer. To do this, we'll need to introduce some new terminology and concepts that are specific to the Semantic Layer. We want to define them up front, as we have specific meanings in mind applicable to the process of migrating code to the Semantic Layer. These terms can mean different things in different settings, but here we mean: * 🔲 **Normalized** — can be defined with varying degrees of technical rigor, but used here we mean something that contains unique data stored only once in one place, so it can be efficiently joined and aggregated into various shapes. You can think of it referring to tables that function as conceptual building blocks in your business, *not* in the sense of say, strict [Codd 3NF](https://en.wikipedia.org/wiki/Third_normal_form). * 🛒 **Mart** — also has a variety of definitions, but here we mean a table that is relatively normalized and functions as the source of truth for a core concept in your business. * 🕸️ **Denormalized** — when we store the same data in multiple places for easier access without joins. The most denormalized data modeling system is OBT (One Big Table), where we try to get every possible interesting column related to a concept (for instance, customers) into one big table so all an analyst needs to do is `select`. * 🗞️ **Rollup** — used here as a catchall term meaning both denormalized tables built on top of normalized marts and those that perform aggregations to a certain grain. For example `active_accounts_per_week` might aggregate `customers` and `orders` data to a weekly time. Another example would be `customer_metrics` which might denormalize a lot of the data from `customers` as well as aggregated data from `orders`. For the sake of brevity in this guide, we’ll call all these types of products built on top of your normalized concepts as **rollups**. We'll also use a couple *new* terms for the sake of brevity. These aren't standard or official dbt-isms, but useful for communicating meaning in the context of refactoring code for the Semantic Layer: * 🧊 **Frozen** — shorthand to indicate code that is statically built in dbt’s logical transformation layer. Does not refer to the materialization type: views, incremental models, and regular tables are all considered *frozen* as they statically generate data or code that is stored in the warehouse as opposed to dynamically querying, as with the Semantic Layer. This is *not* a bad thing! We want some portion of our transformation logic to be frozen and stable as the *transformation* *logic* is not rapidly shifting and we benefit in testing, performance, and stability. * 🫠 **Melting** — the process of breaking up frozen structures into flexible Semantic Layer code. This allows them to create as many combinations and aggregations as possible dynamically in response to stakeholder needs and queries. tip 🏎️ **The Semantic Layer is a denormalization engine.** dbt transforms your data into clean, normalized marts. The Semantic Layer is a denormalization engine that dynamically connects and molds these building blocks into the maximum amount of shapes available *dynamically*. --- ### The rest of the project ##### Project structure review So far we’ve focused on the `models` folder, the primary directory of our dbt project. Next, we’ll zoom out and look at how the rest of our project files and folders fit in with this structure, starting with how we approach YAML configuration files. ```shell models ├── marts │ ├── customers.sql │ ├── customers.yml │ ├── order_items.sql │ ├── order_items.yml │ ├── orders.sql │ ├── orders.yml │ └── ... └── staging ├── __sources.yml ├── stg_customers.sql ├── stg_customers.yml ├── stg_orders.sql ├── stg_orders.yml └── ... ``` ##### YAML in-depth When structuring your YAML configuration files in a dbt project, you want to balance centralization and file size to make specific configs as easy to find as possible. It’s important to note that while the top-level YAML files (`dbt_project.yml`, `packages.yml`) need to be specifically named and in specific locations, the files containing your `sources` and `models` dictionaries can be named, located, and organized however you want. It’s the internal contents that matter here. As such, we’ll lay out our primary recommendation, as well as the pros and cons of a popular alternative. Like many other aspects of structuring your dbt project, what’s most important here is consistency, clear intention, and thorough documentation on how and why you do what you do. * ✅ **Config per folder.** As in the example above, create a `_[directory]__models.yml` per directory in your models folder that configures all the models in that directory. for staging folders, also include a `_[directory]__sources.yml` per directory. * The leading underscore ensures your YAML files will be sorted to the top of every folder to make them easy to separate from your models. * YAML files don’t need unique names in the way that SQL model files do, but including the directory (instead of simply `_sources.yml` in each folder), means you can fuzzy find the right file more quickly. * We’ve recommended several different naming conventions over the years, most recently calling these `schema.yml` files. We’ve simplified to recommend that these simply be labelled based on the YAML dictionary that they contain. * If you utilize [doc blocks](https://docs.getdbt.com/docs/build/documentation.md#using-docs-blocks) in your project, we recommend following the same pattern, and creating a `_[directory]__docs.md` markdown file per directory containing all your doc blocks for that folder of models. * ❌ **Config per project.** Some people put *all* of their source and model YAML into one file. While you can technically do this, and while it certainly simplifies knowing what file the config you’re looking for will be in (as there is only one file), it makes it much harder to find specific configurations within that file. We recommend balancing those two concerns. * ⚠️ **Config per model.** On the other end of the spectrum, some people prefer to create one YAML file per model. This presents less of an issue than a single monolith file, as you can quickly search for files, know exactly where specific configurations exist, spot models without configs (and thus without tests) by looking at the file tree, and various other advantages. In our opinion, the extra files, tabs, and windows this requires creating, copying from, pasting to, closing, opening, and managing creates a somewhat slower development experience that outweighs the benefits. Defining config per directory is the most balanced approach for most projects, but if you have compelling reasons to use config per model, there are definitely some great projects that follow this paradigm. * ✅ **Cascade configs.** Leverage your `dbt_project.yml` to set default configurations at the directory level. Use the well-organized folder structure we’ve created thus far to define the baseline schemas and materializations, and use dbt’s cascading scope priority to define variations to this. For example, as below, define your marts to be materialized as tables by default, define separate schemas for our separate subfolders, and any models that need to use incremental materialization can be defined at the model level. ```yaml # dbt_project.yml models: jaffle_shop: staging: +materialized: view marts: +materialized: table ``` Define your defaults. One of the many benefits this consistent approach to project structure confers to us is this ability to cascade default behavior. Carefully organizing our folders and defining configuration at that level whenever possible frees us from configuring things like schema and materialization in every single model (not very DRY!) — we only need to configure exceptions to our general rules. Tagging is another area this principle comes into play. Many people new to dbt will rely on tags rather than a rigorous folder structure, and quickly find themselves in a place where every model *requires* a tag. This creates unnecessary complexity. We want to lean on our folders as our primary selectors and grouping mechanism, and use tags to define groups that are *exceptions.* A folder-based selection like \*\*`dbt build --select marts.marketing` is much simpler than trying to tag every marketing-related model, hoping all developers remember to add that tag for new models, and using `dbt build --select tag:marketing`. ###### Defining groups A group is a collection of nodes within a dbt DAG. Groups enable intentional collaboration within and across teams by restricting [access to private](https://docs.getdbt.com/reference/resource-configs/access.md) models. Groups are defined in `.yml` files, nested under a `groups:` key. In version 1.10 and higher, you can add a `description` and a `meta` config to add more information about the group. (Applies to dbt v1.10 and later) models/marts/finance/finance.yml ```yaml groups: - name: finance description: "All finance-related models owned by the Finance team." # optional owner: # 'name' or 'email' is required; additional properties will no longer be allowed in a future release email: finance@jaffleshop.com config: meta: # optional data_owner: Finance team cost_center: finance data_classification: sensitive ``` For more information about using groups, see [Add groups to your DAG](https://docs.getdbt.com/docs/build/groups.md). ##### How we use the other folders ```shell jaffle_shop ├── analyses ├── data-tests ├── macros │ └── cents_to_dollars.sql ├── seeds │ └── jaffle-data │ ├── raw_customers.csv │ ├── raw_orders.csv │ └── ... └── snapshots ``` We’ve focused heavily thus far on the primary area of action in our dbt project, the `models` folder. As you’ve probably observed though, there are several other folders in our project. While these are, by design, very flexible to your needs, we’ll discuss the most common use cases for these other folders to help get you started. * ✅ `seeds` for lookup tables. The most common use case for seeds is loading lookup tables that are helpful for modeling but don’t exist in any source systems — think mapping zip codes to states, or UTM parameters to marketing campaigns. In this example project we have seeds in `seeds/jaffle-data/` that can populate the `raw` schema for local development when `load_source_data` is enabled. * ❌ `seeds` for loading source data. Do not use seeds to load data from a source system into your warehouse. If it exists in a system you have access to, you should be loading it with a proper EL tool into the raw data area of your warehouse. dbt is designed to operate on data in the warehouse, not as a data-loading tool. * ✅ `analyses` for storing auditing queries. The `analyses` folder lets you store any queries you want to use Jinja with and version control, but not build into models in your warehouse. There are limitless possibilities here, but the most common use case when we set up projects at dbt Labs is to keep queries that leverage the [audit helper](https://github.com/dbt-labs/dbt-audit-helper) package. This package is incredibly useful for finding discrepancies in output when migrating logic from another system into dbt. * ✅ `tests` for testing multiple specific tables simultaneously. As dbt tests have evolved, writing singular tests has become less and less necessary. It's extremely useful for work-shopping test logic, but more often than not you'll find yourself either migrating that logic into your own custom generic tests or discovering a pre-built test that meets your needs from the ever-expanding universe of dbt packages (between the extra tests in [`dbt-utils`](https://github.com/dbt-labs/dbt-utils) and [`dbt-expectations`](https://github.com/calogica/dbt-expectations) almost any situation is covered). One area where singular tests still shine though is flexibly testing things that require a variety of specific models. If you're familiar with the difference between [unit tests](https://en.wikipedia.org/wiki/Unit_testing) [and](https://www.testim.io/blog/unit-test-vs-integration-test/) [integration](https://www.codecademy.com/resources/blog/what-is-integration-testing/) [tests](https://en.wikipedia.org/wiki/Integration_testing) in software engineering, you can think of generic and singular tests in a similar way. If you need to test the results of how several specific models interact or relate to each other, a singular test will likely be the quickest way to nail down your logic. * ✅ `snapshots` for creating [Type 2 slowly changing dimension](https://en.wikipedia.org/wiki/Slowly_changing_dimension#Type_2:_add_new_row) records from [Type 1](https://en.wikipedia.org/wiki/Slowly_changing_dimension#Type_1:_overwrite) (destructively updated) source data. This is [covered thoroughly in the dbt Docs](https://docs.getdbt.com/docs/build/snapshots.md), unlike these other folders has a more defined purpose, and is out-of-scope for this guide, but mentioned for completion. * ✅ `macros` for DRY-ing up transformations you find yourself doing repeatedly. Like snapshots, a full dive into macros is out-of-scope for this guide and well [covered elsewhere](https://docs.getdbt.com/docs/build/jinja-macros.md), but one important structure-related recommendation is to [write documentation for your macros](https://docs.getdbt.com/faqs/Docs/documenting-macros.md). We recommend creating a `_macros.yml` and documenting the purpose and arguments for your macros once they’re ready for use. ##### Project splitting One important, growing consideration in the analytics engineering ecosystem is how and when to split a codebase into multiple dbt projects. Currently, our advice for most teams, especially those just starting, is fairly simple: in most cases, we recommend doing so with [Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md)! Mesh allows organizations to handle complexity by connecting several dbt projects rather than relying on one big, monolithic project. This approach is designed to speed up development while maintaining governance. As breaking up monolithic dbt projects into smaller, connected projects, potentially within a modern mono repo becomes easier, the scenarios we currently advise against may soon become feasible. So watch this space! * ✅ **Business groups or departments.** Conceptual separations within the project are the primary reason to split up your project. This allows your business domains to own their own data products and still collaborate using Mesh. For more information about Mesh, please refer to our [Mesh FAQs](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-5-faqs.md). * ✅ **Data governance.** Structural, organizational needs — such as data governance and security — are one of the few worthwhile reasons to split up a project. If, for instance, you work at a healthcare company with only a small team cleared to access raw data with PII in it, you may need to split out your staging models into their own projects to preserve those policies. In that case, you would import your staging project into the project that builds on those staging models as a [private package](https://docs.getdbt.com/docs/build/packages.md#private-packages). * ✅ **Project size.** At a certain point, your project may grow to have simply too many models to present a viable development experience. If you have 1000s of models, it absolutely makes sense to find a way to split up your project. * ❌ **ML vs Reporting use cases.** Similarly to the point above, splitting a project up based on different use cases, particularly more standard BI versus ML features, is a common idea. We tend to discourage it for the time being. As with the previous point, a foundational goal of implementing dbt is to create a single source of truth in your organization. The features you’re providing to your data science teams should be coming from the same marts and metrics that serve reports on executive dashboards. #### Final considerations Overall, consistency is more important than any of these specific conventions. As your project grows and your experience with dbt deepens, you will undoubtedly find aspects of the above structure you want to change. While we recommend this approach for the majority of projects, every organization is unique! The only dogmatic advice we’ll put forward here is that when you find aspects of the above structure you wish to change, think intently about your reasoning and document for your team *how* and *why* you are deviating from these conventions. To that end, we highly encourage you to fork this guide and add it to your project’s README, wiki, or docs so you can quickly create and customize those artifacts. Finally, we emphasize that this guide is a living document! It will certainly change and grow as dbt and dbt Labs evolve. We invite you to join in — discuss, comment, and contribute regarding suggested changes or new elements to cover. --- ### Understanding a dbt project with dbt Wizard Use dbt Wizard to move from a broad project map to evidence-backed answers about model logic, lineage, tests, and data. This workflow helps when you join a project, review an unfamiliar area of the DAG, prepare a refactor, or investigate why a model exists. #### Prerequisites Start dbt Wizard from the dbt project root, and make sure the project has a current `target/manifest.json`. Refer to [Use dbt Wizard locally](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) for setup. If you want dbt Wizard to inspect warehouse results, you also need a working development connection and permission to query the relevant relations. The prompts on this page work the same way in dbt platform. #### Start with a project map Ask for a read-only inventory before narrowing the investigation: ```text Map this dbt project for a new analytics engineer. Identify the main model layers, important sources, high fan-out models, test coverage gaps, and the three areas that deserve careful review. Cite the files and metadata you used. Do not edit anything. ``` This gives you a starting hypothesis, not a complete project specification. Ask follow-up questions about any labels or architectural claims that aren't supported by code, configuration, or metadata. #### Investigate one model Name a model and ask dbt Wizard to connect its definition to its place in the DAG: ```text Explain fct_orders to me. Describe its grain, inputs, important transformations, tests, materialization, and direct downstream consumers. Point out anything the code does not make clear. ``` For a thorough investigation, dbt Wizard can: 1. Find the model and its related YAML, tests, macros, and configuration. 2. Describe the selected columns, SQL, materialization, and dependencies. 3. Trace upstream inputs and downstream consumers through project lineage. 4. Query structured project metadata for test coverage and resource properties. 5. Preview or summarize warehouse data when a connection and relation are available. Ask dbt Wizard to separate observed facts from inferences. For example: ```text For each conclusion, label it as code, metadata, warehouse evidence, or an inference. List the business questions I still need to ask the model owner. ``` #### Trace a business concept You don't need to know the model name before you begin. Describe the concept and let dbt Wizard search for likely resources: ```text Trace how recurring revenue is calculated from source to final mart. Include the models, columns, macros, tests, and metrics involved. Call out where the definition changes or where multiple definitions exist. ``` When several candidates match, ask dbt Wizard to show the candidates and explain why each one might be relevant before continuing. #### Assess change impact Once you understand the current behavior, test a proposed change without editing files: ```text What would be affected if stg_payments changed from one row per payment to one row per payment attempt? Identify direct and indirect downstream resources, grain-dependent joins, tests, metrics, and exposures. Do not make changes. ``` Use the result to define the scope of a change plan. High fan-out models, contracts, metrics, exposures, and joins on the changed grain deserve closer review. #### Confirm assumptions with data SQL and metadata show intended behavior. Warehouse data can help you check whether that intent matches current results: ```text Check whether fct_orders is currently one row per order_id and summarize nulls or duplicates in its key columns. Use the smallest reasonable query and show me the query before running it. ``` Treat current data as evidence about the present state, not a permanent guarantee. Sampling, filters, stale relations, and environment differences can all affect the result. #### Turn findings into a plan End the investigation with a handoff that another engineer can review: ```text Summarize what we learned, unresolved business questions, affected resources, and a safe implementation and validation plan. Keep facts separate from recommendations. Do not edit files yet. ``` The summary should include relevant file paths, model names, lineage scope, supporting evidence, and explicit unknowns. #### Understand what can be missing dbt Wizard can reason from the project and the tools connected to the session. It can't reliably infer context that isn't represented there, including: * Undocumented business definitions and ownership decisions. * Dashboard or application dependencies that aren't represented as exposures or connected metadata. * Warehouse behavior that the current connection can't query. * The reason behind a historical design choice when it isn't recorded in code, documentation, or version history. * Data quality guarantees that don't have corresponding tests or contracts. Use model owners, pull request history, and business documentation to fill these gaps. #### Related docs * [Validate dbt changes with dbt Wizard](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-3-validate-changes.md) * [dbt Wizard use cases](https://docs.getdbt.com/docs/dbt-ai/wizard-use-cases.md) * [How dbt Wizard works](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md) * [About model governance](https://docs.getdbt.com/docs/mesh/govern/about-model-governance.md) --- ### Validating dbt changes with dbt Wizard Use dbt Wizard to assess the impact of a change, choose a validation depth, and review evidence before you merge. This workflow is useful after you edit a model, test, macro, or YAML file and want more than a code-only review. dbt Wizard can combine project metadata, dbt commands, development builds, and production comparisons based on the validation level you select. CLI workflow The validation levels on this page are available in dbt Wizard CLI. Validation controls in the dbt platform can differ. Refer to [dbt Wizard in the dbt platform](https://docs.getdbt.com/docs/platform/wizard-platform.md) for platform behavior. #### Prerequisites Before you begin: * [Install and configure dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md). * Open your dbt project in the terminal and start dbt Wizard from the project root. * Make sure the project has a current `target/manifest.json`. Run `dbt parse`, `dbt compile`, or `dbt build` if needed. * Configure a development target that dbt Wizard can use for commands that query or materialize data. * Review your Git status so you know which changes belong to the task. Warehouse validation can consume compute. dbt Wizard shows commands for approval according to your session policy before it runs them. #### Review the change and its impact Start by asking dbt Wizard to establish the scope before it runs commands or edits files: ```text Review my uncommitted changes. Explain which dbt resources changed, trace their downstream impact, and propose a validation plan. Do not edit files yet. ``` For a targeted review, name the resource and the behavior that must remain stable: ```text Validate the changes to fct_orders. Its grain must remain one row per order, order_total must not change for completed orders, and downstream finance models must still compile. Show me the validation plan before running it. ``` dbt Wizard uses the project graph and changed files to identify affected models, tests, and downstream resources. Review this scope carefully. Add any business invariant that cannot be inferred from SQL or metadata. #### Choose a validation level When dbt Wizard asks how thoroughly to validate a change, choose the level that matches its risk and cost. | Level | What dbt Wizard checks | Use it when | | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | | Light | Checks SQL syntax, lints changed files when linting is supported, runs focused dbt tests, and reviews the code. It doesn't materialize changed models. | You made a low-risk change and need fast feedback. | | Medium | Compiles and lints, runs focused tests, materializes modified models in development, and checks affected downstream resources. This is the default level. | You changed model logic and need to verify the development result. | | Heavy | Performs medium validation, forms explicit expectations, compares development and production results, and reports row counts, schema differences, sample records, and downstream impact when the required data is available. | The change affects grain, financial logic, contracts, or other high-risk behavior. | | Skip | Doesn't run the structured validation workflow. | You only need a draft, or you plan to validate through another process. | The exact commands depend on your dbt engine, project, adapter, and selected resources. dbt Wizard can use commands such as `dbt compile`, `dbt lint`, `dbt test`, `dbt run`, and `dbt build`. Linting isn't available in every dbt CLI environment. #### Approve and monitor commands Before approving a command, check: * The selected models and downstream depth match the intended scope. * The target points to a development schema, unless you intentionally approved another target. * Deferral and state settings point to the expected environment. * A production comparison is read-only and uses the intended production relations. * The expected warehouse cost and run time are appropriate for the change. You can redirect the plan at any time. For example: ```text Do not build every downstream model. Build fct_orders and its first-degree children, then compile the rest of the downstream graph. ``` #### Review the validation result A useful validation summary should distinguish evidence from unresolved risk. Check that it includes: * The resources and commands that were validated. * Compile, lint, run, and test outcomes. * Development and production differences, when you selected heavy validation. * Downstream resources that weren't run or inspected. * Failures, warnings, skipped checks, and the reason for each skip. * Any assumptions that still need a subject-matter expert to confirm. If a check fails, ask dbt Wizard to investigate the failure before changing the test or expected result: ```text Explain whether this failure is caused by my code, existing warehouse data, or the validation environment. Do not weaken or remove the test. ``` #### Understand the limits Validation is evidence, not a guarantee that a change is correct. Keep these limits in mind: * A successful compile doesn't validate business logic or warehouse results. * Tests only cover the assertions encoded in the project. * A development-to-production comparison requires accessible relations with comparable schemas and data. * Sample records can reveal differences, but they don't prove that all rows are correct. * External consumers aren't included unless they are represented in dbt metadata or another connected tool. * A skipped check should remain visible in your review and pull request notes. #### Related docs * [Understanding a dbt project with dbt Wizard](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-2-understand-project.md) * [How dbt Wizard works](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md) * [Use subagents with dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-subagents.md) * [About dbt state](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) --- ### Views-only pattern for maximum freshness Snowflake examples ahead This page uses Snowflake for code examples, but you can adapt the views-only pattern to other warehouses. For some workloads, the simplest and most "real-time" pattern is to materialize everything as views on top of a continuously updated source table. When transformations are very lightweight and the source is already being updated in near real-time, this can preserve the source's latency almost perfectly. #### When to use the views-only pattern Use this when: * Source freshness is already "good enough" (for example, ingestion service or operational system writes into a warehouse table every few seconds or minutes). * You have very lightweight transformations, such as: * Simple projections / renames * One to two joins to small reference table * Minimal or no heavy aggregations or window functions * You care most about preserving the source table's latency and are willing to trade off some query performance at read time. * This works best for small-to-medium tables with simple queries. Typical examples: * Dashboards showing current system status (like active user sessions, current queue depth, or recent device heartbeats) where you need to see the latest data immediately. * Event data that you're forwarding to other tools with minimal transformation (raw data with just a bit of normalization, like cleaning up field names or adding a few reference fields). If your transform logic becomes heavier, multiple teams depend on the data, or you need better cost and performance control, transition to [incremental models](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/2-incremental-patterns.md) or [dynamic tables/materialized views](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/3-warehouse-native-features.md). Reserve this pattern for the smallest, most latency‑sensitive use cases. ###### Assumptions The examples used in this page assume the following setup: * A non‑dbt system (ETL, streaming pipeline, app) is already writing into a warehouse‑resident table such as `raw.realtime_events` or `ops.active_sessions`, and that table meets your service level agreement for latency. * Querying that table directly is acceptable from a performance and cost standpoint for your expected concurrency. * You don't need dbt to persist intermediate tables; you mainly care about: * Consistent SQL logic (column naming, type casting) * Tests, contracts, and lineage * Exposures to BI / downstream tools All dbt models in this path are materialized as views, not tables or incremental models. #### Example implementation Here's an example implementation of the views-only pattern, which has the following pattern structure: * [Source table](#source-table-definition) (continuously updated): `raw.realtime_events` * [Thin staging view](#staging-view): `analytics.stg_realtime_events_v` * [Domain view(s)](#domain-view-definition): `analytics.vw_realtime_events_enriched` ##### Source table definition ```yaml # models/sources.yml version: 2 sources: - name: raw schema: raw tables: - name: realtime_events description: "Continuously updated event table from streaming pipeline." loaded_at_field: event_ts ``` ##### Staging view ```sql -- models/staging/stg_realtime_events.sql {{ config( materialized = 'view' ) }} select event_id, event_ts::timestamp_ntz as event_ts, -- Snowflake syntax for type casting to_date(event_ts) as event_date, user_id, event_type, payload from {{ source('raw', 'realtime_events') }}; ``` ##### Domain view definition ```sql -- models/marts/vw_realtime_events_enriched.sql {{ config( materialized = 'view' ) }} with base as ( select * from {{ ref('stg_realtime_events') }} ), user_dim as ( select user_id, user_segment, signup_date from {{ ref('dim_user') }} -- can be a table or incremental model ) select b.event_id, b.event_ts, b.event_date, b.user_id, u.user_segment, b.event_type, b.payload from base as b left join user_dim as u on b.user_id = u.user_id; ``` Downstream tools query `analytics.vw_realtime_events_enriched`. As long as `raw.realtime_events` is continuously updated, this view stack is as fresh as the source. #### Benefits * Maximum freshness: The view reflects new data as soon as it lands in `raw.realtime_events`. * Simple operations: No incremental logic to tune and no extra dbt job needed just to keep the data fresh. You still schedule jobs for tests, docs, and so on. * Best for small datasets: Works well when tables are small and queries are simple. Computing the view on the fly is cheap and fast. #### Limitations and risks This pattern is only safe under tight constraints and has several important limitations: * [Doesn't scale to heavy transformations](#doesnt-scale-to-heavy-transformations) * [No "frozen" intermediate tables](#no-frozen-intermediate-tables) * [Schema change sensitivity](#schema-change-sensitivity) * [Potential impact on operational systems](#potential-impact-on-operational-systems) ##### Doesn't scale to heavy transformations If your logic evolves into large joins, deep view chains, or expensive aggregations, you'll quickly run into performance issues: * Every query must re‑execute all the logic. * The warehouse has to optimize and execute the full stack of views every time. In those cases, use either of the following: * [Incremental models](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/2-incremental-patterns.md) * [Dynamic tables or materialized views](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/3-warehouse-native-features.md), where appropriate ##### No "frozen" intermediate tables Because everything is a view: * There's no persisted intermediate layer to debug or profile. * You can't easily "rerun yesterday's logic" if upstream data changes—everything always reflects the current state. ##### Schema change sensitivity Schema changes in the source table propagate immediately through the view stack, which: * Can break downstream BI if columns are dropped or types change. * Make tests and model contracts more important, since there’s no batch boundary to catch issues before users see them. ##### Potential impact on operational systems If the continuously‑updated source is itself a live operational store (not a warehouse landing table), you must be careful not to overload it with analytics queries. In most cases, it is recommended to: 1. Replicate into a warehouse table first (Snowflake, BigQuery, Databricks, and so on). 2. Apply this views‑only pattern within the warehouse, not directly on the Online Transaction Processing system. --- ### Warehouse-native features for real-time data Modern data warehouses offer native features that can simplify near real-time data patterns. Instead of managing incremental logic yourself, you can declare the desired freshness and let the warehouse handle the refresh mechanics. This section covers when to use dynamic tables and materialized views instead of incremental models for near real-time data. * [Dynamic tables](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#dynamic-tables) are a warehouse-specific feature in Snowflake that lets the warehouse keep a table updated for you. You define what the table should look like, and the warehouse keeps the table fresh automatically. * [Materialized views](https://docs.getdbt.com/docs/build/materializations.md#materialized-view) are a warehouse-specific feature that lets the warehouse save the results of a query so they’re faster to read, and refresh them as the underlying data changes. Note that the exact behavior depends on the warehouse. * [Incremental models](https://docs.getdbt.com/docs/build/incremental-models.md) are a dbt feature that lets dbt update a table by processing only new data. You tell dbt how new data should be added using your incremental logic SQL, and dbt runs the right SQL when the model is built. ###### When to consider warehouse-native features **Use dynamic tables or materialized views when:** * Your requirement is "data always within X minutes of real time" and you don't need precise scheduling control. * You want to simplify operational complexity by offloading refresh logic to the warehouse. * Your transformations are relatively straightforward. * You're willing to trade some control for convenience. **Stick with incremental models when:** * You need fine-grained control over scheduling and refresh logic, * You have complex business rules requiring custom incremental strategies (like microbatching). * You need to coordinate refreshes across multiple models in a specific order. #### Dynamic tables Warehouse support Dynamic tables are currently supported in Snowflake, with similar features available in other warehouses under different names. Check your warehouse documentation for availability. With dynamic tables, you can define the target state with SQL, and the warehouse automatically handles incremental refreshes. For example, the following SQL model uses a dynamic table to keep a table up to date for you: ```sql {{ config( materialized = 'dynamic_table', target_lag = '5 minutes', snowflake_warehouse = 'TRANSFORM_WH' -- Snowflake syntax ) }} select event_id, event_ts::timestamp_ntz as event_ts, to_date(event_ts) as event_date, user_id, event_type, payload from {{ source('raw', 'events') }} where event_ts >= current_timestamp() - interval '7 days'; ``` ###### target\_lag config The [`target_lag` parameter](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#target-lag) tells the warehouse the maximum acceptable staleness of the dynamic table relative to its sources, and helps determine when the table should be refreshed. For example: * `target_lag = '1 minute'` - Warehouse keeps the table within one minute of its source data, refreshing automatically as needed. * `target_lag = '5 minutes'` - Table may lag up to five minutes behind its sources. * `target_lag = 'downstream'` - Table refreshes only when a downstream table depends on it. * `target_lag = '1 minute'` - Data refreshed to be within 1 minute of the source * `target_lag = '5 minutes'` - Data within 5 minutes * `target_lag = 'downstream'` - Refresh when downstream tables need it The warehouse automatically determines when to refresh, whether to do a full or incremental update, and how to optimize the refresh query. ###### Benefits * Declarative freshness: specify "how fresh" not "when to refresh" * Warehouse-managed optimization * Cost predictability: refreshes run only when needed to meet `target_lag` * Simple setup ###### Limitations * Less control over exact timing or orchestration logic * Cost visibility can be harder to predict than scheduled dbt jobs * Less visibility into refresh decisions compared to dbt's explicit incremental logic * Currently warehouse-specific (implementation varies by platform) #### Materialized views Materialized views are available in most modern data warehouses and cache query results that automatically refresh when underlying data changes. Materialized views work like this: * The warehouse detects changes to source tables and refreshes the materialized view. * Many warehouses can incrementally update the view rather than recomputing everything. * Queries against the materialized view read cached results, not the underlying tables. For example, the following sql model uses a materialized view to keep a table up to date for you: ```sql {{ config( materialized = 'materialized_view' ) }} select user_id, date_trunc('hour', event_ts) as event_hour, count(*) as event_count from {{ source('raw', 'events') }} group by 1, 2; ``` #### Resources by warehouse Here are some resources for each warehouse: ###### BigQuery * [dbt developer docs: BigQuery materialized views](https://docs.getdbt.com/reference/resource-configs/bigquery-configs.md#materialized-views) * [BigQuery docs: Materialized views intro](https://cloud.google.com/bigquery/docs/materialized-views-intro) * [BigQuery docs: Streaming API](https://docs.cloud.google.com/bigquery/docs/write-api) ###### Databricks * [dbt developer docs: Databricks materialized views and streaming tables](https://docs.getdbt.com/reference/resource-configs/databricks-configs.md#materialized-views-and-streaming-tables) * [Databricks docs: Materialized views](https://docs.databricks.com/en/views/materialized.html) ###### Postgres * [dbt developer docs: Postgres materialized views](https://docs.getdbt.com/reference/resource-configs/postgres-configs.md#materialized-views) * [Postgres docs: Materialized views](https://www.postgresql.org/docs/current/rules-materializedviews.html) ###### Redshift * [dbt developer docs: Redshift materialized views](https://docs.getdbt.com/reference/resource-configs/redshift-configs.md#materialized-views) * [Redshift docs: Materialized views overview](https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-overview.html) * [Redshift docs: Streaming ingestion to a materialized view](https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-streaming-ingestion.html) ###### Snowflake * [dbt developer docs: Dynamic tables configurations](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#dynamic-tables) * [Snowflake docs: Dynamic tables intro](https://docs.snowflake.com/en/user-guide/dynamic-tables-intro) * [Snowflake blog: Dynamic tables for streaming pipelines](https://www.snowflake.com/en/blog/dynamic-tables-delivering-declarative-streaming-data-pipelines/) * [Snowflake docs: Materialized views](https://docs.snowflake.com/en/user-guide/views-materialized) #### Related docs * [dbt blog: Announcing materialized views](https://docs.getdbt.com/blog/announcing-materialized-views) * [dbt blog: Optimizing query run time with materialization schedules](https://www.getdbt.com/blog/optimizing-query-run-time-with-materialization-schedules/) --- ### Who is dbt Mesh for? Before embarking on a Mesh implementation, it's important to understand if Mesh is the right fit for your team. Here, we outline three common organizational structures to help teams identify whether Mesh might fit your organization's needs. #### The enterprise data mesh Some data teams operate on a global scale. By definition, the team needs to manage, deploy, and distribute data products across a large number of teams. Central IT may own some data products or simply own the platform upon which data products are built. Often, these organizations have “architects” who can advise line-of-business teams on their work while keeping track of what’s happening globally (regarding tooling and the substance of work). This is a lot like how software organizations work beyond a certain scale. The headcount ratio of domain teams to platform teams in this scenario is roughly ≥10:1. For each member of the central platform team, there might be dozens of members of domain-aligned data teams. Is Mesh a good fit in this scenario? Absolutely! There is no other way to share data products at scale. One dbt project would not keep up with the global demands of an organization like this. ##### Tips and tricks * **Managing shared macros**: Teams operating at this scale will benefit from a separate repository containing a dbt package of reusable utility macros that all other projects will install. This is different from public models, which provide data-as-a-service (a set of “API endpoints”) — this is distributed as a **library**. This package can also standardize imports of other third-party packages, as well as providing wrappers / shims for those macros. This package should have a dedicated team of maintainers — probably the central platform team, or a set of “superusers” from domain-aligned data modeling teams. tip To help you get started, check out our [Quickstart with Mesh](https://docs.getdbt.com/guides/mesh-qs.md) or our online [Mesh course](https://learn.getdbt.com/courses/dbt-mesh) to learn more! ##### Adoption challenges * Onboarding hundreds of people and dozens of projects is full of friction! The challenges of a scaled, global organization are not to be underestimated. To start the migration, prioritize teams that have strong dbt familiarity and fundamentals. Mesh is an advancement of core dbt deployments, so these teams are likely to have a smoother transition. Additionally, prioritize teams that manage strategic data assets that need to be shared widely. This ensures that Mesh will help your teams deliver concrete value quickly. If this sounds like your organization, Mesh is the architecture you should pursue. ✅ #### Hub and spoke Some slightly smaller organizations still operate with a central data team serving several business-aligned analytics teams in a ~5:1 headcount ratio. These central teams look less like an IT function and more like a modern data platform team of analytics engineers. This team provides the majority of the data products to the rest of the org, as well as the infrastructure for downstream analytics teams to spin up their own spoke projects to ensure quality and maintenance of the core platform. Is Mesh a good fit in this scenario? Almost certainly! If your central data team starts to bottleneck analysts’ work, you need a way for those teams to operate relatively independently while still ensuring the quality of the most used data products. Mesh is designed to solve this exact problem. ##### Tips and tricks * **Data products by some, for all:** The spoke teams shouldn’t produce public models. By contrast, development in the hub team project should be slower, more careful, and focus on producing foundational public models shared across domains. We’d recommend giving hub team members access (at least read-only) to downstream projects, which will help with more granular impact analysis within Catalog. If a public model isn’t used in any downstream project or a specific column in that model, the hub team can feel better about removing it. However, they should still utilize the dbt governance features like `deprecation_date` and `version` as appropriate to set expectations. If there is a need for a public model in a spoke project to be shared across multiple projects, consider first whether it could or should be moved to the hub project. * **Sources:** Spokes should be allowed/encouraged to define and use *domain-specific* data sources. The platform team should not need to worry about, say, `Thinkific` data when building core data marts, but the Training project may need to. *No two sources anywhere in a dbt mesh should point to the same relation object.* If a spoke feels like they need to use a source the hub already uses, the interfaces should change so that the spoke can get what they need from the platform project. * **Project quality:** More analyst-focused teams will have different skill levels & quality bars. Owning their data means they own the consequences as well. Rather than being accountable for the end-to-end delivery of data assets, the Hub team is an enablement team: their role is to provide guardrails and quality checks, but not to fix all the issues exactly to their liking (and thereby remain a bottleneck). ##### Adoption challenges There are trade-offs to using this architecture, especially for the hub team managing and maintaining public models. This workflow has intentional friction to reduce the chances of unintentional model changes that break unspoken data contracts. These assurances may come with some sacrifices, such as faster onboarding or more flexible development workflows. Compared to having a single project, where a select few are doing all the development work, this architecture optimizes for slower development from a wider group of people. If this sounds like your organization, it's very likely that Mesh is a good fit for you. ✅ #### Single team monolith Some organizations operate on an even smaller scale. If your data org is a single small team that controls the end-to-end process of building and maintaining all data products at the organization, Mesh may not be required. The complexity in projects comes from having a wide variety of data sources and stakeholders. However, given the team's size, operating on a single codebase may be the most efficient way to manage data products. Generally, if a team of this size and scope is looking to implement Mesh, it's likely that they are looking for better interface design and/or performance improvements for certain parts of their dbt DAG, and not because they necessarily have an organizational pain point to solve. *Is Mesh a good fit?* Maybe! There are reasons to separate out parts of a large monolithic project into several to better orchestrate and manage the models. However, if the same people are managing each project, they may find that the overhead of managing multiple projects is not worth the benefits. If this sounds like your organization, it's worth considering whether Mesh is a good fit for you. --- ### Writing custom generic data tests dbt ships with [Not Null](https://docs.getdbt.com/reference/resource-properties/data-tests.md#not-null), [Unique](https://docs.getdbt.com/reference/resource-properties/data-tests.md#unique), [Relationships](https://docs.getdbt.com/reference/resource-properties/data-tests.md#relationships), and [Accepted Values](https://docs.getdbt.com/reference/resource-properties/data-tests.md#accepted-values) generic data tests. (These used to be called "schema tests," and you'll still see that name in some places.) Under the hood, these generic data tests are defined as `test` blocks (like macros). info There are tons of generic data tests defined in open source packages, such as [dbt-utils](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/) and [dbt-expectations](https://hub.getdbt.com/calogica/dbt_expectations/latest/) — the test you're looking for might already be here! ##### Generic tests with standard arguments Generic tests are defined in SQL files. Those files can live in two places: * `tests/generic/`: that is, a special subfolder named `generic` within your [test paths](https://docs.getdbt.com/reference/project-configs/test-paths.md) (`tests/` by default) * `macros/`: Why? Generic tests work a lot like macros, and historically, this was the only place they could be defined. If your generic test depends on complex macro logic, you may find it more convenient to define the macros and the generic test in the same file. To define your own generic tests, simply create a `test` block called ``. All generic tests should accept one or both of the standard arguments: * `model`: The resource on which the test is defined, templated out to its relation name. (Note that the argument is always named `model`, even when the resource is a source, seed, or snapshot.) * `column_name`: The column on which the test is defined. Not all generic tests operate on the column level, but if they do, they should accept `column_name` as an argument. Here's an example of an `is_even` schema test that uses both arguments: tests/generic/test\_is\_even.sql ```sql {% test is_even(model, column_name) %} with validation as ( select {{ column_name }} as even_field from {{ model }} ), validation_errors as ( select even_field from validation -- if this is true, then even_field is actually odd! where (even_field % 2) = 1 ) select * from validation_errors {% endtest %} ``` If this `select` statement returns zero records, then every record in the supplied `model` argument is even! If a nonzero number of records is returned instead, then at least one record in `model` is odd, and the test has failed. To use this generic test, specify it by name in the `data_tests` property of a model, source, snapshot, or seed: (Applies to dbt v1.9 and later) models/\.yml ```yaml models: - name: users columns: - name: favorite_number data_tests: - is_even: description: "This is a test" ``` With one line of code, you've just created a test! In this example, `users` will be passed to the `is_even` test as the `model` argument, and `favorite_number` will be passed in as the `column_name` argument. You could add the same line for other columns, other models—each will add a new test to your project, *using the same generic test definition*. ##### Add description to generic data test logic You can add a description to the Jinja macro that provides the core logic for a data test by including the `description` key under the `macros:` section. You can add descriptions directly to the macro, including descriptions for macro arguments. Here's an example: macros/generic/schema.yml ```yaml macros: - name: test_not_empty_string description: Complementary test to default `not_null` test as it checks that there is not an empty string. It only accepts columns of type string. arguments: - name: model type: string description: Model Name - name: column_name type: string description: Column name that should not be an empty string ``` In this example: * When documenting custom test macros in a `schema.yml` file, add the `test_` prefix to the macro name. For example, if the test block's name is `not_empty_string`, then the macro's name would be `test_not_empty_string`. * We've provided a description at the macro level, explaining what the test does and any relevant notes. * Each argument (like `model`, `column_name`) also includes a description to clarify its purpose. ##### Generic tests with additional arguments The `is_even` test works without needing to specify any additional arguments. Other tests, like `relationships`, require more than just `model` and `column_name`. If your custom tests requires more than the standard arguments, include those arguments in the test signature, as `field` and `to` are included below: tests/generic/test\_relationships.sql ```sql {% test relationships(model, column_name, field, to) %} with parent as ( select {{ field }} as id from {{ to }} ), child as ( select {{ column_name }} as id from {{ model }} ) select * from child where id is not null and id not in (select id from parent) {% endtest %} ``` When calling this test from a `.yml` file, supply the arguments to the test in a dictionary. Note that the standard arguments (`model` and `column_name`) are provided by the context, so you do not need to define them again. (Applies to dbt v1.9 and later) models/\.yml ```yaml models: - name: people columns: - name: account_id data_tests: - relationships: description: "This is a test" arguments: # available in v1.10.5 and higher. Older versions can set the as the top-level property. to: ref('accounts') field: id ``` ##### Generic tests with default config values It is possible to include a `config()` block in a generic test definition. Values set there will set defaults for all specific instances of that generic test, unless overridden within the specific instance's `.yml` properties. tests/generic/warn\_if\_odd.sql ```sql {% test warn_if_odd(model, column_name) %} {{ config(severity = 'warn') }} select * from {{ model }} where ({{ column_name }} % 2) = 1 {% endtest %} ``` Any time the `warn_if_odd` test is used, it will *always* have warning-level severity, unless the specific test overrides that value: (Applies to dbt v1.9 and later) models/\.yml ```yaml models: - name: users columns: - name: favorite_number description: "Test favorite_number" data_tests: - warn_if_odd # default 'warn' - name: other_number description: "Test other_number" data_tests: - warn_if_odd: arguments: # available in v1.10.5 and higher. Older versions can set the as the top-level property. severity: error # overrides ``` ##### Customizing dbt's built-in tests To change the way a built-in generic test works—whether to add additional parameters, re-write the SQL, or for any other reason—you simply add a test block named `` to your own project. dbt will favor your version over the global implementation! tests/generic/\.sql ```sql {% test unique(model, column_name) %} -- whatever SQL you'd like! {% endtest %} ``` ##### Examples Here's some additional examples of custom generic ("schema") tests from the community: * [Creating a custom schema test with an error threshold](https://discourse.getdbt.com/t/creating-an-error-threshold-for-schema-tests/966) * [Using custom schema tests to only run tests in production](https://discourse.getdbt.com/t/conditionally-running-dbt-tests-only-running-dbt-tests-in-production/322) * [Additional examples of custom schema tests](https://discourse.getdbt.com/t/examples-of-custom-schema-tests/181) --- ## Category ### Account FAQs #### [📄️ Account-specific features](https://docs.getdbt.com/faqs/Accounts/account-specific-features.md) [Account-specific features](https://docs.getdbt.com/faqs/Accounts/account-specific-features.md) --- ### Available flags #### [📄️ Anonymous usage stats](https://docs.getdbt.com/reference/global-configs/usage-stats.md) [dbt Labs is on a mission to build the best version of dbt possible, and a crucial part of that is understanding how users work with dbt. To this end, we've added some simple event tracking (or telemetry) to dbt using Snowplow. Importantly, we do not track credentials, raw model contents, or model names: we consider these private, and frankly none of our business.](https://docs.getdbt.com/reference/global-configs/usage-stats.md) --- ### Behavior change flags #### [📄️ allow jinja file extensions](https://docs.getdbt.com/reference/global-configs/behavior-flags/allow_jinja_file_extensions.md) [| allowjinjafile\_extensions | v1 Latest | |](https://docs.getdbt.com/reference/global-configs/behavior-flags/allow_jinja_file_extensions.md) --- ### Cost Insights FAQs #### [📄️ Actual vs displayed costs]() [Explanation of why actual warehouse costs may differ from displayed costs]() --- ### dbt v1 FAQs #### [📄️ Installing dbt v1 with pip](https://docs.getdbt.com/faqs/Core/install-pip-best-practices.md) [Instructions on how to install dbt v1 with pip](https://docs.getdbt.com/faqs/Core/install-pip-best-practices.md) --- ### Documentation FAQs #### [📄️ Types of columns included in doc site](https://docs.getdbt.com/faqs/Docs/document-all-columns.md) [All columns appear in your docs site](https://docs.getdbt.com/faqs/Docs/document-all-columns.md) --- ### Environments FAQs #### [📄️ Custom branch settings](https://docs.getdbt.com/faqs/Environments/custom-branch-settings.md) [Use custom code from your repository](https://docs.getdbt.com/faqs/Environments/custom-branch-settings.md) --- ### General configs #### [📄️ Advanced usage](https://docs.getdbt.com/reference/advanced-config-usage.md) [Alternative SQL file config syntax](https://docs.getdbt.com/reference/advanced-config-usage.md) --- ### General properties #### [📄️ anchors](https://docs.getdbt.com/reference/resource-properties/anchors.md) [Use the anchors key to hold reusable YAML config fragments so they pass file validation.](https://docs.getdbt.com/reference/resource-properties/anchors.md) --- ### Git FAQs #### [📄️ Branches when changing providers or repositories](https://docs.getdbt.com/faqs/Git/branch-migration.md) [How Git branch names and commit history carry over when you change providers, repositories, or remotes—for dbt v1 and the dbt platform, including platform-only notes for Studio IDE and repository settings.](https://docs.getdbt.com/faqs/Git/branch-migration.md) --- ### Jinja FAQs #### [📄️ Compiled sql has a lot of white space](https://docs.getdbt.com/faqs/Jinja/jinja-whitespace.md) [Managing whitespace control](https://docs.getdbt.com/faqs/Jinja/jinja-whitespace.md) --- ### Jinja reference #### [🗃️ dbt Jinja context functions](https://docs.getdbt.com/reference/dbt-jinja-functions-context-variables.md) [47 items](https://docs.getdbt.com/reference/dbt-jinja-functions-context-variables.md) --- ### List of commands #### [📄️ build](https://docs.getdbt.com/reference/commands/build.md) [The dbt build command will:](https://docs.getdbt.com/reference/commands/build.md) --- ### Models FAQs #### [📄️ Model configurations](https://docs.getdbt.com/faqs/Models/available-configurations.md) [Learning about model configurations](https://docs.getdbt.com/faqs/Models/available-configurations.md) --- ### Project configs #### [📄️ dbt\_project.yml](https://docs.getdbt.com/reference/dbt_project.yml.md) [Reference guide for configuring the dbt\_project.yml file.](https://docs.getdbt.com/reference/dbt_project.yml.md) --- ### Project FAQs #### [📄️ Add a seed file](https://docs.getdbt.com/faqs/Project/add-a-seed.md) [Learn how to add a seed file to your project](https://docs.getdbt.com/faqs/Project/add-a-seed.md) --- ### Runs FAQs #### [📄️ Reviewing SQL that dbt runs](https://docs.getdbt.com/faqs/Runs/checking-logs.md) [Review logs to check the SQL dbt is running](https://docs.getdbt.com/faqs/Runs/checking-logs.md) --- ### Seeds FAQs #### [📄️ Build one seed at a time](https://docs.getdbt.com/faqs/Seeds/build-one-seed.md) [Use select flag to build one seed at a time](https://docs.getdbt.com/faqs/Seeds/build-one-seed.md) --- ### Setting flags #### [📄️ Command line options](https://docs.getdbt.com/reference/global-configs/command-line-options.md) [For consistency, command-line interface (CLI) flags should come right after the dbt prefix and its subcommands. This includes "global" flags (supported for all commands). For the full list of global CLI options for or (depending on your selected docs version), refer to Available flags. When set, CLI flags override environment variables and project flags.](https://docs.getdbt.com/reference/global-configs/command-line-options.md) --- ### Snapshots FAQs #### [📄️ Use hooks to run with snapshots](https://docs.getdbt.com/faqs/Snapshots/snapshot-hooks.md) [Run hooks with snapshots](https://docs.getdbt.com/faqs/Snapshots/snapshot-hooks.md) --- ### Tests FAQs #### [📄️ Available data tests to use in dbt](https://docs.getdbt.com/faqs/Tests/available-tests.md) [Types of data tests to use in dbt](https://docs.getdbt.com/faqs/Tests/available-tests.md) --- ### Troubleshooting FAQs #### [📄️ Generate HAR files](https://docs.getdbt.com/faqs/Troubleshooting/generate-har-file.md) [How to generate HAR files for debugging](https://docs.getdbt.com/faqs/Troubleshooting/generate-har-file.md) --- ### Warehouse FAQs #### [📄️ Setting up permissions in BigQuery"](https://docs.getdbt.com/faqs/Warehouse/bq-impersonate-service-account-setup.md) [Use service account to set up permissions in BigQuery](https://docs.getdbt.com/faqs/Warehouse/bq-impersonate-service-account-setup.md) --- ## Community ### Become a contributor #### Want to get involved? Start here The dbt Community predates dbt Labs as an organization and harkens back to the days when a scrappy analytics consultancy of a few [pissed off data analysts](https://www.hashpath.com/2020/12/an-analytics-engineer-is-really-just-a-pissed-off-data-analyst/#:~:text=Often%20times%2C%20an%20analytics%20engineer,necessity%20\(and%20genius%20branding\).) started hacking together an open source project around which gathered a community that would change how the world uses data. The dbt Community exists to allow analytics practitioners to share their knowledge, help others and collectively to drive forward the discipline of analytics engineering. This is something that can’t be done by any one individual or any one organization - to create a new discipline is necessarily a community effort. The only reason that dbt has become as widespread as it has is because people like you choose to get involved and share your knowledge. Contributing to the community can also be a great way to learn new skills, build up a public portfolio and make friends with other practitioners. There are opportunities here for everyone to get involved, whether you are just beginning your analytics engineering journey or you are a seasoned data professional. Contributing isn’t about knowing all of the answers, it’s about learning things together. Below you’ll find a sampling of the ways to get involved. There are a lot of options but these are ultimately just variations on the theme of sharing knowledge with the broader community. [![](/img/icons/pencil-paper.svg)](https://docs.getdbt.com/community/contributing/contributing-writing.md) ###### [Writing contributions](https://docs.getdbt.com/community/contributing/contributing-writing.md) [Learn how to share and grow the collective knowledge of the dbt Community through blogs, guides, and documentation.](https://docs.getdbt.com/community/contributing/contributing-writing.md) [![](/img/icons/folder.svg)](https://docs.getdbt.com/community/contributing/contributing-coding.md) ###### [Coding contributions](https://docs.getdbt.com/community/contributing/contributing-coding.md) [The dbt Community supports a wide variety of open source and source-available projects, and this software is at the heart of everything we do. Learn how to get involved with projects in the dbt ecosystem.](https://docs.getdbt.com/community/contributing/contributing-coding.md) [![](/img/icons/discussions.svg)](https://docs.getdbt.com/community/contributing/contributing-online-community.md) ###### [Online community building](https://docs.getdbt.com/community/contributing/contributing-online-community.md) [Getting involved in the dbt Community Forum or Slack is one of the best entry points for contributing. Share your knowledge and learn from others.](https://docs.getdbt.com/community/contributing/contributing-online-community.md) [![](/img/icons/calendar.svg)](https://docs.getdbt.com/community/contributing/contributing-realtime-events.md) ###### [Realtime event participation](https://docs.getdbt.com/community/contributing/contributing-realtime-events.md) [Want to speak at a Meetup or conference? Learn how to get involved and check out best practices for crafting a talk that everyone will remember.](https://docs.getdbt.com/community/contributing/contributing-realtime-events.md) --- ### Coding contributions ##### Contribute to dbt Packages ###### Overview [dbt Packages](https://docs.getdbt.com/docs/build/packages.md) are the easiest way for analytics engineers to get involved with contributing code to the dbt Community, because dbt Packages are just standard [dbt Projects](https://docs.getdbt.com/docs/build/projects.md). If you can create a dbt Project, write a macro, and ref a model: you can make a dbt Package. Packages function much like libraries do in other programming languages. They allow for prewritten, modularized development of code to solve common problems in analytics engineering. You can view all dbt Packages on the [dbt Package Hub](https://hub.getdbt.com/). ###### Contribution opportunities * Create a new package for the dbt Package Hub. This might be a new set of macros or tests that have been useful to you in your projects, a set of models for engaging with a commonly used datasource or anything else that can be done from within a dbt project. * Improve an existing package: Alternatively you can help improve an existing package. This can be done by creating and engaging with Issues or by functionality to address an existing issue via opening a PR. ###### Sample contributions * [dbt Expectations](https://hub.getdbt.com/calogica/dbt_expectations/latest/) * [dbt Artifacts](https://hub.getdbt.com/brooklyn-data/dbt_artifacts/latest/) ###### Get started * Use packages in your own projects! The best way to know how to improve a package is to use it in a production environment then look for ways it can be modified or improved. * Read the following resources on package development: * [So You Want to Build a dbt Package](https://docs.getdbt.com/blog/so-you-want-to-build-a-package) * [Package Best Practices](https://github.com/dbt-labs/hubcap/blob/main/package-best-practices.md) * Need help: Visit #package-ecosystem in the dbt Slack ##### Contribute to dbt open source or source-available software ###### Overview dbt (including v1, v2, and adapters), as well as the sites powering the Package Hub and Developer Hub, are all vibrant community projects. Unlike dbt Packages, contributing code to these projects typically requires some working knowledge of programming languages outside of SQL and Jinja, but the supportive community around these repositories can help you advance those skills. Even without contributing code, there are many ways to be part of communal development in these projects, detailed below. You can find a curated list of the most active OSS/SA projects that dbt Labs supports [here](https://docs.getdbt.com/community/resources/oss-sa-projects.md). ###### Contribution opportunities There are three primary ways to contribute to the dbt projects. We’ll use dbt v1 as an example, as the "front door" to the dbt ecosystem and a great place to start for newcomers: * [Open an issue](https://github.com/dbt-labs/dbt/issues/new/choose) to suggest an improvement or give feedback. * Comment on or engage with existing [issues](https://github.com/dbt-labs/dbt/issues) or [discussions](https://github.com/dbt-labs/dbt/discussions). For example, upvote issues that would help your organization, comment to add nuance to a feature request, or share how a feature would impact your use of dbt. * Create a pull request that resolves an open Issue. This involves writing the code and tests that add the feature/resolve the bug described in an Issue, and then going through the code review process asynchronously with a dbt Labs engineer. Note that signed commits are required when contributing to dbt v1. For steps on how to sign commits, see [Signing commits](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits). ###### Sample contributions * Check out [this issue](https://github.com/dbt-labs/dbt/issues/3612) about improving error messages and [the PR that the community contributed to fix it](https://github.com/dbt-labs/dbt/pull/3703). * From the above issue [another issue was generated](https://github.com/dbt-labs/dbt-bigquery/issues/202) to change not just the error message, but improve the behavior. This is the virtuous cycle of community-driven development! Bit by bit we, the community, craft the tool to better fit our needs. ###### Get started * Read the dbt v1 [contribution guide](https://github.com/dbt-labs/dbt/blob/main/CONTRIBUTING.md) and the [Contributor Expectations](https://docs.getdbt.com/community/resources/contributor-expectations.md). * If contributing to `dbt-labs/dbt`, find an issue labeled “[good first issue](https://github.com/dbt-labs/dbt/issues?q=is%3Aopen+is%3Aissue+label%3A%22type%3Agood-first-issue%22)”, or look for similar labels on other repositories. If in doubt, also feel free to ask the maintainers for a good first issue, they’ll be excited to welcome you! ###### Need help? The following channels in the dbt Community Slack are a great place to ask questions: * \#dbt-development * \#adapter-ecosystem --- ### Community Forum guidelines #### What is the dbt Community Forum? [The dbt Community Forum](https://discourse.getdbt.com) is the platform for getting support with dbt as well as to have long-lived discussions about dbt, analytics engineering, and analytics. It's a place for us to build up a long-lasting knowledge base around the common challenges, opportunities, and patterns we work with every day. The forum is different from the dbt Community Slack in a few key ways, most importantly it is: * **Asynchronous** and **long-lived** - sometimes conversations continue over weeks, months, and beyond. * **Intentional** - we recommend taking at least 5 to 10 minutes thinking about and shaping your initial post and any comments. * **Citable** - Slack conversations tend to be great in the moment but get lost in the flow — forum posts can more easily shared and referenced. **Guidelines for engaging on the Forum** The community [Rules of the Road](https://docs.getdbt.com/community/resources/community-rules-of-the-road.md) apply, and following them is the best way to get momentum behind your discussion or answers to your questions. The following guidelines will set you up for success: * Be respectful * Put effort into your posts * Mark replies as Solutions in the Help section * Don’t double post #### Categories The forum is broken down into three categories: * [Help](https://discourse.getdbt.com/c/help/19) * This is a Q\&A style forum where you can ask the dbt Community for help with specific questions about dbt, dbt, data modeling, or anything else you want a definitive answer on. * This category is for questions which can plausibly have a *single correct answer*. * ✅ How do I debug this Jinja error? * ✅ How do I set up CI in dbt on GitHub? * ❌ What is the best way to do marketing attribution? (More general Discussions like this are perfect for the [In-Depth Discussions](https://discourse.getdbt.com/c/discussions/21) category) * [Show and Tell](https://discourse.getdbt.com/c/show-and-tell/22) * This is the place to show off all of the cool things you are doing in dbt. Whether it’s a new macro, design pattern, or package, post here to show the community what you are up to! * [In-Depth Discussions](https://discourse.getdbt.com/c/discussions/21) * Share anything you’re thinking about that has to do with dbt or analytics engineering! This is a great place to jot down some thoughts to share with the community or spark a discussion on a topic that’s currently interesting you. #### Inclusivity on the Community Forum We are **strongly** committed to building a community where everyone can feel welcome. The dbt community started with people who were not traditionally considered “technical”, did not have ownership over technical systems, and were often left out of organizational decision-making. We came together to learn from each other, solve hard problems, and help build a new discipline where data folks have greater ownership over our own work. It really matters to us that everyone feels like they can ask questions and engage, no matter their professional or personal background. Technical forums have the potential to replicate harmful power structures, and can feel intimidating or hostile. We are working hard to create and sustain an inclusive environment through community-building, technological solutions, inclusive content, and diverse contributors. This is a long-term project, and we will continue to iterate and make improvements. If you have any ideas or feedback on how to make this space friendlier or more inclusive please let us know, either on the community Slack in the #community-strategy channel or via email at . We want to hear from you! #### Following new and ongoing Discussions The best way to stay up to date is to [browse the forum](https://discourse.getdbt.com/) directly. You can also Track or Watch specific threads or the whole category to receive updates on them without commenting. Each category also has a companion Slack channel (#advice-dbt-for-beginners, #show-and-tell and #in-depth-discussions). You can reply to the initial post in Slack and it will be added as a comment on the forum thread, allowing you to participate from inside Slack if you prefer. --- ### Contributor License Agreements #### Why we have a CLA As the sponsor of dbt, dbt Labs would like to ensure the long-term viability of dbt and its community. The Contributor License Agreement helps ensure everyone can enjoy dbt with confidence that dbt is here to stay. Specifically, our Contributor License Agreements (CLAs) grant the contributor and dbt Labs joint copyright interest in contributed code. Further, it provides assurance from the contributor that contributions are original work that does not violate any third-party license agreement. The agreement between contributors and project is explicit, so dbt users can be confident in the legal status of the source code and their right to use it. #### Our CLAs For all code contributions to dbt, we ask that contributors complete and sign a Contributor License Agreement. We have two different CLAs, depending on whether you are contributing to dbt in a personal or professional capacity: * [Individual Contributor License Agreement v1.0](https://docs.google.com/forms/d/e/1FAIpQLScfOV7K4enYRHozrDRP6BBIXjOij-JDGca6WBTHyP_ANXSqlg/viewform?usp=sf_link) * [Software Grant and Corporate Contributor License Agreement v1.0](https://docs.google.com/forms/d/e/1FAIpQLScDSTwGIlVyGWCMMvmszaXSE5IhIIRyeLQkgWf1-CSC2RnLww/viewform?usp=sf_link) * [Licenses FAQ](http://www.getdbt.com/licenses-faq) #### For Lawyers Our individual and corporate CLAs are based on the Contributor License Agreements published by the [Apache Software Foundation](http://www.apache.org/), with modifications: * [Individual Contributor License Agreement ("Agreement") V2.0](http://www.apache.org/licenses/icla.txt) * [Software Grant and Corporate Contributor License Agreement ("Agreement") v r190612](http://www.apache.org/licenses/cla-corporate.txt) * [Licenses FAQ](http://www.getdbt.com/licenses-faq) If you have questions about these CLAs, please contact us at . --- ### dbt Community Code of Conduct This Code of Conduct applies to all dbt Community spaces, both online and offline. This includes Slack, Discourse, code repositories (dbt v1, dbt v2, dbt packages, etc.), dbt Labs virtual events, and Meetups. Participants are responsible for knowing and abiding by this Code of Conduct. This Code of Conduct has two sections: * **dbt Community Values:** These values apply to all of our community spaces, and all of our guidelines are based on these values. * **Anti-harassment policy:** We are dedicated to providing a harassment-free experience for everyone in our community — here, we outline exactly what that means. We appreciate your support in continuing to build a community we’re all proud of. — The dbt Community Admin Team. #### dbt Community Values ##### Create more value than you capture. Each community member should strive to create more value in the community than they capture. This is foundational to being a community. Ways to demonstrate this value: * [Coding contributions](https://docs.getdbt.com/community/contributing/contributing-coding.md): Contribute to dbt v1, a package, or an adapter. Beyond implementing new functionality, you can also open issues or participate in discussions. * [Writing contributions](https://docs.getdbt.com/community/contributing/contributing-writing.md): You can suggest edits to every page of the dbt documentation, or suggest a topic for the dbt Developer Blog. * [Join in online](https://docs.getdbt.com/community/contributing/contributing-online-community.md): Ask and answer questions on the Discourse forum, kick off a lively discussion in Slack, or even maintain a Slack channel of your own. * [Participate in events](https://docs.getdbt.com/community/contributing/contributing-realtime-events.md): Organise a community Meetup, speak at an event, or provide office space/sponsorship for an existing event. ##### Be you. Some developer communities allow and even encourage anonymity — we prefer it when people identify themselves clearly. It helps to build empathy, and form relationships. Ways to demonstrate this value: * Update your profile on dbt Community platforms to include your name and a clear picture of yourself. Where available, use the "what I do" section to add your role, title and current company. * Join your `#local-` channel in Slack, or if it doesn't exist then propose a new one. * Write in your own voice, and offer your own advice, rather than speaking in your company’s marketing or support voice. ##### Encourage diversity and participation. People with different mindsets and experiences, working together, create better outcomes. This includes diversity of race and gender, as well as the diversity of academic and career backgrounds, socio-economic backgrounds, geographic backgrounds, ideologies, and interests. Ways to demonstrate this value: * Make everyone in our community feel welcome, regardless of their background, and do everything possible to encourage participation in our community. * Demonstrate empathy for a community member’s experience — not everyone comes from the same career background, so adjust answers accordingly. * If you are sourcing speakers for events, put in additional effort to find speakers from underrepresented groups. ##### Be curious. Always ask yourself "why?" and strive to be continually learning. Ways to demonstrate this value: * Try solving a problem yourself before asking for help, e.g. rather than asking "what happens when I do X", experiment and observe the results! * When asking questions, explain the "why" behind your decisions, e.g. "I’m trying to solve X problem, by writing Y code. I’m getting Z problem" * When helping someone else, explain why you chose that solution, or if no solution exists, elaborate on the reason for that, e.g. "That’s not possible in dbt today — but here’s a workaround / check out this GitHub issue for a relevant discussion" #### Anti-harassment policy We are dedicated to providing a harassment-free experience for everyone. We do not tolerate harassment of participants in any form. Harassment includes: * Offensive comments related to gender, gender identity and expression, sexual orientation, disability, mental illness, neuro(a)typicality, physical appearance, body size, age, race, or religion. * Unwelcome comments regarding a person’s lifestyle choices and practices, including those related to food, health, parenting, drugs, and employment. * Deliberate misgendering or use of ‘dead’ or rejected names. * Gratuitous or off-topic sexual images or behaviour in spaces where they’re not appropriate. * Physical contact and simulated physical contact (eg, textual descriptions like "*hug*" or "*backrub*") without consent or after a request to stop. * Threats of violence. * Incitement of violence towards any individual, including encouraging a person to commit suicide or to engage in self-harm. * Deliberate intimidation. * Stalking or following. * Harassing photography or recording, including logging online activity for harassment purposes. * Sustained disruption of discussion. * Unwelcome sexual attention. * Pattern of inappropriate social contact, such as requesting/assuming inappropriate levels of intimacy with others * Continued one-on-one communication after requests to cease. * Deliberate "outing" of any aspect of a person’s identity without their consent except as necessary to protect vulnerable people from intentional abuse. * Publication of non-harassing private communication. Be mindful that others may not want their image or name on social media. Ask permission prior to posting about another person at in-person events. The dbt Community prioritizes marginalized people’s safety over privileged people’s comfort. The dbt Community Admin team reserves the right not to act on complaints regarding: * ‘Reverse’ -isms, including ‘reverse racism,’ ‘reverse sexism,’ and ‘cisphobia’ * Reasonable communication of boundaries, such as "leave me alone," "go away," or "I’m not discussing this with you." * Communicating in a ‘tone’ you don’t find congenial * Criticizing racist, sexist, cissexist, or otherwise oppressive behavior or assumptions ##### Reporting harassment If you are being harassed by a member of the dbt Community, notice that someone else is being harassed, or have any other concerns, please contact us at or use the workflows in [#moderation-and-administration](https://getdbt.slack.com/archives/C02JJ8N822H) on Slack. We will respect confidentiality requests for the purpose of protecting victims of abuse. At our discretion, we may publicly name a person about whom we’ve received harassment complaints, or privately warn third parties about them, if we believe that doing so will increase the safety of dbt community members or the general public. We will not name harassment victims without their affirmative consent. ##### Consequences Participants asked to stop any harassing behavior are expected to comply immediately. If a participant engages in harassing behavior, the dbt Community Admin team may take any action they deem appropriate, up to and including expulsion from all dbt Community spaces and identification of the participant as a harasser to other dbt Community members or the general public. #### dbt Summit and other events We reserve the right to prohibit certain directly competitive companies from attending and / or sponsoring dbt Summit and other dbt Labs-hosted events. While we actively support the broader ecosystem around dbt, it is not the right business decision for us to allow directly competitive companies to market to or sell to our users and customers at events that we invest very significant company resources into hosting. Any event declines will be handled directly with the individuals / companies in question, and full refunds will be issued. Events that are hosted by other members of the dbt community, such as most dbt Community Meetups, are free to make their own guidelines about attendance. #### Credits Credit to [01.org](https://01.org/community/participation-guidelines), [Tizen.org](https://www.tizen.org/community/guidelines), and [Geek Feminism](https://geekfeminism.wikia.org/wiki/Community_anti-harassment/Policy) for some of the wording used in this Code of Conduct. --- ### dbt Community Rules of the Road As of June 2023, the dbt Community includes over 50,000 data professionals and is still growing. People genuinely love this community. It's filled with smart, kind, and helpful people who share our commitment to elevating the analytics profession. We are committed to maintaining the spirit of this community, and have written these rules alongside its members to help everyone understand how to best participate. We appreciate your support in continuing to build a community we're all proud of. #### Expectations for all members ##### Rule 1: Be respectful We want everyone in this community to have a fulfilling and positive experience. Therefore, this first rule is serious and straightforward; we simply will not tolerate disrespectful behavior of any kind. Everyone interacting on a dbt platform – including Slack, the forum, codebase, issue trackers, and mailing lists – is expected to follow the [Community Code of Conduct](https://docs.getdbt.com/community/resources/code-of-conduct.md). If you are unable to abide by the code of conduct set forth here, we encourage you not to participate in the community. ##### Rule 2: Keep it in public spaces Unless you have someone's express permission to contact them directly, do not directly message other community members, whether on a dbt Community platform or other spaces like LinkedIn. We highly value the time community members put into helping each other, and we have precisely zero tolerance for people who abuse their access to experienced professionals. If you are being directly messaged with requests for assistance without your consent, let us know in the [#moderation-and-administration](https://getdbt.slack.com/archives/C02JJ8N822H) Slack channel. We will remove that person from the community. Your time and attention is valuable. ##### Rule 3: Follow messaging etiquette In short: put effort into your question, use threads, post in the right channel, and do not seek extra attention by tagging individuals or double-posting. For more information, see our [guide on getting help](https://docs.getdbt.com/community/resources/getting-help.md). ##### Rule 4: Do not solicit community members This community is built for data practitioners to discuss the work that they do, the ideas that they have, and the things that they are learning. It is decidedly not intended to be lead generation for vendors or recruiters. Vendors and recruiters are subject to additional rules to ensure this space remains welcoming to everyone. These requirements are detailed below and are enforced vigorously. #### Vendor expectations As a vendor/dbt partner, you are also a member of this community, and we encourage you to participate fully in the space. We have seen folks grow fantastic user relationships for their products when they come in with the mindset to share rather than pushing a pitch. At the same time, active community members have a finely honed sense of when they are being reduced to an audience or a resource to be monetized, and their response is reliably negative. Who is a vendor? Vendors are generally individuals belonging to companies that are creating products or services primarily targeted at data professionals, but this title also includes recruiters, investors, open source maintainers (with or without a paid offering), consultants and freelancers. If in doubt, err on the side of caution. ##### Rule 1: Identify yourself Include your company in your display name, e.g. "Alice (DataCo)". When joining a discussion about your product (after the waiting period below), be sure to note your business interests. ##### Rule 2: Let others speak first If a community member asks a question about your product directly, or mentions that they have a problem that your product could help with, wait 1 business day before responding to allow other members to share their experiences and recommendations. (This doesn't apply to unambiguously support-style questions from existing users, or in your `#tools-` channel if you have one). ##### Rule 3: Keep promotional content to specified spaces As a space for professional practice, the dbt Community is primarily a non-commercial space. However, as a service to community members who want to be able to keep up to date with the data industry, there are several areas available on the Community Slack for vendors to share promotional material: * [#vendor-content](https://getdbt.slack.com/archives/C03B0Q4EBL3) * [#events](https://getdbt.slack.com/archives/C80RCAZ5E) * \#tools-\* (post in [#moderation-and-administration](https://getdbt.slack.com/archives/C02JJ8N822H) to request a channel for your tool/product) Recruiters may also post in [#jobs](https://getdbt.slack.com/archives/C7A7BARGT)/[#jobs-eu](https://getdbt.slack.com/archives/C04JMHHK6CD) but may not solicit applications in DMs. The definition of "vendor content" can be blurry at the edges, and we defer to members' instincts in these scenarios. As a rule, if something is hosted on a site controlled by that company or its employees (including platforms like Substack and Medium), or contains a CTA such as signing up for a mailing list or trial account, it will likely be considered promotional. ##### One more tip: Be yourself Speak in your own voice, and join in any or all of the conversations that interest you. Share your expertise as a data professional. Make a meme if you're so inclined. Get in a (friendly) debate. You are not limited to only your company's products and services, and making yourself known as a familiar face outside of commercial contexts is one of the most effective ways of building trust with the community. Put another way, [create more value than you capture](https://docs.getdbt.com/community/resources/code-of-conduct.md#create-more-value-than-you-capture). Because unaffiliated community members are able to share links in any channel, the most effective way to have your work reach a wider audience is to create things that are genuinely useful to the community. #### Handling violations The point of these rules is not to find opportunities to punish people, but to ensure the longevity of the community. Participation in this community is a privilege, and we reserve the right to remove people from it. To report an issue or appeal a judgement, email or use the workflows in [#moderation-and-administration](https://getdbt.slack.com/archives/C02JJ8N822H) on Slack. Violations related to our anti-harassment policy will result in immediate removal. Other issues are handled in proportion to their impact, and may include: * a friendly, but public, reminder that the behavior is inappropriate according to our guidelines. * a private message with a warning that any additional violations will result in removal from the community. * temporary or permanent suspension of your account. --- ### dbt Labs Community #jobs Channels Terms and Conditions I agree to abide by the [dbt Community Code of Conduct](https://docs.getdbt.com/community/resources/code-of-conduct.md) and all laws applicable to me in my use of the dbt Community's #jobs channels. I further agree: * dbt Labs is not responsible for not does it warrant or guarantee the validity, accuracy, completeness, legality, or reliability of any functionality of any #jobs channel, any posting's content, or any application and/or solicitation of any kind of employment. * dbt Labs does not review and approve job-related content. * dbt Labs disclaims liability of any kind whatsoever for any type of damage that occurs while using the community Slack for job-related reasons, and I waive any type of claim (including actual, special or consequential damages) to the maximum extent permitted by law. * Without limitation, dbt Labs disclaims liability for quality, performance, merchantability, and fitness for a particular purpose, express or implied, that may arise out of my use of the community Slack for job-related content, my reliance on such information, and/or my provision/receipt of job-related information. * I understand that no internet-based site is without risk, and my use is at my own risk. * My use of any job-posting template (or other forum for providing job-related information) confirms my consent to provide the data posted, confirms that I have permission to post such data, and is subject to the terms of the [dbt Labs privacy policy](https://www.getdbt.com/cloud/privacy-policy). For further information, please contact . --- ### Expectations for dbt contributors Whether it's `dbt-labs/dbt`, adapters, packages, or this very documentation site, contributing to the open source or source-available code that supports the dbt ecosystem is a great way to share your knowledge, level yourself up as a developer, and to give back to the community. The goal of this page is to help you understand what to expect when contributing to dbt ecosystem projects. Have you seen things in other projects that you like, and think we could learn from? [Open a discussion on the dbt Community Forum](https://discourse.getdbt.com), or start a conversation in the [dbt Community Slack](https://www.getdbt.com/community/join-the-community) (for example: `#community-strategy`, `#dbt-development`, `#package-ecosystem`, `#adapter-ecosystem`). We always appreciate hearing from you! #### Principles ##### dbt is a team sport We all build dbt together -- whether you write code or contribute your ideas. By using dbt, you're invested in the future of the tool, and have an active role in pushing forward the standard of analytics engineering. You already benefit from using code and documentation contributed by community members. Contributing to the dbt community is your way to be an active participant in the thing we're all creating together. There's a very practical reason, too: building in public prioritizes our collective knowledge and experience over any one person's. We don't have experience using every database, operating system, security environment, ... We rely on the community of users to hone our product capabilities and documentation to the wide variety of contexts in which it operates. In this way, dbt gets to be the handiwork of thousands, rather than a few dozen. ##### We take seriously our role as maintainers of a standard As a standard, dbt must be reliable and consistent. Our first priority is ensuring the continued high quality of existing dbt capabilities before we introduce net-new capabilities. We also believe dbt as a framework should be extensible enough to ["make the easy things easy, and the hard things possible"](https://en.wikipedia.org/wiki/Perl#Philosophy). To that end, we *don't* believe it's appropriate for dbt to have an out-of-the-box solution for every niche problem. Users have the flexibility to achieve many custom behaviors by defining their own macros, materializations, hooks, and more. We view it as our responsibility as maintainers to decide when something should be "possible" — via macros, packages, etc. — and when something should be "easy" — built into the dbt standard. So when will we say "yes" to new capabilities for dbt? The signals we look for include: * Upvotes on issues in our GitHub repos * Open source dbt packages trying to close a gap * Technical advancements in the ecosystem In the meantime — we'll do our best to respond to new issues with: * Clarity about whether the proposed feature falls into the intended scope of dbt's source-available components * Context (including links to related issues) * Alternatives and workarounds * When possible, pointers to code that would aid a community contributor ##### Initiative is everything Given that we, as maintainers, will not be able to resolve every bug or flesh out every feature request, we empower you, as a community member, to initiate a change. * If you open the bug report, it's more likely to be identified. * If you open the feature request, it's more likely to be discussed. * If you comment on the issue, engaging with ideas and relating it to your own experience, it's more likely to be prioritized. * If you open a PR to fix an identified bug, it's more likely to be fixed. * If you comment on an existing PR, to confirm it solves the concrete problem for your team in practice, it's more likely to be merged. Sometimes, this can feel like shouting into the void, especially if you aren't met with an immediate response. We promise that there are dozens (if not hundreds) of folks who will read your comment, including us as maintainers. It all adds up to a real difference. #### Practicalities ##### Discussions A discussion is best suited to propose a Big Idea, such as brand-new capability in the v1 framework. Anyone can open a discussion, comment on an existing one, or reply in a thread. When you open a new discussion, you might be looking for validation from other members of the community — folks who identify with your problem statement, who like your proposed idea, and who may have their own ideas for how it could be improved. The most helpful comments propose nuances or desirable user experiences to be considered in design and refinement. Unlike an **issue**, there is no specific code change that would “resolve” a discussion. If, over the course of a discussion, we reach a consensus on specific elements of a proposed design, we can open new implementation issues that reference the discussion for context. Those issues will connect desired user outcomes to specific implementation details, acceptance testing, and remaining questions that need answering. ##### Issues An issue could be a bug you've identified while using the product or reading the documentation. It could also be a specific idea you've had for a narrow extension of existing functionality. ###### Best practices for issues * Issues are **not** for support / troubleshooting / debugging help. Please see [dbt support](https://docs.getdbt.com/docs/dbt-support.md) for more details and suggestions on how to get help. * Always search existing issues first, to see if someone else had the same idea / found the same bug you did. * Many dbt repositories offer templates for creating issues, such as reporting a bug or requesting a new feature. If available, please select the relevant template and fill it out to the best of your ability. This information helps us (and others) understand your issue. ###### You've found an existing issue that interests you. What should you do? Comment on it! Explain that you've run into the same bug, or had a similar idea for a new feature. If the issue includes a detailed proposal for a change, say which parts of the proposal you find most compelling, and which parts give you pause. ###### You've opened a new issue. What can you expect to happen? In our most critical repositories (such as `dbt-labs/dbt`), our goal is to respond to new issues as soon as possible. This initial response will often be a short acknowledgement that the maintainers are aware of the issue, signalling our perception of its urgency. Depending on the nature of your issue, it might be well suited to an external contribution, from you or another community member. **What if you're opening an issue in a different repository?** We have engineering teams dedicated to active maintenance of [`dbt-labs/dbt`](https://github.com/dbt-labs/dbt), home to both v1 and v2. Some things to note about the `dbt-labs/dbt` repo when you're contributing: * v2 lives on the `main` branch * v1 lives on the `latest.1` branch, as well as the component libraries of v1 ([`dbt-common`](https://github.com/dbt-labs/dbt-common) + [`dbt-adapters`](https://github.com/dbt-labs/dbt-adapters) (which also includes the dbt Labs managed adapters). `dbt-adapters` applies to v1 only. We've open-sourced a number of other software projects over the years, and the majority of them do not have the same activity or maintenance guarantees. Check to see if other recent issues have responses, or when the last commit was added to the `main` branch. **You're not sure about the status of your issue.** If your issue is in an actively maintained repo and has a `triage` label attached, we're aware it's something that needs a response. If the issue has been triaged, but not prioritized, this could mean: * The intended scope or user experience of a proposed feature requires further refinement from a maintainer * We believe the required code change is too tricky for an external contributor We'll do our best to explain the open questions or complexity, and when / why we could foresee prioritizing it. **Automation that can help us:** In many repositories, we use a bot that marks issues as stale if they haven't had any activity for 180 days. This helps us keep our backlog organized and up-to-date. We encourage you to comment on older open issues that you're interested in, to keep them from being marked stale. You're also always welcome to comment on closed issues to say that you're still interested in the proposal. ###### Issue labels In all likelihood, the maintainer who responds will also add a number of labels. Not all of these labels are used in every repository. In some cases, the right resolution to an open issue might be tangential to the codebase. The right path forward might be in another codebase (we'll transfer it), a documentation update, or a change that you can make yourself in user-space code. In other cases, the issue might describe functionality that the maintainers are unwilling or unable to incorporate into the main codebase. In these cases, a maintainer will close the issue (perhaps using a `wontfix` label) and explain why. Many labels follow a namespaced `facet:value` model, where each facet answers one question about an issue — the kind of work it is, which subsystem it touches, which dbt engine it targets, and so on. Some facets take exactly one value; others can take several. You don't need to apply these yourself, but knowing what they mean helps you follow an issue's status. | Facet | Example values | What it means | | ------------- | --------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | | `type:` | `type:bug`, `type:feature`, `type:docs`, `type:tech-debt`, `type:epic`, `type:good-first-issue` | The kind of work. Exactly one per issue. | | `area:` | `area:static-analysis`, `area:engine`, `area:compute`, `area:adapters`, `area:vscode`, `area:wizard`, `area:autofix`, `area:cicd` | Which subsystem the issue touches. | | `adapter:` | `adapter:bigquery`, `adapter:snowflake`, `adapter:databricks`, `adapter:postgres`, … | Which adapter or SQL dialect is involved. | | `engine:` | `engine:v1`, `engine:v2` | Which dbt engine the issue concerns: `engine:v1` is the Python engine (dbt v1); `engine:v2` is the v2 Rust rewrite. | | `priority:` | `priority:p0`, `priority:p1`, `priority:p2` | Maintainer triage urgency. | | `status:` | `status:triage`, `status:needs-repro`, `status:has-repro`, `status:ready-for-review` | Where the issue sits in the workflow. | | `resolution:` | `resolution:wontfix` | Why an issue was closed without a code change. | Automation labels new issues for triage — currently with both the faceted `status:triage` and the legacy `triage` label while it migrates to the faceted set — to mark them as awaiting a maintainer's first response. Older issues may also carry previous, un-namespaced labels such as `bug`, `enhancement`, `good_first_issue`, `help_wanted`, `duplicate`, `stale`, and `wontfix`. These map onto the faceted labels above (`bug` → `type:bug`, `enhancement` → `type:feature`, `good_first_issue` → `type:good-first-issue`, `wontfix` → `resolution:wontfix`, and so on) and remain in place for continuity. You may see either set while the transition settles. ##### Pull requests **Every PR should be associated with an issue.** Why? Before you spend a lot of time working on a contribution, we want to make sure that your proposal will be accepted. You should open an issue first, describing your desired outcome and outlining your planned change. If you've found an older issue that's already open, comment on it with an outline for your planned implementation *before* putting in the work to open a pull request. **PRs must include robust testing.** Comprehensive testing within pull requests is crucial for the stability of dbt. By prioritizing robust testing, we ensure the reliability of our codebase, minimize unforeseen issues, and safeguard against potential regressions. **We cannot merge changes that risk the backward incompatibility of existing documented behaviors.** We understand that creating thorough tests often requires significant effort, and your dedication to this process greatly contributes to the project's overall reliability. Thank you for your commitment to maintaining the integrity of our codebase and the experience of everyone using dbt! **PRs go through two review steps.** First, we aim to respond with feedback on whether we think the implementation is appropriate from a product & usability standpoint. At this point, we will close PRs that we believe fall outside the scope of dbt, or which might lead to an inconsistent user experience. Many of these are great ideas for add-on functionality that is better suited for a package or plugin. This is an important part of our role as maintainers; we're always open to hearing disagreement. If a PR passes this first review, we will queue it up for code review, at which point we aim to test it ourselves and provide thorough feedback. **We receive more PRs than we can thoroughly review, test, and merge.** Our teams have finite capacity, and our top priority is maintaining a well-scoped, high-quality framework for the tens of thousands of people who use it every week. To that end, we must prioritize overall stability and planned improvements over a long tail of niche potential features. For best results, say what in particular you'd like feedback on, and explain what would it mean to you, your team, and other community members to have the proposed change merged. Smaller PRs tackling well-scoped issues tend to be easier and faster for review. Two examples of community-contributed PRs: * [(dbt-labs/dbt#9347) Fix configuration of turning test warnings into failures](https://github.com/dbt-labs/dbt/pull/9347) * [(dbt-labs/dbt#9863) Better error message when trying to select a disabled model](https://github.com/dbt-labs/dbt/pull/9863) **Automation that can help us:** Many repositories have a template for pull request descriptions, which will include a checklist that must be completed before the PR can be merged. You don't have to do all of these things to get an initial PR, but they will delay our review process. Those include: * **Tests, tests, tests.** When you open a PR, some tests and code checks will run. (For security reasons, some may need to be approved by a maintainer.) We will not merge any PRs with failing tests. If you're not sure why a test is failing, please say so, and we'll do our best to get to the bottom of it together. * **Contributor License Agreement** (CLA): This ensures that we can merge your code, without worrying about unexpected implications for the copyright or license of open source or source-available dbt software. For more details, read: ["Contributor License Agreements"](https://docs.getdbt.com/community/resources/contributor-license-agreements.md) * **Changelog:** In projects that include a number of changes in each release, we need a reliable way to signal what's been included. The mechanism for this will vary by repository, so keep an eye out for notes about how to update the changelog. * **Semver label:** Some repositories require every pull request to carry exactly one `semver:patch` (bug fixes and other safe changes) or `semver:minor` (new, non-breaking features) label; `semver:major` isn't accepted. Where this applies, CI enforces it. ##### Inclusion in release versions ###### dbt v1 Both bug fixes and backwards-compatible new features will be included in the [next minor release of dbt v1](https://docs.getdbt.com/docs/dbt-versions.md#how-dbt-v1-uses-semantic-versioning). Fixes for regressions and net-new bugs that were present in the minor version's original release will be backported to versions with [active support](https://docs.getdbt.com/docs/dbt-versions.md). Other bug fixes may be backported when we have high confidence that they're narrowly scoped and won't cause unintended side effects. ###### dbt v2 dbt v2 is released continuously, and it contains a superset of the code and functionality available in the open source layer. When dbt v2.0.0 is released (General Availability), the version numbers for dbt OSS and dbt v2 will converge. The next major version of our [open source](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md) offering is built on the v2 runtime. Contributions to `dbt-core` targeting the v2.x series follow the same principles as v1.x: bug fixes and backwards-compatible features are included in the next available release. Because v2 is currently in beta, we will be releasing new versions rapidly as we get closer to the final v2.0.0 release (General Availability). --- ### Getting help #### Community help dbt is powered by open source and source-available software, and has a generous community behind it. Asking questions well contributes to the community by building our collective body of knowledge. By following these steps, you'll be more likely to receive help from another community member. ##### 1. Try to solve your problem first before asking for help ###### Search the existing documentation The docs site you're on is highly searchable, make sure to explore for the answer here as a first step. If you're new to dbt, try working through the [quickstart guide](https://docs.getdbt.com/guides.md) first to get a firm foundation on the essential concepts. ###### Try to debug the issue yourself We have a handy guide on [debugging errors](https://docs.getdbt.com/guides/debug-errors.md) to help out! This guide also helps explain why errors occur, and which docs you might need to search for help. ###### Search for answers using your favorite search engine We're committed to making more errors searchable, so it's worth checking if there's a solution already out there! Further, some errors related to installing dbt, the SQL in your models, or getting YAML right, are errors that are not-specific to dbt, so there may be other resources to check. ###### Experiment! If the question you have is "What happens when I do `X`", try doing `X` and see what happens! Assuming you have a solid dev environment set up, making mistakes in development won't affect your end users ##### 2. Take a few minutes to formulate your question well Explaining the problems you are facing clearly will help others help you. ###### Include relevant details in your question Include exactly what's going wrong! When asking your question, you should: * Paste the error message or relevant code inside three backticks in your question, instead of sharing a screenshot * Include the version of dbt you're on (which you can check with `dbt --version`) * Let us know which warehouse you're using ###### Avoid generalizing your code While we understand that you may wish to generalize your problem, or that you may have sensitive information you wish to anonymize, often replacing references in SQL can result in invalid code that creates an error different to the one you're hitting. This makes it harder for us to understand your problem. Wherever possible, share the exact code that you're trying to run. ###### Let us know what you've already tried In general, people are much more willing to help when they know you've already given something your best shot! ###### Share the context of the problem you're trying to solve Sometimes you might hit a boundary of dbt because you're trying to use it in a way that doesn't align with the opinions we've built into dbt. By sharing the context of the problem you're trying to solve, we might be able to share insight into whether there's an alternative way to think about it. ###### Post a single message and use threads The dbt Slack's culture revolves around threads. When posting a message, try drafting it to yourself first to make sure you have included all the context. Include big code blocks in a thread to avoid overwhelming the channel. ###### Don't tag individuals to demand help If someone feels inclined to answer your question, they will do so. We are a community of volunteers, and we're generally pretty responsive and helpful! If nobody has replied to your question, consider if you've asked a question that helps us understand your problem. If you require in-depth, ongoing assistance, we have a wonderful group of experienced dbt consultants in our ecosystem. You can find a full list [below](#receiving-dedicated-support). ##### 3. Choose the right medium for your question We use a number of different mediums to share information * If your question is roughly "I've hit this error and am stuck", please ask it on [the dbt Community Forum](https://discourse.getdbt.com). * If you think you've found a bug, please report it on the relevant GitHub repo (e.g. [dbt repo](https://github.com/dbt-labs/dbt), [dbt-utils repo](https://github.com/dbt-labs/dbt-utils)) * If you are looking for a more wide-ranging conversation (e.g. "What's the best approach to X?", "Why is Y done this way?"), join our [Slack community](https://getdbt.com/community). Channels are consistently named with prefixes to aid discoverability. #### Receiving dedicated support If you need dedicated support to build your dbt project, consider reaching out regarding [professional services](https://www.getdbt.com/contact/), or engaging one of our [consulting partners](https://www.getdbt.com/partner-directory). #### dbt Training If you want to receive dbt training, check out our [dbt Learn](https://learn.getdbt.com/) program. #### dbt support **Note:** If you are a **dbt user** and need help with one of the following issues, please reach out to us by clicking [**Create a support ticket**](https://docs.getdbt.com/docs/dbt-support.md#create-a-support-ticket) through the dbt navigation or [emailing support@getdbt.com](mailto:support@getdbt.com): * Account setup (e.g. connection issues, repo connections) * Billing * Bug reports related to the web interface As a rule of thumb, if you are using dbt, but your problem is related to code within your dbt project, then please follow the above process or checking out the [FAQs](https://docs.getdbt.com/docs/faqs.md) rather than reaching out to support. Refer to [dbt support](https://docs.getdbt.com/docs/dbt-support.md) for more information. --- ### How to deliver a fantastic meetup talk **Speaking at a dbt meetup? Here’s all the details you’ll need to know. If you’re speaking at another event, check out our additional tips at the end of the article.** #### Understanding dbt meetups dbt meetups are an opportunity for the dbt community to learn from each other. We’re typically on the lookout for talks that last for ~15 minutes, and we reserve an additional 5-10 minutes for Q\&A after your talk. We’re not *just* looking for talks that feature dbt — if your topic feels relevant to analytics engineers, we’d love to chat. In general, you can assume that around three quarters of the audience are dbt users. When shaping your talk, consider whether there’s something in there that might be new to an experienced dbt user, and, on the other end of the scale, something that feels relevant to a data practitioner that isn’t yet a dbt user. If you feel that your talk idea requires in-depth knowledge of dbt, consider speaking on Office Hours instead. Similarly, if you’re interested in giving a more introductory talk about dbt, consider reaching out to a local data meetup to see if it’s the right fit. For topic inspiration, you can find videos of past dbt meetup presentations [here](https://www.youtube.com/playlist?list=PL0QYlrC86xQn-jxWmEqtQRbZoyjq_ffq5). If you want to present at a dbt meetup, let us know [here](https://docs.google.com/forms/d/e/1FAIpQLScU4c0UvXLsasc7uwFBrzt6YzuGiMzEH_EyFfXGnIYDmTBDfQ/viewform). If we haven’t met you before, we might book-in a call to say hi and help shape your topic! We’ll also book a meeting before the event for a dry-run of the presentation to give any additional feedback. #### Recognize when you’re ready to give a talk Below, we’ve listed four signs that you’re ready to give a talk (originally based on [this article](https://thinkgrowth.org/how-to-write-about-your-work-652441747f41) from our Head of Marketing, Janessa — read that too!). We’ve also included examples for each category — where possible these are dbt meetup talks, but some of them are also links to blog posts from members in our community. ##### You recently finished a high-impact project These are a great option for first-time speakers as they mix together both big-picture thinking and tactics. For example: * "Improving data reliability" — Andrea Kopitz ([video](https://www.youtube.com/watch?v=M_cNspn2XsE), [slides](https://docs.google.com/presentation/d/1gHChax5aM3tqKkhepX7Mghmg0DTDbY5yoDBCfUR23lg/)) * "Predicting customer conversions using dbt + machine learning" — Kenny Ning ([video](https://www.youtube.com/watch?v=BF7HH8JDUS0), [slides](https://docs.google.com/presentation/d/1iqVjzxxRggMnRoI40ku88miDKw795djpKV_v4bbLpPE/)) * "Migrating 387 models from Redshift to Snowflake" — Sam Swift and Travis Dunlop ([video](https://www.youtube.com/watch?v=VhH614WVufM), [slides](https://docs.google.com/presentation/d/1wE8NSkFPLFKGQ8fvFUUKoZFVoUhws_FhFip-9mDhoPU/)) ##### You hit an inflection point in your career Have you recently changed something about your career that you think others can learn from? Started a new job, grown in your role? These topics might not mention dbt at all, but will be relevant to many people in the audience. For example: * “Getting hired as an analytics engineer: a candidate’s perspective” — Danielle Leong ([video](https://www.youtube.com/watch?v=6VCr30ZFxZ0)) * “One analyst's guide for going from good to great” — Jason Ganz ([blog post](https://blog.getdbt.com/one-analysts-guide-for-going-from-good-to-great/)) Other ideas: * You moved from a team of many to a team of one (or vice-versa), and want to share what each can learn from the other * You started to manage others and learned some things along the way ##### You’re digging deep into a topic If you’ve spent many hours going deep on a topic, it could be a good idea to share what you’ve learned. For example: * “The farm-to-table testing framework” — Andrea Fabry ([blog post](https://blog.getdbt.com/data-testing-framework/)) * “How to create a career ladder” — Caitlin Moorman ([blog post](https://locallyoptimistic.com/post/career-ladders-part-2/)) ##### You have a strong opinion about something Is there a “best practice” that you think is outdated? Want to convince others to see your point of view? In the data-space, we’ve seen this in topics like: * “Engineers shouldn’t write ETL” — Jeff Magnusson ([blog post](https://multithreaded.stitchfix.com/blog/2016/03/16/engineers-shouldnt-write-etl/)) * “You probably don’t need a data dictionary” — Michael Kaminsky & Alexander Jia ([blog post](https://locallyoptimistic.com/post/data_dictionaries/)) * “Write better SQL: In defense of `group by 1`” — Claire Carroll ([blog post](https://blog.getdbt.com/write-better-sql-a-defense-of-group-by-1/)) #### Checkpoint: Is someone else well-placed to give this talk? Once you have a topic idea, stop for a moment and consider whether someone else on your team might also be a great fit for delivering this talk. Individuals from underrepresented groups are far less likely to self-nominate to give a talk — sometimes a shoulder tap is the nudge that’s needed. #### Shaping your talk Now, it’s time to write! Rather than starting with a slide deck, open up a blank document (or use [our template](https://docs.google.com/document/d/16aog0VitdLSScgxSNKe36q1C92QmG2vjXmtXYcPAhfw/edit#)), and start writing some notes. This helps you clarify your thinking, and is a great way to get feedback early, rather than investing the time into creating slides that might later be reworked. Don’t get too hung up on a title at this stage — we’re happy to work with you on that later in the process. #### The basic structure Below, we’ve outlined a common structure used for meetup talks — if this is your first talk, this is a great way to get started (in fact, even experienced speakers often use a structure like this). Use this as a starting point, rather than an exact formula! 1. What is the business problem? Relating to a business problem helps audience members understand why you undertook a project. For example: * The finance team didn’t trust our numbers * We were never sure what led to an increase in customer conversion * The data team couldn’t find a balance between ad hoc requests and roadmap work * Our tracking across mobile and web was completely inconsistent 2. How did this manifest? Include evidence that this is a genuine problem — this helps create buy-in from the audience. Slack screenshots, quotes, charts, etc. are all good here! 3. What tactics were used to solve the problem? Three feels like a good number here. Make sure to emphasize people and process solutions as well as technology solutions. 4. What was the impact on the business problem? Since you set out a problem to be solved, it’s worth revisiting it. It’s okay if you found that your project didn’t go as planned — there’s a valuable lesson in there. Again, including evidence of improvement feels valuable. 5. What other things were learned, and/or what next steps are you taking? Summarize high level lessons that others can take-away, and potentially talk about what you’d do differently, or what you plan on doing next. ##### Why does this structure work? The previous structure might seem formulaic, but we’ve seen it work a number of times. In our opinion, this structure works because: * **Your presentation has the structure of a story** — problem, journey, solution. Human beings love stories, and so the flow feels natural and easy for your audience to follow. * **It increases the target audience**. Sharing a few different tactics means that it’s more likely there will be something in your talk for different audience members. Compare that to narrowly scoping a talk on “[Writing packages when a source table may or may not exist](https://discourse.getdbt.com/t/writing-packages-when-a-source-table-may-or-may-not-exist/1487)”— it’s not going to feel relevant to most people in the room. * **It covers both theory and application.** Too much theory and you’re giving a TedTalk, too much application and you’re just giving a product demo. The best Meetup talks help people understand how you thought through a problem and why you made certain decisions so they can apply your knowledge within their unique context. #### Examples that follow this structure Here's a few of our favorite talks mapped to the structure — trust us, it works! ##### Improving data reliability — Andrea Kopitz, Envoy *[Video](https://www.youtube.com/watch?v=M_cNspn2XsE), [slides](https://docs.google.com/presentation/d/1gHChax5aM3tqKkhepX7Mghmg0DTDbY5yoDBCfUR23lg/).* 1. What is the business problem? Envoy’s financial data appeared inconsistent. 2. How did this manifest? Respondents to the team’s data survey said they no longer trusted the data. 3. What tactics were used to solve the problem? 1. Determine responsibility 2. Build more specific dbt tests 3. Track progress 4. What was the impact on the business problem? In their next data survey, satisfaction rating increased, and no mention of financial data accuracy. 5. What other things were learned, and/or what next steps are you taking? Lesson: Send out a data survey to your company to inform your roadmap. ##### Predicting customer conversions with dbt + machine learning — Kenny Ning, Better.com *[Video](https://www.youtube.com/watch?v=BF7HH8JDUS0), [slides](https://docs.google.com/presentation/d/1iqVjzxxRggMnRoI40ku88miDKw795djpKV_v4bbLpPE/).* 1. What is the business problem? No one knew why conversion rates for better.com customers would improve or worsen, making it difficult to know the value of different parts of the business. 2. How did this manifest? Different parts of the business took responsibility when it improved, no one took responsibility when it worsened. 3. What tactics were used to solve the problem? 1. Use a different approach to conversion rates — kaplan-meier conversion rates 2. Sketch out an ideal ML solution and see if it theoretically solves the problem 3. Build it! (ft. demonstration of solution) 4. What was the impact on the business problem? In the end — not as valuable as originally hoped (and that’s ok!). Editor note: [this article](https://better.engineering/2020-06-24-wizard-part-ii/) was a great follow up on the initial project. 5. What other things were learned, and/or what next steps are you taking? * Focus on end-to-end solutions * Materialize your clean dataset to improve collaboration * Sell to the business ##### Migrating 387 models from Redshift to Snowflake — Bowery Farming Data Team *[Video](https://www.youtube.com/watch?v=VhH614WVufM), [slides](https://docs.google.com/presentation/d/1wE8NSkFPLFKGQ8fvFUUKoZFVoUhws_FhFip-9mDhoPU/).* 1. What is the business problem? A new Bowery Farming site had increased the amount of data the team were dealing with, which put a strain on their data stack. 2. How did this manifest? Charts show increased dbt times, and increased Redshift costs. 3. What tactics were used to solve the problem? 1. Push Redshift to its limit: Leverage Athena, Redshift configurations, separate clusters, python pre-processing 2. Trial Snowflake for cost and performance 3. Commit to a migration with strong project management 4. What was the impact on the business problem? Yet to be determined (at the time, they had just finished the project). But the team showed evidence that the project has been successfully completed! 5. What other things were learned, and/or what next steps are you taking? Other things learned: * Differences between Redshift and Snowflake SQL syntax * Teamwork and coordination are key to completing a migration #### Turn it into a presentation Now, it's time to take your idea and turn it into a presentation. ##### Structuring your slides As well as the slides that directly support your content, consider including: * At the start: * An intro slide for yourself (and teammates) * An intro slide for your company — you might also include some impressive numbers about your business, after all, your audience is full of people who love numbers! * Potentially include your tech stack for context — there’s no need to spend too much time on this, most audience members will be familiar with the tools. * Before diving into the specific tactics used: * Use a slide to list the three tactics at a high level — this signposting helps set expectations for audience members. * At the end: * A closing slide to prompt questions, and list your contact details. * If your company is hiring, mention that too! If available, use your corporate-branded slide deck. We also have dbt-branded slides if you want to use those. ##### Making your presentation shine When turning your story into a presentation, also consider doing the following: ###### Use full sentences in your slide headings When presenting (especially virtually), it’s hard to hold everyone’s focus. That’s ok! By including full sentences as your heading, people can “hook” back into the presentation. For example, rather than having a slide on "Slide headings", use a title like “Use full sentences in your slide headings” (woah — meta!) ###### Make your slides accessible This is a [great guide](https://www.smashingmagazine.com/2018/11/inclusive-design-accessible-presentations/) on making your slides accessible — read it! ###### Use evidence in your slides Evidence is a key part of getting buy-in that the story you’re telling is valuable. Consider including: * Screenshots of slack conversations * Quotes, survey results, charts * If talking about a complex transformation, include small samples of data to demonstrate the concept. You may need to generate some fake data to simplify the problem (example) * If one of your tactics is heavily code-based, consider sharing that code in a separate piece so that interested folks can refer back to it later. (Discourse is great for this) ###### (Virtual events) Create moments for interactivity For virtual events: is there a poll you can launch, or a question you can throw out to the chat? This can help create a sense of community at the event. #### Pair it with a blog post The hardest part of nailing a great talk is the content, so if you’ve made it this far, you’ve already done most of the work. Turning your content into a blog post is a great way to solidify your thinking, and get some extra exposure. If you’d like to be featured on the [dbt Blog](https://blog.getdbt.com/), please email us at . We’ll also be adding more resources on how to write about your work soon! #### Speaking at a non-dbt event Above, we’ve given specific advice for speaking at a dbt meetup. If you’re a dbt community member who wants to speak at a non-dbt meetup or conference, there’s a few extra ways you can adjust your process. ##### Questions to ask the event organizer ###### What is the technical baseline for the audience? Do they know about dbt? If not, are they familiar with SQL? You’ll likely have a range of people in the audience so there won’t be one exact answer, but gathering information about the median knowledge is useful. As a guideline, aim to teach something new to at least half of the audience. ###### What kind of talks have been the most successful? Is the event oriented around technical talks or strategic talks? Is there an expectation of demo-ing code? Do they have past examples of talks that were well-received, or any tips? ###### What are the event logistics? How long is your talk supposed to go for? Is there an opportunity to do Q\&A? If the event is virtual, what is the software setup like? How will questions be moderated? If the event is in-person, will you be able to use your own computer, or will you use someone else’s? What sort of screen is there? How do you connect to it? And do you have the right dongle for your MacBook Pro? ###### Is there an opportunity for topic feedback? Is the organizer interested in working with you to make your topic great? If not, can they point you to someone in their community who might be interested in helping out? ###### Are there any additional accessibility considerations you should be aware of? Do any audience members use a communication device? Can you share your slides ahead of time to make them easier for audience members to access? Will the event be recorded for those who can’t attend in person? ##### Responding to a conference Call for Speakers If you’re submitting a response for a Call for Speakers, and talking about dbt, we’re happy to work with you on this. You may email us at for more information. --- ### Join the Community Want to learn how organizations around the world are tackling the biggest challenges in data while making new friends from the best analytics teams? Join the dbt Community — data practitioners’ favorite place to learn new skills, keep on top of industry trends, and forge connections. [![](/img/icons/slack.svg)](https://www.getdbt.com/community/join-the-community/) ###### [Join us on Slack](https://www.getdbt.com/community/join-the-community/) [Follow the pulse of the dbt Community! Chat with other practitioners in your city, country or worldwide about data work, tech stacks, or simply share a killer meme.](https://www.getdbt.com/community/join-the-community/) [![](/img/icons/discussions.svg)](https://discourse.getdbt.com/) ###### [Community Forum](https://discourse.getdbt.com/) [Have a question about how to do something in dbt? Hop into the Community Forum and work with others to create long lived community knowledge.](https://discourse.getdbt.com/) [![](/img/icons/pencil-paper.svg)](https://docs.getdbt.com/community/contribute.md) ###### [How to contribute](https://docs.getdbt.com/community/contribute.md) [Want to get involved? This is the place! Learn how to contribute to our public repositories, write for the blog, speak at a meetup and more.](https://docs.getdbt.com/community/contribute.md) [![](/img/icons/folder.svg)](https://docs.getdbt.com/community/resources/code-of-conduct.md) ###### [Code of Conduct](https://docs.getdbt.com/community/resources/code-of-conduct.md) [We are committed to creating a space where everyone can feel welcome and safe. Our Code of Conduct reflects the agreement that all Community members make to uphold these ideals.](https://docs.getdbt.com/community/resources/code-of-conduct.md) [![](/img/icons/calendar.svg)](https://www.getdbt.com/resources/webinars) ###### [Webinars](https://www.getdbt.com/resources/webinars) [Browse upcoming and on-demand dbt Labs webinars and workshops.](https://www.getdbt.com/resources/webinars) [![](/img/icons/globe.svg)](https://www.getdbt.com/events) ###### [Events](https://www.getdbt.com/events) [Find meetups, conferences, training, and other dbt community events online and in person.](https://www.getdbt.com/events) [![](/img/icons/star.svg)](https://www.youtube.com/playlist?list=PL0QYlrC86xQl1DGKBopQZiZ6tSqrMlD2M) ###### [Watch past events](https://www.youtube.com/playlist?list=PL0QYlrC86xQl1DGKBopQZiZ6tSqrMlD2M) [Get a taste for the energy of our live events, get inspired, or prepare for an upcoming event by watching recordings from our YouTube archives.](https://www.youtube.com/playlist?list=PL0QYlrC86xQl1DGKBopQZiZ6tSqrMlD2M) --- ### Maintaining a Slack channel #### TL;DR There are three things you should do to be a good channel maintainer in the [dbt Slack community](https://community.getdbt.com/): * Once you see some folks in the channel, post initial conversation topics to help them get to know each other. * Keep an eye out in #introductions for folks who might benefit from your new channel. For example, if someone works in the space or on the same problems, then direct them to the channel. * Make sure folks follow the [Rules of the Road](https://docs.getdbt.com/community/resources/community-rules-of-the-road.md) in the channel. If you notice someone is not following one, gently remind them of the rule in thread, and, ideally, provide an example of how they can rephrase their message or where they can redirect it. If you have a question about how to proceed, just post it in #moderation-and-administration with a screenshot or link to the thread and someone will give you advice. #### Scope of the role A maintainer can be a dbt Labs employee but does not have to be. *Slack channel maintainer* is philosophically similar to OSS maintainer. At the onset, the channel maintainer will help build up this new space in Slack and stir up conversation during the first few weeks of the channel's existence. They are someone who stays on top of feedback and encourages generative contributions. This is not necessarily someone who is the generator of content and contributions, or answers every question. #### Initial instructions 1. Review the [Rules of the Road](https://docs.getdbt.com/community/resources/community-rules-of-the-road.md) and [Code of Conduct](https://docs.getdbt.com/community/resources/code-of-conduct.md) and please let the folks who created the channel know that you read both documents and you agree to be mindful of them. 2. To request a new channel, go to the [#moderation-and-administration channel](https://getdbt.slack.com/archives/C02JJ8N822H), click **Workflows** on the top of the channel description. Click **Request a New Channel**. * Fill out the fields and click **Submit** to submit your request. Someone will get in touch from there. ![request-slack-chnl](https://github.com/siljamardla/docs.getdbt.com/assets/89008547/b14abc52-4164-40a8-b48a-e8061fb4b51a) 3. If you are a vendor, review the [Vendor Expectations](https://docs.getdbt.com/community/resources/community-rules-of-the-road.md#vendor-expectations). 4. Add the Topic and Description to the channel. @Mention your name in the channel Description, identifying yourself as the maintainer. Ex: *Maintainer: First Last (pronouns).* If you are a vendor, make sure your Handle contains your affiliation. 5. Complete or update your Slack profile by making sure your Company (in the ‘What I do’ field), Pronouns, and Handle, if you’re a vendor, are up-to-date. 6. Post initial conversation topics once a few folks get in the channel to help folks get to know each other. Check out this [example introductory post](https://getdbt.slack.com/archives/C02FXAZRRDW/p1632407767005000). 7. Stir up conversation during the first few weeks of the channel's existence. As you get started, answer the questions you can or help find someone with answers, seed discussions about once a week, and making sure folks follow the Rules of the Road. #### Long-term expectations * Maintaining the channel, checking in, and being active on a regular basis by answering folks' questions, and seeding discussions. Want an example? Check out [this poll](https://getdbt.slack.com/archives/C022A67TLFL/p1628279819038800). * For guidance on how to answer a question, see [Answering Community Questions](https://www.getdbt.com/community/answering-community-questions). If you are not sure how to answer a lingering or unanswered question, you can post about it in #moderation-and-administration or direct it to another channel, if relevant. * If the channel is an industry channel, it’s helpful to monitor [#introductions](https://getdbt.slack.com/archives/CETJLH1V3) and invite people. Keep an eye out for folks who might benefit from being in the new channel if they mention they are working in the space, or are thinking about some of these problems. * Make sure folks follow the [Rules of the Road](https://docs.getdbt.com/community/resources/community-rules-of-the-road.md). For example, if you notice someone is not following one, gently remind them of the rule in thread, and, ideally, provide an example of how they can rephrase their message or where they can redirect it. If you have a question about how to proceed, just post about it in #moderation-and-administration with a link to the thread or screenshot and someone will give you advice. * In tools channels, sharing customer stories and product updates is very okay in this channel because folks expect that when they join. However, please avoid any direct sales campaigns, pricing offers, etc. * If you have any questions/doubts about the [Rules of the Road and Vendor Expectations](https://docs.getdbt.com/community/resources/community-rules-of-the-road.md), please post a question in #moderation-and-administration about what sort of things the community expects from interactions with vendors. * A reminder that we never DM anyone in Slack without their permission in a public channel or some prior relationship. * A reminder that @ here/all/channel are disabled. * Use and encourage the use of threads 🧵 to keep conversations tidy! --- ### Online community building ##### Maintaining a channel in the dbt Community Slack ###### Overview The dbt Slack is the place for real time conversations with the dbt Community. Slack channels exist for specific locations, tools, industries and methodologies. In order to make sure that every channel has dedicated attention from a committed community member, we have Community Maintainers who oversee the discussion in particular channels. ###### Contribution opportunities Every channel can benefit from people who are engaged and committed to making it a more interesting place to hang out! If there's a channel that you're interested in, spend time there. For new channels that you'd like to create and maintain, post a message in the #channel-requests channel. ###### Sample contributions: * Karen Hsieh's [contributions](https://getdbt.slack.com/archives/C02TU2DSKND/p1661483529756289) to the #local-taipei channel are a fantastic example to learn from. ###### Get started * Read the guide to [Maintaining a Slack Channel](https://docs.getdbt.com/community/resources/maintaining-a-channel.md) ##### Participating on the Community Forum ###### Overview [The dbt Community Forum](https://discourse.getdbt.com) is the preferred platform for support questions as well as a space for long-lived discussions about dbt, analytics engineering, and the analytics profession. It's a place for us to build up a long-lasting knowledge base around the common challenges, opportunities, and patterns we work with every day. ###### Contribution opportunities Participate in the Forum by asking and answering questions. These discussions are what allows us to find gaps in our best practices, documentation and other recommendation, as well as to get folks onboarded and understanding dbt. Remember, it’s a mitzvah to answer a question. If you see a great question or answer, be generous with your 💜 reactions. Click the Solved button when your question is answered, so others can benefit. ###### Sample contributions * An analytics engineer wrote about [how they modified dbt to automatically put models into the correct schema](https://discourse.getdbt.com/t/extracting-schema-and-model-names-from-the-filename/575) based on their filename. * Here's [an example of the supportive, thorough answers](https://discourse.getdbt.com/t/is-it-possible-to-have-multiple-files-with-the-same-name-in-dbt/647) you can receive when you take the time to ask a question well. ###### Get started * Read the [Community Forum Guidelines](https://docs.getdbt.com/community/resources/forum-guidelines.md) to understand what topics are a good fit and why this space is important in building long-term community knowledge. * Head over to the “[Help](https://discourse.getdbt.com/c/help/19)” section of the forum and look for areas to hop in! You don’t need to know the exact answer to a question to be able to provide a helpful pointer. --- ### Open source and source-available projects Looking for a good place to get involved contributing code? dbt Labs supports the following repositories, organized by the language primarily needed for contribution: #### Rust * [dbt OSS](https://github.com/dbt-labs/dbt) - the Apache 2.0 open-source distribution of dbt v2 (previously in [`dbt-fusion`](https://github.com/dbt-labs/dbt-fusion)) #### Python * [dbt v1](https://github.com/dbt-labs/dbt/tree/1.latest) - the original engine powering dbt * [hubcap](https://github.com/dbt-labs/hubcap) - the code powering the dbt Package hub #### dbt * [dbt Labs' packages](https://hub.getdbt.com/dbt-labs/) - the dbt packages created and supported by dbt Labs. Packages are just dbt projects, so if you know the SQL, Jinja, and YAML necessary to work in dbt, you can contribute to packages. #### YAML and JSON Config * [dbt-jsonschema](https://github.com/dbt-labs/dbt-jsonschema) - powering completion and linting for YAML configuration in dbt projects. #### Shell * [dbt-completion.bash](https://github.com/dbt-labs/dbt-completion.bash) - provides shell completion of CLI commands and selectors such as models and tests for bash and zsh. --- ### Realtime event participation ##### Speak at a Meetup ###### Overview Meetups are all about knowledge sharing; they are a place to connect and learn with your fellow dbt Community members. They usually take place in person, with some happening virtually. The Meetups take place across the globe, and you can check them all out [here](https://www.meetup.com/pro/dbt/). ###### Contribution opportunities * Become a Meetup organizer * Speak at an event * Sponsor an event or provide your office space For all of these opportunities, please fill out an [interest form](https://docs.google.com/forms/d/e/1FAIpQLScdzuz9Ouo1b07BMHveEBJsJ3rJAYuFvbTKep2fXDL0iZTZUg/viewform) and we will get back to you. ###### Sample contributions * Take a look at [the slides](https://docs.google.com/presentation/d/1iqVjzxxRggMnRoI40ku88miDKw795djpKV_v4bbLpPE/edit#slide=id.g553a984de0_0_19) and [watch the video](https://www.youtube.com/watch?v=BF7HH8JDUS0) from Kenny Ning's 2020 Meetup talk on predicting customer conversions with dbt and ML for Better.com. * Dig into [the deck](https://docs.google.com/presentation/d/1wE8NSkFPLFKGQ8fvFUUKoZFVoUhws_FhFip-9mDhoPU/edit#slide=id.p) and [the video](https://www.youtube.com/watch?v=VhH614WVufM) from Bowery Farmings talk on migrating dbt models from Redshift to Snowflake. ###### Get Started * Read [How to Deliver a Fantastic Meetup Talk](https://docs.getdbt.com/community/resources/speaking-at-a-meetup.md). * Find a [Meetup near you](https://www.meetup.com/pro/dbt/), start attending and let the organizers know you are interested! ß ##### Speak at dbt Summit ###### Overview [dbt Summit](https://www.getdbt.com/dbt-summit) is the annual analytics engineering conference hosted by dbt Labs. While Meetups are focused on sharing knowledge with a specific local hub of the Community, dbt Summit is the way to share ideas with everyone. Each year we gather together, take stock of what we’ve learned and pool our best ideas about analytics. ###### Contribution opportunities * Attend dbt Summit: * dbt Summit is the once a year gathering for analytics engineers. Whether you choose to join online or at one of our in-person events, attending dbt Summit is the best way to get an immersive experience of what the dbt Community is like. * Speak at dbt Summit! * We’d love to hear what you’ve been working on, thinking about and dreaming up in the analytics engineering space. dbt Summit talks can be forward looking views on the industry, deep dives into particular technical solutions or personal stories about your journey in data. ###### Sample contributions * [Run Your Data Team as a Product Team](https://www.getdbt.com/coalesce-2020/run-your-data-team-as-a-product-team/) * [Tailoring dbt's incremental\_strategy to Artsy's data needs](https://www.getdbt.com/coalesce-2021/tailoring-dbts-incremental-strategy-to-artsys-data-needs/) ###### Get started * If registrations are open register on the [dbt Summit website](https://www.getdbt.com/dbt-summit) * Join #dbt-summit-updates on the dbt Community Slack --- ### The dbt Viewpoint Building a Mature Analytics Workflow: The dbt Viewpoint! In 2015-2016, a team of folks at RJMetrics had the opportunity to observe, and participate in, a significant evolution of the analytics ecosystem. The seeds of dbt were conceived in this environment, and the viewpoint below was written to reflect what we had learned and how we believed the world should be different. **dbt is our attempt to address the workflow challenges we observed, and as such, this viewpoint is the most foundational statement of the dbt project's goals.** The remainder of this document is largely unedited from [the original post](https://getdbt.com/blog/building-a-mature-analytics-workflow). #### Analytics today The center of gravity in mature analytics organizations has shifted away from proprietary, end-to-end tools towards more composable solutions made up of: * data integration scripts and/or tools, * high-performance analytic databases, * SQL, R, and/or Python, and * visualization tools. This change has unlocked significant possibility, but analytics teams (ours included) have still faced challenges in consistently delivering high-quality, low-latency analytics. We believe that analytics teams have a workflow problem. Too often, analysts operate in isolation, and this creates suboptimal outcomes. Knowledge is siloed. We too often rewrite analyses that a colleague had already written. We fail to grasp the nuances of datasets that we’re less familiar with. We differ in our calculations of a shared metric. We have convinced ourselves after hundreds of conversations that these conditions are by and large the status quo for even sophisticated analytics teams. As a result, organizations suffer from reduced decision speed and reduced decision quality. Analytics doesn’t have to be this way. In fact, the playbook for solving these problems already exists — on our software engineering teams. The same techniques that software engineering teams use to collaborate on the rapid creation of quality applications can apply to analytics. We believe it’s time to build an open set of tools and processes to make that happen. #### Analytics is collaborative We believe a mature analytics team’s techniques and workflow should have the following collaboration features: ##### Version Control Analytic code — whether it’s Python, SQL, Java, or anything else — should be version controlled. Analysis changes as data and businesses evolve, and it’s important to know who changed what, when. ##### Quality Assurance Bad data can lead to bad analyses, and bad analyses can lead to bad decisions. Any code that generates data or analysis should be reviewed and tested. ##### Documentation Your analysis is a software application, and, like every other software application, people are going to have questions about how to use it. Even though it might seem simple, in reality the “Revenue” line you’re showing could mean dozens of things. Your code should come packaged with a basic description of how it should be interpreted, and your team should be able to add to that documentation as additional questions arise. ##### Modularity If you build a series of analyses about your company’s revenue, and your colleague does as well, you should use the same input data. Copy-paste is not a good approach here — if the definition of the underlying set changes, it will need to be updated everywhere it was used. Instead, think of the schema of a data set as its public interface. Create tables, views, or other data sets that expose a consistent schema and can be modified if business logic changes. #### Analytic code is an asset The code, processes, and tooling required to produce that analysis are core organizational investments. We believe a mature analytics organization’s workflow should have the following characteristics so as to protect and grow that investment: ##### Environments Analytics requires multiple environments. Analysts need the freedom to work without impacting users, while users need service level guarantees so that they can trust the data they rely on to do their jobs. ##### Service level guarantees Analytics teams should stand behind the accuracy of all analysis that has been promoted to production. Errors should be treated with the same level of urgency as bugs in a production product. Any code being retired from production should go through a deprecation process. ##### Design for maintainability Most of the cost involved in software development is in the maintenance phase. Because of this, software engineers write code with an eye towards maintainability. Analytic code, however, is often fragile. Changes in underlying data break most analytic code in ways that are hard to predict and to fix. Analytic code should be written with an eye towards maintainability. Future changes to the schema and data should be anticipated and code should be written to minimize the corresponding impact. #### Analytics workflows require automated tools Frequently, much of an analytic workflow is manual. Piping data from source to destination, from stage to stage, can eat up a majority of an analyst’s time. Software engineers build extensive tooling to support the manual portions of their jobs. In order to implement the analytics workflows we are suggesting, similar tools will be required. Here’s one example of an automated workflow: * models and analysis are downloaded from multiple source control repositories, * code is configured for the given environment, * code is tested, and * code is deployed. Workflows like this should be built to execute with a single command. --- ### Tips for organizing inclusive events The dbt community is filled with dedicated community leaders who create opportunities for connection, learning and professional development within the analytics community. This guide is a resource to help organizers execute **inclusive digital events**. We understand that organizers, presenters, speakers, etc. might not be able to apply these tips to *every* event, but this guide will offer some food for thought. Additionally, this list can grow. If you would like to contribute a tip, please email . #### General logistics * Try to choose a date that does not overlap with [holidays](http://www.holidayscalendar.com/months/) or general major events. Don’t forget to check international holidays (if applicable) * Avoid really large national/local events (i.e. World Cup) #### Marketing * If you are using photos, share images that include community members with a wide range of presentations, including people from underrepresented groups. * Put event accessibility information on your event page (i.e. “closed captioning available for all video resources”) * In the registration process provide an opportunity for attendees to: * share pronouns * ask questions in advance * request specific needs or other accommodations (interpreting services, braille transcription, dietary restrictions, etc.) * If this is a paid event (e.g. a conference), create a scholarship for attendees that might need financial support * Think about how you are promoting your event — are you reaching underrepresented communities, marginalized populations and people who might not have access to the internet? #### Programming * Book diverse speakers. Include speakers that represent underrepresented and marginalized populations. * Do research on your speakers. Is there any reason that your speakers would make the audience uncomfortable? * Design an [accessible presentation](https://www.smashingmagazine.com/2018/11/inclusive-design-accessible-presentations/) * If possible, share a recording after the event for community members who are not able to make it and add closed captioning. * Ask speakers to introduce themselves before starting their presentation, so that transcription services can capture who is talking. #### Digital platforms for online events * Take a minute or two to explain the features of the platform that attendees will be using in the beginning of the event * Offer the option for attendees to dial-in by phone and participate without a computer or internet * Explore the accessibility features your platform offers and apply it where necessary (i.e. closed captioning, automatic transcripts, screen reader support, etc.) * Check if your platform is compatible with assistive technology #### Attendee communication * Make sure that attendees have any addresses, links, codes, numbers to accessing the event beforehand * Share the agenda of the event beforehand so that attendees are able to make arrangements (if necessary) * Share contact information with attendees so that they are able to reach out with questions before and after the event. * Ask attendees for feedback in a post-event survey so that you are able to improve future experiences. #### Speaker communication * Ask speakers how to pronounce their names before the event * Ask speakers for their pronouns before the event * Suggest that speakers use headphones to ensure clear audio * Ask speakers to use plain language and avoid jargon, slang, idioms, etc. --- ### Writing contributions ##### Contribute to the product documentation ###### Overview The [dbt Product Documentation](https://docs.getdbt.com/docs/introduction.md) sits at the heart of how people learn to use and engage with dbt. From explaining dbt to newcomers to providing references for advanced functionality and APIs, the product docs are a frequent resource for *every* dbt Developer. ###### Contribution opportunities We strive to create pathways that inspire you to learn more about dbt and enable you to continuously improve the way you solve data problems. We always appreciate the vigilance of the Community helping us to accurately represent the functionality and capabilities of dbt. You can participate by: * [Opening an issue](https://github.com/dbt-labs/docs.getdbt.com/issues/new/choose) when you see something that can be fixed, whether it’s large or small. * Creating a PR when you see something you want to fix, or to address an existing issue. You can do this by clicking **Edit this page** at the bottom of any page on [docs.getdbt.com](http://docs.getdbt.com). ###### Sample contributions We appreciate these contributions because they contain context in the original post (OP) that helps us understand their relevance. They also add value to the docs, even in small ways! * Larger contribution: * Smaller contribution: ###### Get started * You can contribute to [docs.getdbt.com](http://docs.getdbt.com) by looking at our repository’s [README](https://github.com/dbt-labs/docs.getdbt.com#readme) or clicking **Edit this page** at the bottom of most pages at docs.getdbt.com. * Read the [Contributor Expectations](https://docs.getdbt.com/community/resources/contributor-expectations.md). * Find an issue labeled “[good first issue](https://github.com/dbt-labs/docs.getdbt.com/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22+).” * Need help: Visit #community-writers on the Community Slack or mention `@dbt-labs/product-docs` in a pull request or issue comment. ##### Write a Developer Blog Post ###### Overview The [dbt Developer Blog](https://docs.getdbt.com/blog) is the place for analytics practitioners to talk about *what it’s like to do data work right now.* This is the place to share tips and tricks, hard won knowledge and stories from the trenches with the dbt Community. ###### Contribution opportunities We want to hear your stories! Did you recently solve a cool problem, discover an interesting bug or lead an organizational change? Come tell the story on the dbt Developer Blog. ###### Sample contributions * [Founding an Analytics Engineering Team From Scratch](https://docs.getdbt.com/blog/founding-an-analytics-engineering-team-smartsheet#our-own-take-on-data-mesh) * [Tackling the Complexity of Joining Snapshots](https://docs.getdbt.com/blog/joining-snapshot-complexity) ###### Get started * [Read the contribution guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/developer-blog.md) * [Open up an issue with your idea for a post](https://github.com/dbt-labs/docs.getdbt.com/issues/new?assignees=\&labels=content%2Cdeveloper+blog\&template=contribute-to-developer-blog.yml) ###### Need help? Visit #community-writers in the dbt Community Slack --- ## Docs ### _wizard-cli-full-generated #### How to read this reference * **Flag** — the full flag as typed. Short flags (e.g. `-m`) are listed in the Short column. * **Type** — `boolean` (presence/absence), `string`, `path`, or an enumeration of allowed values. * **Description** — what the flag does and when to use it. * Global flags apply to the base interactive `wizard` command. Subcommands can have different flag sets; use each command's section for automation. See it in action and share your feedback Want to see dbt Wizard in action? Check out the [demo video](https://www.youtube.com/watch?v=-lIzh1xQWMA). We'd love to hear how dbt Wizard is working for you. Share your feedback by either running the `/feedback` slash command in your interactive terminal session or by going to the [#dbt-wizard](https://getdbt.slack.com/archives/C0B6KLW6T26) channel in the [dbt Community Slack](https://docs.getdbt.com/community/join?version=2.0). Thanks so much for your help in improving dbt Wizard and dbt data development! #### Global flags These flags work on the base interactive `wizard` command. | Flag | Short | Type | Description | | -------------------------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config ` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable ` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable ` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | | `--remote ` | — | string | Connect the TUI to a remote app server endpoint. | | `--remote-auth-token-env ` | — | string | Name of the environment variable containing the bearer token to send to a remote app server websocket | | `--strict-config` | — | boolean | Error out when config.toml contains fields that are not recognized by this version of wizard | | `-i, --image ...` | `-i` | path | Optional image(s) to attach to the initial prompt | | `-m, --model ` | `-m` | enum | Model the agent should use | | `--oss` | — | boolean | Use open-source provider | | `--local-provider ` | — | enum | Specify which local provider to use (lmstudio or ollama). If not specified with --oss, will use config default or show selection | | `-p, --profile ` | `-p` | path | Configuration profile from config.toml to specify default options | | `--profile-v2 ` | — | path | Layer $DBT\_WIZARD\_HOME/\.config.toml on top of the base user config | | `-s, --sandbox ` | `-s` | enum | Select the sandbox policy to use when executing model-generated shell commands | | `--dangerously-bypass-approvals-and-sandbox` | — | boolean | Skip all confirmation prompts and execute commands without sandboxing. EXTREMELY DANGEROUS. Intended solely for running in environments that are externally sandboxed | | `--dangerously-bypass-hook-trust` | — | boolean | Run enabled hooks without requiring persisted hook trust for this invocation. DANGEROUS. Intended only for automation that already vets hook sources | | `-C, --cd ` | `-C` | path | Tell the agent to use the specified directory as its working root | | `--add-dir ` | — | path | Additional directories that should be writable alongside the primary workspace | | `-a, --ask-for-approval ` | `-a` | enum | Configure when the model requires human approval before executing a command | | `--search` | — | boolean | Enable live web search. When enabled, the native Responses `web_search` tool is available to the model (no per‑call approval) | | `--no-alt-screen` | — | boolean | Disable alternate screen mode | #### Commands | Command | Aliases | Description | | ----------------------------------- | ------- | ------------------------------------------------------------------------------------------------- | | [`exec`](#exec) | `e` | Run wizard non-interactively | | [`review`](#review) | — | Run a code review non-interactively | | [`login`](#login) | — | Manage login | | [`logout`](#logout) | — | Remove stored authentication credentials | | [`mcp`](#mcp) | — | Manage external MCP servers for wizard | | [`plugin`](#plugin) | — | Manage wizard plugins | | [`providers`](#providers) | — | Manage model providers | | [`mcp-server`](#mcp-server) | — | Start wizard as an MCP server (stdio) | | [`app-server`](#app-server) | — | \[experimental] Run the app server or related tooling | | [`remote-control`](#remote-control) | — | \[experimental] Manage the app-server daemon with remote control enabled | | [`completion`](#completion) | — | Generate shell completion scripts | | [`update`](#update) | — | Update wizard to the latest version | | [`doctor`](#doctor) | — | Diagnose local wizard installation, config, auth, and runtime health | | [`sandbox`](#sandbox) | — | Run commands within a wizard-provided sandbox | | [`debug`](#debug) | — | Debugging tools | | [`apply`](#apply) | `a` | Apply the latest diff produced by wizard agent as a `git apply` to your local working tree | | [`resume`](#resume) | — | Resume a previous interactive session (picker by default; use --last to continue the most recent) | | [`fork`](#fork) | — | Fork a previous interactive session (picker by default; use --last to fork the most recent) | | [`cloud`](#cloud) | — | \[EXPERIMENTAL] Browse tasks from wizard in dbt platform and apply changes locally | | [`exec-server`](#exec-server) | — | \[EXPERIMENTAL] Run the standalone exec-server service | | [`features`](#features) | — | Inspect feature flags | #### exec Run wizard non-interactively ```bash wizard exec [OPTIONS] [ARGS] ``` **Arguments:** | Argument | Description | | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `[PROMPT]` | Initial instructions for the agent. If not provided as an argument (or if `-` is used), instructions are read from stdin. If stdin is piped and a prompt is also provided, stdin is appended as a `<stdin>` block | | Flag | Short | Type | Description | | -------------------------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config ` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable ` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable ` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | | `--strict-config` | — | boolean | Error out when config.toml contains fields that are not recognized by this version of wizard | | `-i, --image ...` | `-i` | path | Optional image(s) to attach to the initial prompt | | `-m, --model ` | `-m` | enum | Model the agent should use | | `--oss` | — | boolean | Use open-source provider | | `--local-provider ` | — | enum | Specify which local provider to use (lmstudio or ollama). If not specified with --oss, will use config default or show selection | | `-p, --profile ` | `-p` | path | Configuration profile from config.toml to specify default options | | `--profile-v2 ` | — | path | Layer $DBT\_WIZARD\_HOME/\.config.toml on top of the base user config | | `-s, --sandbox ` | `-s` | enum | Select the sandbox policy to use when executing model-generated shell commands | | `--dangerously-bypass-approvals-and-sandbox` | — | boolean | Skip all confirmation prompts and execute commands without sandboxing. EXTREMELY DANGEROUS. Intended solely for running in environments that are externally sandboxed | | `--dangerously-bypass-hook-trust` | — | boolean | Run enabled hooks without requiring persisted hook trust for this invocation. DANGEROUS. Intended only for automation that already vets hook sources | | `-C, --cd ` | `-C` | path | Tell the agent to use the specified directory as its working root | | `--add-dir ` | — | path | Additional directories that should be writable alongside the primary workspace | | `--skip-git-repo-check` | — | boolean | Allow running wizard outside a Git repository | | `--ephemeral` | — | boolean | Run without persisting session files to disk | | `--ignore-user-config` | — | boolean | Do not load `$DBT_WIZARD_HOME/config.toml`; auth still uses `DBT_WIZARD_HOME` | | `--ignore-rules` | — | boolean | Do not load user or project execpolicy `.rules` files | | `--output-schema ` | — | path | Path to a JSON Schema file describing the model's final response shape | | `--color ` | — | enum | Specifies color settings for use in the output | | `--json` | — | boolean | Print events to stdout as JSONL | | `--include-subagent-events` | — | boolean | When `--json` is set, capture and emit sub-agent events tagged with their thread ID. Each sub-agent event gets a `sub_agent_thread_id` field added to the JSONL line so consumers can distinguish it from the primary agent's events | | `--no-validation` | — | boolean | When set, the validation gate always responds with "declined" so the validation sub-agent is never spawned. Useful for benchmarking runs where sub-agent overhead is not desired | | `-o, --output-last-message ` | `-o` | path | Specifies file where the last message from the agent should be written | #### review Run a code review non-interactively **Arguments:** | Argument | Description | | ---------- | ----------------------------------------------------------- | | `[PROMPT]` | Custom review instructions. If `-` is used, read from stdin | | Flag | Short | Type | Description | | -------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config ` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--strict-config` | — | boolean | Error out when config.toml contains fields that are not recognized by this version of wizard | | `--enable ` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--uncommitted` | — | boolean | Review staged, unstaged, and untracked changes | | `--base ` | — | string | Review changes against the given base branch | | `--disable ` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | | `--commit ` | — | string | Review the changes introduced by a commit | | `--title ` | — | string | Optional commit title to display in the review summary | #### login Manage login | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### logout Remove stored authentication credentials | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### mcp Manage external MCP servers for dbt Wizard | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### plugin Manage dbt Wizard plugins | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### providers Manage model providers | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### mcp-server Start dbt Wizard as an MCP server (stdio) | Flag | Short | Type | Description | | -------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--strict-config` | — | boolean | Error out when config.toml contains fields that are not recognized by this version of wizard | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### app-server \[experimental] Run the app server or related tooling | Flag | Short | Type | Description | | --------------------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | | `--strict-config` | — | boolean | Error out when config.toml contains fields that are not recognized by this version of wizard | | `--listen <URL>` | — | string | Transport endpoint URL. Supported values: `stdio://` (default), `unix://`, `unix://PATH`, `ws://IP:PORT`, `off` | | `--analytics-default-enabled` | — | boolean | Controls whether analytics are enabled by default. | | `--ws-auth <MODE>` | — | enum | Websocket auth mode for non-loopback listeners | | `--ws-token-file <PATH>` | — | path | Absolute path to the capability-token file | | `--ws-token-sha256 <HEX>` | — | string | Hex-encoded SHA-256 digest of the capability token | | `--ws-shared-secret-file <PATH>` | — | path | Absolute path to the shared secret file for signed JWT bearer tokens | | `--ws-issuer <ISSUER>` | — | string | Expected issuer for signed JWT bearer tokens | | `--ws-audience <AUDIENCE>` | — | string | Expected audience for signed JWT bearer tokens | | `--ws-max-clock-skew-seconds <SECONDS>` | — | string | Maximum clock skew when validating signed JWT bearer tokens | #### remote-control \[experimental] Manage the app-server daemon with remote control enabled | Flag | Short | Type | Description | | -------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--json` | — | boolean | Emit machine-readable JSON | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### completion Generate shell completion scripts **Arguments:** | Argument | Description | | --------- | --------------------------------- | | `[SHELL]` | Shell to generate completions for | | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### update Update dbt Wizard to the latest version | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### doctor Diagnose local dbt Wizard installation, config, auth, and runtime health | Flag | Short | Type | Description | | -------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--json` | — | boolean | Emit a redacted machine-readable report | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--summary` | — | boolean | Only show grouped check rows and the final count summary | | `--all` | — | boolean | Expand long lists in detailed human output | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | | `--no-color` | — | boolean | Disable ANSI color in human output | | `--ascii` | — | boolean | Use ASCII status labels and separators in human output | #### sandbox Run commands within a dbt Wizard-provided sandbox | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### debug Debugging tools | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### apply Apply the latest diff produced by dbt Wizard agent as a `git apply` to your local working tree **Arguments:** | Argument | Description | | ----------- | ----------- | | `<TASK_ID>` | | | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### resume Resume a previous interactive session (picker by default; use --last to continue the most recent) **Arguments:** | Argument | Description | | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | `[SESSION_ID]` | Conversation/session id (UUID) or thread name. UUIDs take precedence if it parses. If omitted, use --last to pick the most recent recorded session | | `[PROMPT]` | Optional user prompt to start the session | | Flag | Short | Type | Description | | -------------------------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--last` | — | boolean | Continue the most recent session without showing the picker | | `--all` | — | boolean | Show all sessions (disables cwd filtering and shows CWD column) | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | | `--include-non-interactive` | — | boolean | Include non-interactive sessions in the resume picker and --last selection | | `--remote <ADDR>` | — | string | Connect the TUI to a remote app server endpoint. | | `--remote-auth-token-env <ENV_VAR>` | — | string | Name of the environment variable containing the bearer token to send to a remote app server websocket | | `--strict-config` | — | boolean | Error out when config.toml contains fields that are not recognized by this version of wizard | | `-i, --image <FILE>...` | `-i` | path | Optional image(s) to attach to the initial prompt | | `-m, --model <MODEL>` | `-m` | enum | Model the agent should use | | `--oss` | — | boolean | Use open-source provider | | `--local-provider <OSS_PROVIDER>` | — | enum | Specify which local provider to use (lmstudio or ollama). If not specified with --oss, will use config default or show selection | | `-p, --profile <CONFIG_PROFILE>` | `-p` | path | Configuration profile from config.toml to specify default options | | `--profile-v2 <CONFIG_PROFILE_V2>` | — | path | Layer $DBT\_WIZARD\_HOME/\<name>.config.toml on top of the base user config | | `-s, --sandbox <SANDBOX_MODE>` | `-s` | enum | Select the sandbox policy to use when executing model-generated shell commands | | `--dangerously-bypass-approvals-and-sandbox` | — | boolean | Skip all confirmation prompts and execute commands without sandboxing. EXTREMELY DANGEROUS. Intended solely for running in environments that are externally sandboxed | | `--dangerously-bypass-hook-trust` | — | boolean | Run enabled hooks without requiring persisted hook trust for this invocation. DANGEROUS. Intended only for automation that already vets hook sources | | `-C, --cd <DIR>` | `-C` | path | Tell the agent to use the specified directory as its working root | | `--add-dir <DIR>` | — | path | Additional directories that should be writable alongside the primary workspace | | `-a, --ask-for-approval <APPROVAL_POLICY>` | `-a` | enum | Configure when the model requires human approval before executing a command | | `--search` | — | boolean | Enable live web search. When enabled, the native Responses `web_search` tool is available to the model (no per‑call approval) | | `--no-alt-screen` | — | boolean | Disable alternate screen mode | #### fork Fork a previous interactive session (picker by default; use --last to fork the most recent) **Arguments:** | Argument | Description | | -------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | `[SESSION_ID]` | Conversation/session id (UUID). When provided, forks this session. If omitted, use --last to pick the most recent recorded session | | `[PROMPT]` | Optional user prompt to start the session | | Flag | Short | Type | Description | | -------------------------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--last` | — | boolean | Fork the most recent session without showing the picker | | `--all` | — | boolean | Show all sessions (disables cwd filtering and shows CWD column) | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | | `--remote <ADDR>` | — | string | Connect the TUI to a remote app server endpoint. | | `--remote-auth-token-env <ENV_VAR>` | — | string | Name of the environment variable containing the bearer token to send to a remote app server websocket | | `--strict-config` | — | boolean | Error out when config.toml contains fields that are not recognized by this version of wizard | | `-i, --image <FILE>...` | `-i` | path | Optional image(s) to attach to the initial prompt | | `-m, --model <MODEL>` | `-m` | enum | Model the agent should use | | `--oss` | — | boolean | Use open-source provider | | `--local-provider <OSS_PROVIDER>` | — | enum | Specify which local provider to use (lmstudio or ollama). If not specified with --oss, will use config default or show selection | | `-p, --profile <CONFIG_PROFILE>` | `-p` | path | Configuration profile from config.toml to specify default options | | `--profile-v2 <CONFIG_PROFILE_V2>` | — | path | Layer $DBT\_WIZARD\_HOME/\<name>.config.toml on top of the base user config | | `-s, --sandbox <SANDBOX_MODE>` | `-s` | enum | Select the sandbox policy to use when executing model-generated shell commands | | `--dangerously-bypass-approvals-and-sandbox` | — | boolean | Skip all confirmation prompts and execute commands without sandboxing. EXTREMELY DANGEROUS. Intended solely for running in environments that are externally sandboxed | | `--dangerously-bypass-hook-trust` | — | boolean | Run enabled hooks without requiring persisted hook trust for this invocation. DANGEROUS. Intended only for automation that already vets hook sources | | `-C, --cd <DIR>` | `-C` | path | Tell the agent to use the specified directory as its working root | | `--add-dir <DIR>` | — | path | Additional directories that should be writable alongside the primary workspace | | `-a, --ask-for-approval <APPROVAL_POLICY>` | `-a` | enum | Configure when the model requires human approval before executing a command | | `--search` | — | boolean | Enable live web search. When enabled, the native Responses `web_search` tool is available to the model (no per‑call approval) | | `--no-alt-screen` | — | boolean | Disable alternate screen mode | #### cloud \[EXPERIMENTAL] Browse tasks from dbt Wizard in dbt platform and apply changes locally | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### exec-server \[EXPERIMENTAL] Run the standalone exec-server service | Flag | Short | Type | Description | | --------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--listen <URL>` | — | string | Transport endpoint URL. Supported values: `ws://IP:PORT` (default), `stdio`, `stdio://` | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--remote <URL>` | — | string | Register this exec-server as a remote environment using the given base URL | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | | `--environment-id <ID>` | — | string | Environment id to attach to when registering remotely | | `--name <NAME>` | — | string | Human-readable environment name | | `--use-agent-identity-auth` | — | boolean | Use Agent Identity auth from DBT\_WIZARD\_ACCESS\_TOKEN for remote registration | #### features Inspect feature flags | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | --- ### _wizard-slash-commands-generated #### Slash commands Type `/` in the composer to open the command picker. Use arrow keys to navigate or keep typing to filter. Press **Tab** to queue a command while a task is running — it executes at the end of the current turn. ##### All slash commands | Command | Alias | Purpose | Available during task | | --------------------------------------------------- | -------- | ---------------------------------------------------------------------- | --------------------- | | [`/model`](#model-and-ai) | — | Choose what model and reasoning effort to use | ✗ | | [`/providers`](#model-and-ai) | — | Configure model providers and BYOK credentials | ✓ | | [`/ide`](#review-and-context) | — | Include current selection, open files, and other context from your IDE | ✓ | | [`/permissions`](#permissions-and-safety) | — | Choose what Wizard is allowed to do | ✗ | | [`/keymap`](#customization) | — | Remap TUI shortcuts | ✗ | | [`/vim`](#customization) | — | Toggle Vim mode for the composer | ✗ | | [`/setup-default-sandbox`](#permissions-and-safety) | — | Set up elevated agent sandbox | ✗ | | [`/sandbox-add-read-dir`](#permissions-and-safety) | — | Let sandbox read a directory: /sandbox-add-read-dir \<absolute\_path> | ✗ | | [`/experimental`](#customization) | — | Toggle experimental features | ✗ | | [`/approve`](#customization) | — | Approve one retry of a recent auto-review denial | ✓ | | [`/memories`](#memory) | — | Configure memory use and generation | ✗ | | [`/skills`](#skills-and-extensions) | — | Use skills to improve how Wizard performs specific tasks | ✓ | | [`/hooks`](#skills-and-extensions) | — | View and manage lifecycle hooks | ✓ | | [`/review`](#review-and-context) | — | Review my current changes and find issues | ✗ | | [`/rename`](#session-management) | — | Rename the current thread | ✓ | | [`/new`](#session-management) | — | Start a new chat during a conversation | ✗ | | [`/resume`](#session-management) | — | Resume a saved chat | ✗ | | [`/fork`](#session-management) | — | Fork the current chat | ✗ | | [`/init`](#session-management) | — | Create an AGENTS.md file with instructions for Wizard | ✗ | | [`/compact`](#session-management) | — | Summarize conversation to prevent hitting the context limit | ✗ | | [`/plan`](#long-running-tasks) | — | Switch to Plan mode | ✗ | | [`/goal`](#long-running-tasks) | — | Set or view the goal for a long-running task | ✓ | | [`/agent`](#long-running-tasks) | — | Switch the active agent thread | ✓ | | [`/side`](#long-running-tasks) | — | Start a side conversation in an ephemeral fork | ✓ | | [`/btw`](#long-running-tasks) | — | Start a side conversation in an ephemeral fork | ✓ | | [`/copy`](#review-and-context) | — | Copy last response as markdown | ✓ | | [`/raw`](#review-and-context) | — | Toggle raw scrollback mode for copy-friendly terminal selection | ✓ | | [`/diff`](#review-and-context) | — | Show git diff (including untracked files) | ✓ | | [`/mention`](#review-and-context) | — | Mention a file | ✓ | | [`/status`](#session-info) | — | Show current session configuration and token usage | ✓ | | [`/config`](#session-info) | — | View and manage configuration | ✓ | | [`/debug-config`](#session-info) | — | Show config layers and requirement sources for debugging | ✓ | | [`/title`](#customization) | — | Configure which items appear in the terminal title | ✓ | | [`/statusline`](#customization) | — | Configure which items appear in the status line | ✓ | | [`/theme`](#customization) | — | Choose a syntax highlighting theme | ✗ | | [`/pets`](#customization) | `/pet` | Choose or hide the terminal pet | ✗ | | [`/overview`](#review-and-context) | — | Render the dbt project overview card | ✓ | | [`/mcp`](#skills-and-extensions) | — | List configured MCP tools; use /mcp verbose for details | ✓ | | [`/apps`](#skills-and-extensions) | — | Manage apps | ✓ | | [`/plugins`](#skills-and-extensions) | — | Browse plugins | ✓ | | [`/quit`](#exit) | — | Exit Wizard | ✓ | | [`/exit`](#exit) | — | Exit Wizard | ✓ | | [`/feedback`](#session-info) | — | Send feedback to maintainers | ✓ | | [`/ps`](#background-terminals) | — | List background terminals | ✓ | | [`/stop`](#background-terminals) | `/clean` | Stop all background terminals | ✓ | | [`/clear`](#session-management) | — | Clear the terminal and start a new chat | ✗ | | [`/personality`](#model-and-ai) | — | Choose a communication style for Wizard | ✗ | | [`/realtime`](#realtime-experimental) | — | Toggle realtime voice mode (experimental) | ✓ | | [`/settings`](#realtime-experimental) | — | Configure realtime microphone/speaker | ✓ | | [`/subagents`](#long-running-tasks) | — | Switch the active agent thread | ✓ | *** ##### Model and AI Control the AI model, provider, speed, and response style. | Command | Inline args | Description | | -------------- | ----------- | ---------------------------------------------- | | `/model` | — | Choose what model and reasoning effort to use | | `/providers` | — | Configure model providers and BYOK credentials | | `/personality` | — | Choose a communication style for Wizard | *** ##### Session management Start, resume, branch, and clean up sessions. | Command | Inline args | Description | | ---------- | ----------- | ----------------------------------------------------------- | | `/new` | — | Start a new chat during a conversation | | `/clear` | — | Clear the terminal and start a new chat | | `/resume` | Yes | Resume a saved chat | | `/fork` | — | Fork the current chat | | `/compact` | — | Summarize conversation to prevent hitting the context limit | | `/rename` | Yes | Rename the current thread | | `/init` | — | Create an AGENTS.md file with instructions for Wizard | *** ##### Review and context Pull information into the session or trigger a code review. | Command | Inline args | Description | | ----------- | ----------- | ---------------------------------------------------------------------- | | `/review` | Yes | Review my current changes and find issues | | `/diff` | — | Show git diff (including untracked files) | | `/mention` | — | Mention a file | | `/copy` | — | Copy last response as markdown | | `/raw` | Yes | Toggle raw scrollback mode for copy-friendly terminal selection | | `/overview` | — | Render the dbt project overview card | | `/ide` | Yes | Include current selection, open files, and other context from your IDE | *** ##### Permissions and safety Control what Wizard is allowed to execute. | Command | Inline args | Description | | ------------------------ | ----------- | --------------------------------------------------------------------- | | `/permissions` | — | Choose what Wizard is allowed to do | | `/setup-default-sandbox` | — | Set up elevated agent sandbox | | `/sandbox-add-read-dir` | Yes | Let sandbox read a directory: /sandbox-add-read-dir \<absolute\_path> | *** ##### Customization Appearance, keybindings, and UI preferences. | Command | Inline args | Description | | --------------------- | ----------- | -------------------------------------------------- | | `/theme` | — | Choose a syntax highlighting theme | | `/keymap` | Yes | Remap TUI shortcuts | | `/vim` | — | Toggle Vim mode for the composer | | `/statusline` | — | Configure which items appear in the status line | | `/title` | — | Configure which items appear in the terminal title | | `/experimental` | — | Toggle experimental features | | `/approve` | — | Approve one retry of a recent auto-review denial | | `/pets` (alias: /pet) | Yes | Choose or hide the terminal pet | *** ##### Skills and extensions Manage capabilities Wizard can use during a session. | Command | Inline args | Description | | ---------- | ----------- | -------------------------------------------------------- | | `/skills` | — | Use skills to improve how Wizard performs specific tasks | | `/hooks` | — | View and manage lifecycle hooks | | `/mcp` | Yes | List configured MCP tools; use /mcp verbose for details | | `/apps` | — | Manage apps | | `/plugins` | — | Browse plugins | *** ##### Long-running tasks Manage multi-turn goals, parallel agents, and branched conversations. | Command | Inline args | Description | | ------------ | ----------- | ---------------------------------------------- | | `/plan` | Yes | Switch to Plan mode | | `/goal` | Yes | Set or view the goal for a long-running task | | `/agent` | — | Switch the active agent thread | | `/subagents` | — | Switch the active agent thread | | `/side` | Yes | Start a side conversation in an ephemeral fork | | `/btw` | Yes | Start a side conversation in an ephemeral fork | *** ##### Background terminals Inspect and control shell processes Wizard has running in the background. | Command | Inline args | Description | | ----------------------- | ----------- | ----------------------------- | | `/ps` | — | List background terminals | | `/stop` (alias: /clean) | — | Stop all background terminals | *** ##### Memory Control how Wizard stores and uses memory across sessions. | Command | Inline args | Description | | ----------- | ----------- | ----------------------------------- | | `/memories` | — | Configure memory use and generation | *** ##### Session info Inspect the current session state without changing anything. | Command | Inline args | Description | | --------------- | ----------- | -------------------------------------------------------- | | `/status` | — | Show current session configuration and token usage | | `/config` | — | View and manage configuration | | `/debug-config` | — | Show config layers and requirement sources for debugging | | `/feedback` | — | Send feedback to maintainers | *** ##### Exit | Command | Inline args | Description | | ------- | ----------- | ----------- | | `/quit` | — | Exit Wizard | | `/exit` | — | Exit Wizard | *** ##### Realtime (experimental) | Command | Inline args | Description | | ----------- | ----------- | ----------------------------------------- | | `/realtime` | — | Toggle realtime voice mode (experimental) | | `/settings` | — | Configure realtime microphone/speaker | *** --- ### 2022 dbt platform release notes dbt platform | Free Archived release notes for dbt from 2022 #### December 2022 ##### Threads default value changed to 4 Threads help parallelize node execution in the dbt directed acyclic graph [(DAG)](https://docs.getdbt.com/terms/dag). Previously, the thread value defaulted to 1, which can increase the runtime of your project. To help reduce the runtime of your project, the default value for threads in user profiles is now set to 4 threads. You can supply a custom thread count if you'd prefer more or less parallelization. For more information, read [Understanding threads](https://docs.getdbt.com/docs/running-a-dbt-project/using-threads.md). ##### Creating a new job no longer triggers a run by default To help save compute time, new jobs will no longer be triggered to run by default. When you create a new job in dbt, you can trigger the job to run by selecting **Run on schedule** and completing the desired schedule and timing information. For more information, refer to [Deploy jobs](https://docs.getdbt.com/docs/deploy/deploy-jobs.md). ##### Private packages must be cloned using access tokens provided by environment variables The supported method for cloning private GitHub packages is the [git token method](https://docs.getdbt.com/docs/build/packages.md#git-token-method), where an appropriate access token is passed into the package repository URL with an environment variable. A small number of people have been able to clone private packages using dbt's native GitHub application without explicitly providing an access token. This functionality is being deprecated as it’s limited in flexibility. If you have been using a package hosted in a private repository on GitHub, you must start passing an access token into the URL. An example of passing an access token: packages.yml ```yaml packages: - git: "https://{{env_var('DBT_ENV_SECRET_GIT_CREDENTIAL')}}@github.com/dbt-labs/awesome_repo.git" ``` #### November 2022 ##### The dbt Cloud + Databricks experience is getting even better dbt is the easiest and most reliable way to develop and deploy a dbt project. It helps remove complexity while also giving you more features and better performance. A simpler Databricks connection experience with support for Databricks’ Unity Catalog and better modeling defaults is now available for your use. For all the Databricks customers already using dbt with the dbt-spark adapter, you can now [migrate](https://docs.getdbt.com/guides/migrate-from-spark-to-databricks.md) your connection to the [dbt-databricks adapter](https://docs.getdbt.com/docs/local/connect-data-platform/databricks-setup.md) to get the benefits. [Databricks](https://www.databricks.com/blog/2022/11/17/introducing-native-high-performance-integration-dbt-cloud.html) is committed to maintaining and improving the adapter, so this integrated experience will continue to provide the best of dbt and Databricks. Check out our [live blog post](https://www.getdbt.com/blog/dbt-cloud-databricks-experience/) to learn more. ##### Extra features in new and refreshed IDE The refreshed version of the Studio IDE has launched four brand-new additional features, making it easier and faster for you to develop in the Studio IDE. The new features are: * **Formatting** — Format your dbt SQL files to a single code style with a click of a button. This uses the tool [sqlfmt](https://github.com/tconbeer/sqlfmt). * **Git diff view** — Highlights the changes in a file before opening a pull request. * **dbt autocomplete** — There are four new types of autocomplete features to help you develop faster: * Use `ref` to autocomplete your model names * Use `source` to autocomplete your source name + table name * Use `macro` to autocomplete your arguments * Use `env var` to autocomplete env var * **Dark mode** — Use dark mode in the Studio IDE for low-light environments. Read more about all the [Cloud Studio IDE features](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md#studio-ide-features). ##### Classic IDE deprecation notice In December 2022, dbt Labs will deprecate the classic Studio IDE. The [new and refreshed Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) will be available for *all* dbt users. You will no longer be able to access the classic Studio IDE and dbt Labs might introduce changes that break the classic Studio IDE. With deprecation, dbt Labs will only support the refreshed version of the Studio IDE. Virtual Private Cloud (VPC) customers with questions about when this change will affect your account can contact your account team or support contact for assistance. #### October 2022 ##### Announcing dbt Cloud’s native integration with Azure DevOps dbt now offers a native integration with Azure DevOps for dbt customers on the enterprise plan. We built this integration to remove friction, increase security, and unlock net new product experiences for our customers. [Setting up the Azure DevOps integration](https://docs.getdbt.com/docs/platform/git/connect-azure-devops.md) in dbt provides: * easy dbt project set up, * an improved security posture, * repo permissions enforcement in Studio IDE, and * dbt Slim CI. Check out our [live blog post](https://www.getdbt.com/blog/dbt-cloud-integration-azure-devops/) to learn more! ##### Introducing a snappier, improved, and powerful Cloud IDE The new version of the Cloud Studio IDE makes it easy for you to build data models without thinking much about environment setup and configuration. The new Cloud Studio IDE includes performance upgrades, ergonomics improvements, and some delightful enhancements! Some of the improvements include: * Improved Cloud Studio IDE startup time (starting the Studio IDE), interaction time (saving and committing), and reliability. * Better organization and navigation with features like drag and drop of files, breadcrumb, build button drop-down, and more. * You can use new features like auto-format your file, auto-complete model names, and git diff view to see your changes before making a pull request. Read more about the new [Cloud Studio IDE features](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md#studio-ide-features) and check out [New and improved Cloud Studio IDE](https://www.getdbt.com/blog/new-improved-cloud-ide/) blog for more info! #### September 2022 ##### List Steps API endpoint deprecation warning On October 14th, 2022 dbt Labs is deprecating the List Steps API endpoint. From October 14th, any GET requests to this endpoint will fail. Please prepare to stop using the List Steps endpoint as soon as possible. dbt Labs will continue to maintain the [Retrieve Run](https://docs.getdbt.com/dbt-cloud/api-v2#/operations/Retrieve%20Run) endpoint, which is a viable alternative depending on the use case. You can fetch run steps for an individual run with a GET request to the following URL, replacing `YOUR_ACCESS_URL` with the [appropriate Access URL](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for your region and plan: `https://YOUR_ACCESS_URL/api/v2/accounts/{accountId}/runs/{runId}/?include_related=["run_steps"]` ##### Query the previous three months of data using the metadata API In order to make the metadata API more scalable and improve its latency, we’ve implemented data retention limits. The metadata API can now query data from the previous three months. For example, if today was March 1, you could query data back to January 1st. For more information, see [Metadata API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md) #### August 2022 ##### Support for cross-database sources on Redshift RA3 instances Cross-database queries for RA3 instances are now supported by dbt projects using a Redshift connection. With cross-database queries, you can seamlessly query data from any database in the cluster, regardless of which database you are connected to with dbt. The [connection configuration](https://docs.getdbt.com/docs/local/connect-data-platform/redshift-setup.md) `ra3_node` has been defaulted to `true`. This allows users to: * benefit from the full RA3 nodes’ capabilities, * generate appropriate dbt documentation. #### July 2022 ##### Large DAG feature You can now select **Render Lineage** to visualize large DAGs. Large DAGs can take a long time (10 or more seconds, if not minutes) to render and can cause browsers to crash. The new button prevents large DAGs from rendering automatically. Instead, you can select **Render Lineage** to load the visualization. This should affect about 15% of the DAGs. [![Render Lineage](</img/docs/dbt-platform/dag v1.1.56 release.png?v=2> "Render Lineage")](#)Render Lineage #### May 2022 ##### Refresh expired access tokens in the IDE when using GitLab On May 22, GitLab changed how they treat [OAuth access tokens that don't expire](https://docs.gitlab.com/ee/update/deprecations.html#oauth-tokens-without-expiration). We updated our Studio IDE logic to handle OAuth token expiration more gracefully. Now, the first time your token expires after 2 hours of consecutive Studio IDE usage, you will have to re-authenticate in GitLab to refresh your expired OAuth access token. We will handle subsequent refreshes for you if you provide the authorization when you re-authenticate. This additional security layer in the Studio IDE is available only to the dbt enterprise plan. #### April 2022 ##### Audit log To review actions performed by people in your organization, dbt provides logs of audited user and system events. The dbt audit log lists events triggered in your organization within the last 90 days. The audit log includes details such as who performed the action, what the action was, and when it was performed. For more details, review [the audit log for dbt Enterprise](https://docs.getdbt.com/docs/platform/manage-access/audit-log.md) documentation. ##### Credentials no longer accidentally wiped when editing an environment We resolved a bug where when updating unencrypted fields (for example, threads, schema name) in an environment setting would cause secret fields (for example, password, keypair, credential details) to be deleted from that environment. Now users can freely update environment settings without fear of unintentionally wiping credentials. ##### Email verification To enhance the security of user creation, dbt users created using SAML Just-in-Time (JIT) will now confirm identity via email to activate their account. Using email to confirm identity ensures the user still has access to the same email address they use to login via SAML. ##### Scheduler performance improvements We rolled out our new distributed scheduler, which has much faster prep times, especially at the top of the hour. We share more about our work and improvements in our [product news blog post](https://www.getdbt.com/blog/a-good-problem-to-have/). #### March 2022 ##### Spotty internet issues no longer cause a session time out message We fixed an issue where a spotty internet connection could cause the “Studio IDE session timed out” message to appear unexpectedly. People using a VPN were most likely to see this issue. We updated the health check logic so it now excludes client-side connectivity issues from the Studio IDE session check. If you lose your internet connection, we no longer update the health-check state. Now, losing internet connectivity will no longer cause this unexpected message. [![Fix Session Timeout](</img/docs/dbt-platform/Fix Session Timeout.png?v=2> "Fix Session Timeout")](#)Fix Session Timeout ##### Dividing queue time into waiting and prep time dbt now shows "waiting time" and "prep time" for a run, which used to be expressed in aggregate as "queue time". Waiting time captures the time dbt waits to run your job if there isn't an available run slot or if a previous run of the same job is still running. Prep time represents the time it takes dbt to ready your job to run in your cloud data warehouse. [![New prep time and waiting time](/img/docs/dbt-platform/v1.1.46releasenotes_img1.png?v=2 "New prep time and waiting time")](#)New prep time and waiting time #### February 2022 ##### DAG updates and performance improvements Love the DAG in the Studio IDE as much as we do? Now when you click on a node in the DAG, the model or config file will open as a new tab in the Studio IDE, so you can directly view or edit the code. We'll continue to ship better developer ergonomic functionality throughout the year. ###### Performance improvements and enhancements * Updated recommended dbt commands in the Studio IDE to include dbt v1 commands, such as "build" and the "--select" argument. ##### Service tokens and bug fixes Service tokens can now be assigned granular permissions to enforce least privilege access. If you're on Enterprise, you can assign any enterprise permission set to newly issued service tokens. If you're on Teams, you can assign the Job Admin permission set to newly issued service tokens. We highly recommend you re-issue service tokens with these new permissions to increase your security posture! See docs [here](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md#permissions-for-service-account-tokens). ###### New products and features * We are joining the [GitHub secret scanning partner program](https://docs.github.com/en/developers/overview/secret-scanning-partner-program) to better secure your token against accidental public exposure and potential fraudulent usage. ###### Bug fixes * Credentials are no longer accidentally deleted when a user updates an environment setting. #### January 2022 ##### Autocomplete snippets for SQL and YAML files in IDE Some noteworthy improvements include autocomplete snippets for SQL and YAML files in the IDE, which are available for use now! We also added a [new metric layer page](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) to docs.getdbt.com to help you begin thinking about the metrics layer in dbt Cloud. ###### Performance improvements and enhancements * Branch names now default to "main" instead of "master" in new managed and unmanaged Git repositories. * Update IDE autocomplete snippets. ##### Model timing for Multi-tenant Team and Enterprise accounts We started the new year with a gift! Multi-tenant Team and Enterprise accounts can now use the new [Model timing](https://docs.getdbt.com/docs/deploy/deploy-jobs.md#model-timing) tab in dbt. You can use this tab to further explore long-running models to see if they need refactoring or rescheduling. ###### Performance improvements and enhancements * We added client-side naming validation for file or folder creation. --- ### 2023 dbt platform release notes dbt platform | Free Archived release notes for dbt from 2023 #### December 2023 *  Semantic Layer updates The dbt Labs team continues to work on adding new features, fixing bugs, and increasing reliability for the dbt Semantic Layer. The following list explains the updates and fixes for December 2023 in more detail. #### Bug fixes * Tableau integration — The dbt Semantic Layer integration with Tableau now supports queries that resolve to a "NOT IN" clause. This applies to using "exclude" in the filtering user interface. Previously it wasn’t supported. * `BIGINT` support — The dbt Semantic Layer can now support `BIGINT` values with precision greater than 18. Previously it would return an error. * Memory leak — Fixed a memory leak in the JDBC API that would previously lead to intermittent errors when querying it. * Data conversion support — Added support for converting various Redshift and Postgres-specific data types. Previously, the driver would throw an error when encountering columns with those types. #### Improvements * Deprecation — We deprecated dbt Metrics and the legacy dbt Semantic Layer, both supported on dbt version 1.5 or lower. This change came into effect on December 15th, 2023. * Improved dbt converter tool — The [dbt converter tool](https://github.com/dbt-labs/dbt-converter) can now help automate some of the work in converting from LookML (Looker's modeling language) for those who are migrating. Previously this wasn’t available. *  External attributes The extended attributes feature in dbt Cloud is now GA! It allows for an environment level override on any YAML attribute that a dbt adapter accepts in its `profiles.yml`. You can provide a YAML snippet to add or replace any [profile](https://docs.getdbt.com/docs/local/profiles.yml.md) value. To learn more, refer to [Extended attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes). The **Extended Attributes** text box is available from your environment's settings page: [![Example of the Extended attributes text box](/img/docs/dbt-platform/using-dbt-platform/extended-attributes.png?v=2 "Example of the Extended attributes text box")](#)Example of the Extended attributes text box *  Legacy semantic layer dbt Labs has deprecated dbt Metrics and the legacy dbt Semantic Layer, both supported on dbt version 1.5 or lower. This change starts on December 15th, 2023. This deprecation means dbt Metrics and the legacy Semantic Layer are no longer supported. We also removed the feature from the dbt Cloud user interface and documentation site. ##### Why this change? The [re-released dbt Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md), powered by MetricFlow, offers enhanced flexibility, performance, and user experience, marking a significant advancement for the dbt community. ##### Key changes and impact * **Deprecation date** — The legacy Semantic Layer and dbt Metrics will be officially deprecated on December 15th, 2023. * **Replacement** — [MetricFlow](https://docs.getdbt.com/docs/build/build-metrics-intro.md) replaces dbt Metrics for defining semantic logic. The `dbt_metrics` package will no longer be supported post-deprecation. * **New feature** — Exports replaces the materializing data with `metrics.calculate` functionality and will be available in dbt Cloud in December or January. ##### Breaking changes and recommendations * For users on dbt version 1.5 and lower with dbt Metrics and Snowflake proxy: * **Impact**: Post-deprecation, queries using the proxy *will not* run. * For users on dbt version 1.5 and lower using dbt Metrics without Snowflake proxy: * **Impact**: No immediate disruption, but the package will not receive updates or support after deprecation * **Recommendation**: Plan migration to the re-released Semantic Layer for compatibility with dbt version 1.6 and higher. ##### Engage and support * Feedback and community support — Engage and share feedback with the dbt Labs team and dbt Community slack using channels like [#dbt-cloud-semantic-layer](https://getdbt.slack.com/archives/C046L0VTVR6) and [#dbt-metricflow](https://getdbt.slack.com/archives/C02CCBBBR1D). Or reach out to your dbt Cloud account representative. * Resources for upgrading — Refer to some additional info and resources to help you upgrade your dbt version: * [Upgrade version in dbt Cloud](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) * [Version migration guides](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade.md) #### November 2023 *  New features and UI changes to dbt Catalog There are new quality-of-life improvements in dbt Cloud for email and Slack notifications about your jobs: * You can add external email addresses and send job notifications to them. External emails can be: * Addresses that are outside of your dbt Cloud account * Third-party integration addresses for configuring notifications to services like Microsoft Teams or PagerDuty * You can configure notifications for multiple Slack channels. Previously, you could only configure one Slack channel. * Any account admin can now edit slack notifications, not just the person who created them. To learn more, check out [Job notifications](https://docs.getdbt.com/docs/deploy/job-notifications.md). *  Job notifications There are new quality-of-life improvements in dbt Cloud for email and Slack notifications about your jobs: * You can add external email addresses and send job notifications to them. External emails can be: * Addresses that are outside of your dbt Cloud account * Third-party integration addresses for configuring notifications to services like Microsoft Teams or PagerDuty * You can configure notifications for multiple Slack channels. Previously, you could only configure one Slack channel. * Any account admin can now edit slack notifications, not just the person who created them. To learn more, check out [Job notifications](https://docs.getdbt.com/docs/deploy/job-notifications.md). *  Repo caching Now available for dbt Cloud Enterprise plans is a new option to enable Git repository caching for your job runs. When enabled, dbt Cloud caches your dbt project's Git repository and uses the cached copy instead if there's an outage with the Git provider. This feature improves the reliability and stability of your job runs. To learn more, refer to [Repo caching](https://docs.getdbt.com/docs/platform/account-settings.md#git-repository-caching). [![Example of the Repository caching option](/img/docs/deploy/account-settings-repository-caching.png?v=2 "Example of the Repository caching option")](#)Example of the Repository caching option #### October 2023 *  dbt Cloud APIs Beginning December 1, 2023, the [Administrative API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md) v2 and v3 will expect you to limit all "list" or `GET` API methods to 100 results per API request. This limit enhances the efficiency and stability of our services. If you need to handle more than 100 results, then use the `limit` and `offset` query parameters to paginate those results; otherwise, you will receive an error. This maximum limit applies to [multi-tenant instances](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) only, and *does not* apply to single tenant instances. Refer to the [API v3 Pagination](https://docs.getdbt.com/dbt-cloud/api-v3#/) or [API v2 Pagination](https://docs.getdbt.com/dbt-cloud/api-v2#/) sections for more information on how to paginate your API responses. *  dbt CLI We are excited to announce the dbt CLI, **unified command line for dbt**, is available in public preview. It’s a local development experience, powered by dbt Cloud. It’s easy to get started: `pip3 install dbt` or `brew install dbt` and you’re ready to go. We will continue to invest in the dbt Cloud IDE as the easiest and most accessible way to get started using dbt, especially for data analysts who have never developed software using the command line before. We will keep improving the speed, stability, and feature richness of the IDE, as we have been [all year long](https://www.getdbt.com/blog/improvements-to-the-dbt-cloud-ide/). We also know that many people developing in dbt have a preference for local development, where they can use their favorite terminal, text editor, keybindings, color scheme, and so on. This includes people with data engineering backgrounds, as well as those analytics engineers who started writing code in the dbt Cloud IDE and have expanded their skills. The new dbt CLI offers the best of both worlds, including: * The power of developing against the dbt Cloud platform * The flexibility of your own local setup Run whichever community-developed plugins, pre-commit hooks, or other arbitrary scripts you like. Some of the unique capabilities of this dbt CLI include: * Automatic deferral of build artifacts to your Cloud project's production environment * Secure credential storage in the dbt Cloud platform * Support for dbt Mesh ([cross-project `ref`](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md)) * Development workflow for dbt Semantic Layer * Speedier, lower cost builds Refer to [dbt CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) to learn more. *  Custom branch fix If you don't set a [custom branch](https://docs.getdbt.com/docs/dbt-platform-environments.md#custom-branch-behavior) for your dbt Cloud environment, it now defaults to the default branch of your Git repository (for example, `main`). Previously, [CI jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md) would run for pull requests (PRs) that were opened against *any branch* or updated with new commits if the **Custom Branch** option wasn't set. #### Azure DevOps Your Git pull requests (PRs) might not trigger against your default branch if you're using Azure DevOps and the default branch isn't `main` or `master`. To resolve this, [set up a custom branch](https://docs.getdbt.com/faqs/Environments/custom-branch-settings.md) with the branch you want to target. *  dbt deps auto install The dbt Cloud IDE and dbt CLI now automatically installs `dbt deps` when your environment starts or when necessary. Previously, it would prompt you to run `dbt deps` during initialization. This improved workflow is available to all multi-tenant dbt Cloud users (Single-tenant support coming next week) and applies to dbt versions. However, you should still run the `dbt deps` command in these situations: * When you make changes to the `packages.yml` or `dependencies.yml` file during a session * When you update the package version in the `packages.yml` or `dependencies.yml` file. * If you edit the `dependencies.yml` file and the number of packages remains the same, run `dbt deps`. (Note that this is a known bug dbt Labs will fix in the future.) *  Native retry support Previously in dbt Cloud, you could only rerun an errored job from start but now you can also rerun it from its point of failure. You can view which job failed to complete successfully, which command failed in the run step, and choose how to rerun it. To learn more, refer to [Retry jobs](https://docs.getdbt.com/docs/deploy/retry-jobs.md). [![Example of the Rerun options in dbt Cloud](/img/docs/deploy/native-retry.gif?v=2 "Example of the Rerun options in dbt Cloud")](#)Example of the Rerun options in dbt Cloud *  Product docs updates Hello from the dbt Docs team: @mirnawong1, @matthewshaver, @nghi-ly, and @runleonarun! First, we’d like to thank the 15 new community contributors to docs.getdbt.com. We merged [107 PRs](https://github.com/dbt-labs/docs.getdbt.com/pulls?q=is%3Apr+merged%3A2023-09-01..2023-09-31) in September. Here's what's new to [docs.getdbt.com](http://docs.getdbt.com/): * Migrated docs.getdbt.com from Netlify to Vercel. #### ☁ Cloud projects * Continuous integration jobs are now generally available and no longer in beta! * Added [Postgres PrivateLink set up page](https://docs.getdbt.com/docs/platform/secure/private-connectivity/aws/aws-postgres.md) * Published beta docs for [dbt Explorer](https://docs.getdbt.com/docs/explore/explore-projects.md). * Added a new Semantic Layer [GraphQL API doc](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md) and updated the [integration docs](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) to include Hex. Responded to dbt community feedback and clarified Metricflow use cases for dbt Core and dbt Cloud. * Added an [FAQ](https://docs.getdbt.com/faqs/Git/git-migration.md) describing how to migrate from one git provider to another in dbt Cloud. * Clarified an example and added a [troubleshooting section](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-snowflake.md#troubleshooting) to Snowflake connection docs to address common errors and provide solutions. #### 🎯 Core projects * Deprecated dbt Core v1.0 and v1.1 from the docs. * Added configuration instructions for the [AWS Glue](https://docs.getdbt.com/docs/local/connect-data-platform/glue-setup.md) community plugin. * Revised the dbt Core quickstart, making it easier to follow. Divided this guide into steps that align with the [other guides](https://docs.getdbt.com/guides/manual-install.md?step=1). #### New 📚 Guides, ✏️ blog posts, and FAQs Added a [style guide template](https://docs.getdbt.com/best-practices/how-we-style/6-how-we-style-conclusion.md#style-guide-template) that you can copy & paste to make sure you adhere to best practices when styling dbt projects! #### Upcoming changes Stay tuned for a flurry of releases in October and a filterable guides section that will make guides easier to find! *  Semantic layer GA If you're using the legacy Semantic Layer, we *highly* recommend you [upgrade your dbt version](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) to dbt v1.6 or higher and migrate to the latest Semantic Layer. dbt Labs is thrilled to announce that the [dbt Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) is now generally available. It offers consistent data organization, improved governance, reduced costs, enhanced efficiency, and accessible data for better decision-making and collaboration across organizations. It aims to bring the best of modeling and semantics to downstream applications by introducing: * Brand new [integrations](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) such as Tableau, Google Sheets, Hex, Mode, and Lightdash. * New [Semantic Layer APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md) using GraphQL and JDBC to query metrics and build integrations. * dbt Cloud [multi-tenant regional](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) support for North America, EMEA, and APAC. Single-tenant support coming soon. * Coming soon — Schedule exports (a way to build tables in your data platform) as part of your dbt Cloud job. Use the APIs to call an export, then access them in your preferred BI tool. [![Use the universal dbt Semantic Layer to define and queried metrics in integration tools.](/img/docs/dbt-platform/semantic-layer/sl-architecture.jpg?v=2 "Use the universal dbt Semantic Layer to define and queried metrics in integration tools.")](#)Use the universal dbt Semantic Layer to define and queried metrics in integration tools. The dbt Semantic Layer is available to [dbt Cloud Team or Enterprise](https://www.getdbt.com/) multi-tenant plans on dbt v1.6 or higher. * Team and Enterprise customers can use 1,000 Queried Metrics per month for no additional cost on a limited trial basis, subject to reasonable use limitations. Refer to [Billing](https://docs.getdbt.com/docs/platform/billing/how-pricing-works.md#what-counts-as-a-queried-metric) for more information. * dbt Developer plans and dbt v1 users can define metrics but won't be able to query them with integrated tools. #### September 2023 *  CI updates dbt Cloud now has two distinct job types: [deploy jobs](https://docs.getdbt.com/docs/deploy/deploy-jobs.md) for building production data assets, and [continuous integration (CI) jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md) for checking code changes. These jobs perform fundamentally different tasks so dbt Labs improved the setup experience with better defaults for each. With two types of jobs, instead of one generic type, we can better guide you through the setup flow. Best practices are built into the default settings so you can go from curious to being set up in seconds. And, we now have more efficient state comparisons on CI checks: never waste a build or test on code that hasn’t been changed. We now diff between the Git pull request (PR) code and what’s running in production more efficiently with the introduction of deferral to an environment versus a job. To learn more, refer to [Continuous integration in dbt](https://docs.getdbt.com/docs/deploy/continuous-integration.md). Below is a comparison table that describes how deploy jobs and CI jobs behave differently: | | Deploy Jobs | CI Jobs | | ---------------------- | ------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------- | | Purpose | Builds production data assets. | Builds and tests new code before merging changes into production. | | Trigger types | Triggered by a schedule or by API. | Triggered by a commit to a PR or by API. | | Destination | Builds into a production database and schema. | Builds into a staging database and ephemeral schema, lived for the lifetime of the PR. | | Execution mode | Runs execute sequentially, so as to not have collisions on the underlying DAG. | Runs execute in parallel to promote team velocity. | | Efficiency run savings | Detects over-scheduled jobs and cancels unnecessary runs to avoid queue clog. | Cancels existing runs when a newer commit is pushed to avoid redundant work. | | State comparison | Only sometimes needs to detect state. | Almost always needs to compare state against the production environment to build on modified code and its dependents. | #### What you need to update * If you want to set up a CI environment for your jobs, dbt Labs recommends that you create your CI job in a dedicated [deployment environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md#create-a-deployment-environment) that's connected to a staging database. To learn more about these environment best practices, refer to the guide [Get started with continuous integration tests](https://docs.getdbt.com/guides/set-up-ci.md). * If you had set up a CI job before October 2, 2023, the job might've been misclassified as a deploy job with this update. Below describes how to fix the job type: If you used the [Create Job](https://docs.getdbt.com/dbt-cloud/api-v2#/operations/Create%20Job) API endpoint but didn't set `"triggers":triggers.git_provider_webhook`, the job was misclassified as a deploy job and you must re-create it as described in [Trigger a CI job with the API](https://docs.getdbt.com/docs/deploy/ci-jobs.md#trigger-a-ci-job-with-the-api). If you used the dbt UI but didn't enable the **Run on Pull Requests** option that was in the **Continuous Integration** (CI) tab, the job was misclassified as a deploy job and you must re-create it as described in [Set up CI jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md#set-up-ci-jobs). To check for the job type, review your CI jobs in dbt's [Run History](https://docs.getdbt.com/docs/deploy/run-visibility.md#run-history) and check for the **CI Job** tag below the job name. If it doesn't have this tag, it was misclassified and you need to re-create the job. **CI update phase 3 — Update: Improved automatic deletion of temporary schemas** Temporary schemas are now being automatically deleted (dropped) for all adapters (like Databricks), PrivateLink connections, and environment variables in connection strings. dbt Labs has rearchitected how schema deletion works for [continuous integration (CI)](https://docs.getdbt.com/docs/deploy/continuous-integration.md) runs. We created a new service to delete any schema with a prefix of `dbt_cloud_pr_` that's been generated by a PR run. However, temporary schemas will not be automatically deleted if: * Your project overrides the [generate\_schema\_name macro](https://docs.getdbt.com/docs/build/custom-schemas.md) but it doesn't contain the required prefix `dbt_cloud_pr_`. For details, refer to [Troubleshooting](https://docs.getdbt.com/docs/deploy/ci-jobs.md#troubleshooting). * You're using a [non-native Git integration](https://docs.getdbt.com/docs/deploy/ci-jobs.md#trigger-a-ci-job-with-the-api). This is because automatic deletion relies on incoming webhooks from Git providers, which is only available through the native integrations. *  Product docs updates Hello from dbt's Product Documentation team (the stewards of the docs.getdbt.com site): @mirnawong1, @matthewshaver, @nghi-ly, and @runleonarun. What a busy summer! We merged 256 PRs between July 1st and August 31. We'd like to recognize all of the docs and support from our partner team, Developer Experience: @jasnonaz @gwenwindflower @dbeatty10 @dataders @joellabes @Jstein77 @dave-connors-3! We'd also like to give a special thanks to the 22 community members who contributed to the [dbt Product docs](https://docs.getdbt.com) for the first time. 🙏 Based on feedback from the dbt community, we made these changes: * Added a [permissions table](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) for Enterprise accounts * Added a [browser session page](https://docs.getdbt.com/docs/platform/about-platform/browsers.md#browser-sessions) that clarifies dbt Cloud’s browser session time and when it logs users off. You can provide feedback by opening a pull request or issue in [our repo](https://github.com/dbt-labs/docs.getdbt.com) or reaching out in the dbt community Slack channel [#dbt-product-docs](https://getdbt.slack.com/archives/C0441GSRU04)). #### ⚡ General docs projects * Added the ability to collapse sections you’re not currently looking at. There were quite a few people who wanted this, and it bugged us too, so we were happy to get this shipped! * Introduced the idea of [“Trusted” adapters](https://docs.getdbt.com/docs/supported-data-platforms.md#types-of-adapters). #### ☁ Cloud projects * The **What’s new?** product update widget is back in the dbt Cloud UI! The Docs team will begin updating the content to keep you informed about new features. * Launched the re-released [Semantic Layer beta docs](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md), which introduces users to the new API, new guide to set up MetricFlow and the new Semantic Layer, as well as revamp the ‘Use the dbt Semantic Layer’ section for users. * Updated [Admin API v2 and v3](https://docs.getdbt.com/docs/dbt-apis/admin-api.md) to help you understand the differences between them and which version includes the endpoints you use. * To improve discoverability, the docs team made changes to the [deploy dbt sidebar](https://docs.getdbt.com/docs/deploy/deployments.md). We added cards and aligned better with the dbt Cloud UI and the way it’s used. * Deprecated legacy job schemas in the [Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md). * Added a page to describe [experimental and beta features](https://docs.getdbt.com/docs/dbt-versions/experimental-features.md) in dbt Cloud and what you need to know about them. * Added a section to introduce a new beta feature [**Extended Attributes**](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes-beta), which allows users to set a flexible `profiles.yml` snippet in their dbt Cloud Environment settings. #### 🎯 Core projects * We released [dbt 1.6](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.6.md>)! We added docs for the new commands `dbt retry` and `dbt clone` #### New 📚 Guides, ✏️ blog posts, and FAQs * Learn how to [join the dbt community](https://docs.getdbt.com/community/join.md). * Blog posts published this summer include [Optimizing Materialized Views with dbt](https://docs.getdbt.com/blog/announcing-materialized-views), [Data Vault 2.0 with dbt Cloud](https://docs.getdbt.com/blog/data-vault-with-dbt-cloud), and [Create dbt Documentation and Tests 10x faster with ChatGPT](https://docs.getdbt.com/blog/create-dbt-documentation-10x-faster-with-ChatGPT) - We now have two new best practice guides: [How we build our metrics](https://docs.getdbt.com/best-practices/how-we-build-our-metrics/semantic-layer-1-intro.md) and [Set up Continuous Integration](https://docs.getdbt.com/guides/set-up-ci.md). *  Removing prerelease versions Previously, when dbt Labs released a new [version](https://docs.getdbt.com/docs/dbt-versions.md#how-dbt-v1-uses-semantic-versioning) in dbt Cloud, the older patch *prerelease* version and the *latest* version remained as options in the dropdown menu available in the **Environment settings**. Now, when the *latest* version is released, the *prerelease* version will be removed and all customers remaining on it will be migrated seamlessly. There will be no interruptions to service when this migration occurs. To see which version you are currently using and to upgrade, select **Deploy** in the top navigation bar and select **Environments**. Choose the preferred environment and click **Settings**. Click **Edit** to make a change to the current dbt version. dbt Labs recommends always using the latest version whenever possible to take advantage of new features and functionality. #### August 2023 *  Deprecation of endpoints in the Discovery API dbt Labs has deprecated and will be deprecating certain query patterns and replacing them with new conventions to enhance the performance of the dbt Cloud [Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md). All these changes will be in effect on *September 7, 2023*. We understand that these changes might require adjustments to your existing integration with the Discovery API. Please [contact us](mailto:support@getdbt.com) with any questions. We're here to help you during this transition period. #### Job-based queries Job-based queries that use the data type `Int` for IDs will be deprecated. They will be marked as deprecated in the [GraphQL explorer](https://metadata.cloud.getdbt.com/graphql). The new convention will be for you to use the data type `BigInt` instead. This change will be in effect starting September 7, 2023. Example of query before deprecation: ```graphql query ($jobId: Int!) { models(jobId: $jobId){ uniqueId } } ``` Example of query after deprecation: ```graphql query ($jobId: BigInt!) { job(id: $jobId) { models { uniqueId } } } ``` #### modelByEnvironment queries The `modelByEnvironment` object has been renamed and moved into the `environment` object. This change is in effect and has been since August 15, 2023. Example of query before deprecation: ```graphql query ($environmentId: Int!, $uniqueId: String) { modelByEnvironment(environmentId: $environmentId, uniqueId: $uniqueId) { uniqueId executionTime executeCompletedAt } } ``` Example of query after deprecation: ```graphql query ($environmentId: BigInt!, $uniqueId: String) { environment(id: $environmentId) { applied { modelHistoricalRuns(uniqueId: $uniqueId) { uniqueId executionTime executeCompletedAt } } } } ``` #### Environment and account queries Environment and account queries that use `Int` as a data type for ID have been deprecated. IDs must now be in `BigInt`. This change is in effect and has been since August 15, 2023. Example of query before deprecation: ```graphql query ($environmentId: Int!, $first: Int!) { environment(id: $environmentId) { applied { models(first: $first) { edges { node { uniqueId executionInfo { lastRunId } } } } } } } ``` Example of query after deprecation: ```graphql query ($environmentId: BigInt!, $first: Int!) { environment(id: $environmentId) { applied { models(first: $first) { edges { node { uniqueId executionInfo { lastRunId } } } } } } } ``` *  dbt Cloud IDE v1.2 We're excited to announce that we replaced the backend service that powers the Cloud IDE with a more reliable server -- dbt-server. Because this release contains foundational changes, IDE v1.2 requires dbt v1.6 or higher. This significant update follows the rebuild of the IDE frontend last year. We're committed to improving the IDE to provide you with a better experience. Previously, the Cloud IDE used dbt-rpc, an outdated service that was unable to stay up-to-date with changes from dbt-core. The dbt-rpc integration used legacy dbt-core entry points and logging systems, causing it to be sluggish, brittle, and poorly tested. The Core team had been working around this outdated technology to avoid breaking it, which prevented them from developing with velocity and confidence. #### New features * **Better dbt-core parity:** The Cloud IDE has better command parity with dbt-core, including support for commands like `dbt list` and improved treatment of flags like `--vars`, `--fail-fast`, etc. * **Improved maintainability:** With the new dbt-server, it's easier to fix bugs and improve the overall quality of the product. With dbt-rpc, fixing bugs was a time-consuming and challenging process that required extensive testing. With the new service, we can identify and fix bugs more quickly, resulting in a more stable and reliable IDE. * **A more reliable service:** Simplified architecture that's less prone to failure. ##### Product refinements * Improved `Preview` capabilities with Core v1.6 + IDE v1.2. [This Loom](https://www.loom.com/share/12838feb77bf463c8585fc1fc6aa161b) provides more information. ##### Bug fixes * Global page can become "inert" and stop handling clicks * Switching back and forth between files in the git diff view can cause overwrite * Browser gets stuck during markdown preview for doc with large table * Editor right click menu is offset * Unable to Cancel on the Save New File component when Closing All Files in the IDE * Mouse flicker in the modal's file tree makes it difficult to select a folder where you want to save a new file * Snapshots not showing in Lineage when inside a subfolder and is mixed cased named * Tooltips do not work for Format and Save * When a dbt invocation is in progress or if parsing is ongoing, attempting to switch branches will cause the `Git Branch` dropdown to close automatically ##### Known issues * `{{this}}` function does not display properly in preview/compile with dbt-server #### July 2023 *  Faster runs and unlimited job concurrency for Enterprise account We’ve introduced significant improvements to the dbt Cloud Scheduler, offering improved performance, durability, and scalability. Read more on how you can experience faster run start execution and how enterprise users can now run as many jobs concurrently as they want to. #### Faster run starts The Scheduler takes care of preparing each dbt Cloud job to run in your cloud data platform. This [prep](https://docs.getdbt.com/docs/deploy/job-scheduler.md#scheduler-queue) involves readying a Kubernetes pod with the right version of dbt installed, setting environment variables, loading data platform credentials, and git provider authorization, amongst other environment-setting tasks. Only after the environment is set up, can dbt execution begin. We display this time to the user in dbt Cloud as “prep time”. [![The scheduler prepares a job for execution and displays it as 'prep time' in dbt Cloud.](/img/run-start.jpg?v=2 "The scheduler prepares a job for execution and displays it as 'prep time' in dbt Cloud.")](#)The scheduler prepares a job for execution and displays it as 'prep time' in dbt Cloud. For all its strengths, Kubernetes has challenges, especially with pod management impacting run execution time. We’ve rebuilt our scheduler by ensuring faster job execution with a ready pool of pods to execute customers’ jobs. This means you won't experience long prep times at the top of the hour, and we’re determined to keep runs starting near instantaneously. Don’t just take our word, review the data yourself. [![Job prep time data has seen a 75% speed improvement from Jan 2023 to July 2023. Prep time took 106 secs in Jan and now takes 27 secs as of July.](/img/prep-start.jpg?v=2 "Job prep time data has seen a 75% speed improvement from Jan 2023 to July 2023. Prep time took 106 secs in Jan and now takes 27 secs as of July.")](#)Job prep time data has seen a 75% speed improvement from Jan 2023 to July 2023. Prep time took 106 secs in Jan and now takes 27 secs as of July. Jobs scheduled at the top of the hour used to take over 106 seconds to prepare because of the volume of runs the scheduler has to process. Now, even with increased runs, we have reduced prep time to 27 secs (at a maximum) — a 75% speed improvement for runs at peak traffic times! #### Unlimited job concurrency for Enterprise accounts Our enhanced scheduler offers more durability and empowers users to run jobs effortlessly. This means Enterprise, multi-tenant accounts can now enjoy the advantages of unlimited job concurrency. Previously limited to a fixed number of run slots, Enterprise accounts now have the freedom to operate without constraints. Single-tenant support will be coming soon. Something to note, each running job occupies a run slot for its duration, and if all slots are occupied, jobs will queue accordingly. For more feature details, refer to the [dbt pricing page](https://www.getdbt.com/pricing/). Note, Team accounts created after July 2023 benefit from unlimited job concurrency: * Legacy Team accounts have a fixed number of run slots. * Both Team and Developer plans are limited to one project each. For larger-scale needs, our [Enterprise plan](https://www.getdbt.com/pricing/) offers features such as audit logging, unlimited job concurrency and projects, and more. #### June 2023 *  Lint format dbt Labs is excited to announce you can now lint and format your dbt code in the dbt Cloud IDE. This is an enhanced development workflow which empowers you to effortlessly prioritize code quality. You can perform linting and formatting on five different file types: SQL, YAML, Markdown, Python, and JSON. For SQL files, you can easily lint and format your code using [SQLFluff](https://sqlfluff.com/) and apply consistent formatting using [sqlfmt](http://sqlfmt.com/). Additionally, for other file types like YAML, Markdown, JSON, and Python, you can utilize the respective tools powered by [Prettier](https://prettier.io/) and [Black](https://black.readthedocs.io/en/latest/) to ensure clean and standardized code formatting. For more info, read [Lint and format your code](https://docs.getdbt.com/docs/platform/studio-ide/lint-format.md). [![Use SQLFluff to lint/format your SQL code, and view code errors in the Code Quality tab.](/img/docs/dbt-platform/platform-ide/sqlfluff.gif?v=2 "Use SQLFluff to lint/format your SQL code, and view code errors in the Code Quality tab.")](#)Use SQLFluff to lint/format your SQL code, and view code errors in the Code Quality tab. [![Use sqlfmt to format your SQL code.](/img/docs/dbt-platform/platform-ide/sqlfmt.gif?v=2 "Use sqlfmt to format your SQL code.")](#)Use sqlfmt to format your SQL code. [![Format YAML, Markdown, and JSON files using Prettier.](/img/docs/dbt-platform/platform-ide/prettier.gif?v=2 "Format YAML, Markdown, and JSON files using Prettier.")](#)Format YAML, Markdown, and JSON files using Prettier. *  CI updates dbt Cloud CI is a critical part of the analytics engineering workflow. Large teams rely on process to ensure code quality is high, and they look to dbt Cloud CI to automate testing code changes in an efficient way, enabling speed while keep the bar high. With status checks directly posted to their dbt PRs, developers gain the confidence that their code changes will work as expected in production, and once you’ve grown accustomed to seeing that green status check in your PR, you won’t be able to work any other way. [![CI checks directly from within Git](/img/docs/release-notes/ci-checks.png?v=2 "CI checks directly from within Git")](#)CI checks directly from within Git What separates dbt CI from other CI providers is its ability to keep track of state of what’s running in your production environment, so that when you run a CI job, only the modified data assets in your pull request and their downstream dependencies get built and tested in a staging schema. dbt aims to make each CI check as efficient as possible, so as to not waste any data warehouse resources. As soon as the CI run completes, its status posts directly back to the PR in GitHub, GitLab, or Azure DevOps, depending on which Git provider you’re using. Teams can set up guardrails to let only PRs with successful CI checks be approved for merging, and the peer review process is greatly streamlined because dbt does the first testing pass. We're excited to introduce a few critical capabilities to dbt CI that will improve productivity and collaboration in your team’s testing and integration workflow. As of this week, you can now: * **Run multiple CI checks in parallel**. If more than one contributor makes changes to the same dbt project in dbt Cloud in short succession, the later arriving CI check no longer has to wait for the first check to complete. Both checks will execute concurrently. * **Automatically cancel stale CI runs**. If you push multiple commits to the same PR, dbt will cancel older, now-out-of-date CI checks automatically. No resources wasted on checking stale code. * **Run CI checks without blocking production runs**. CI checks will no longer consume run slots, meaning you can have as many CI checks running as you want, without impeding your production jobs. To learn more, refer to [Continuous integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md) and [CI jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md). *  Admin API dbt Labs updated the docs for the [dbt Cloud Administrative API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md) and they are now available for both [v2](https://docs.getdbt.com/dbt-cloud/api-v2#/) and [v3](https://docs.getdbt.com/dbt-cloud/api-v3#/). * Now using Spotlight for improved UI and UX. * All endpoints are now documented for v2 and v3. Added automation to the docs so they remain up to date. * Documented many of the request and response bodies. * You can now test endpoints directly from within the API docs. And, you can choose which [regional server](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) to use (North America, APAC, or EMEA). * With the new UI, you can more easily generate code for any endpoint. *  Product docs updates Hello from the dbt Docs team: @mirnawong1, @matthewshaver, @nghi-ly, and @runleonarun! First, we’d like to thank the 17 new community contributors to docs.getdbt.com — ✨ @aaronbini, @sjaureguimodo, @aranke, @eiof, @tlochner95, @mani-dbt, @iamtodor, @monilondo, @vrfn, @raginjason, @AndrewRTsao, @MitchellBarker, @ajaythomas, @smitsrr, @leoguyaux, @GideonShils, @michaelmherrera! Here's what's new to [docs.getdbt.com](http://docs.getdbt.com/) in June: #### ☁ Cloud projects * We clarified the nuances of [CI and CI jobs](https://docs.getdbt.com/docs/deploy/continuous-integration.md), updated the [Scheduler content](https://docs.getdbt.com/docs/deploy/job-scheduler.md), added two new pages for the job settings and run visibility, moved the project state page to the [Syntax page](https://docs.getdbt.com/reference/node-selection/syntax.md), and provided a landing page for [Deploying with Cloud](https://docs.getdbt.com/docs/deploy/jobs.md) to help readers navigate the content better. * We reformatted the [Supported data platforms page](https://docs.getdbt.com/docs/supported-data-platforms.md) by adding dbt Cloud to the page, splitting it into multiple pages, using cards to display verified adapters, and moving the [Warehouse setup pages](https://docs.getdbt.com/docs/local/connect-data-platform/about-dbt-connections.md) to the Docs section. * We launched a new [Lint and format page](https://docs.getdbt.com/docs/platform/studio-ide/lint-format.md), which highlights the awesome new dbt Cloud IDE linting/formatting function. * We enabled a connection between [dbt Cloud release notes](https://docs.getdbt.com/docs/dbt-versions/release-notes.md) and the dbt Slack community. This means new dbt Cloud release notes are automatically sent to the slack community [#dbt-cloud channel](https://getdbt.slack.com/archives/CMZ2V0X8V) via RSS feed, keeping users up to date with changes that may affect them. * We’ve added two new docs links in the dbt Cloud Job settings user interface (UI). This will provide additional guidance and help users succeed when setting up a dbt Cloud job: [job commands](https://docs.getdbt.com/docs/deploy/job-commands.md) and job triggers. * We added information related to the newly created [IT license](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#license-based-access-control), available for Team and Enterprise plans. * We added a new [Supported browser page](https://docs.getdbt.com/docs/platform/about-platform/browsers.md), which lists the recommended browsers for dbt Cloud. * We launched a new page informing users of [new Experimental features option](https://docs.getdbt.com/docs/dbt-versions/experimental-features.md) in dbt Cloud. * We worked with dbt Engineering to help publish new beta versions of the dbt [dbt Cloud Administrative API docs](https://docs.getdbt.com/docs/dbt-apis/admin-api.md). #### 🎯 Core projects * We launched the new [MetricFlow docs](https://docs.getdbt.com/docs/build/build-metrics-intro.md) on dbt Core v1.6 beta. * Split [Global configs](https://docs.getdbt.com/reference/global-configs/about-global-configs.md) into individual pages, making it easier to find, especially using search. #### New 📚 Guides, ✏️ blog posts, and FAQs * Add an Azure DevOps example in the [Customizing CI/CD with custom pipelines](https://docs.getdbt.com/guides/custom-cicd-pipelines.md) guide. #### May 2023 *  dbt Cloud IDE To continue improving your [Cloud IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) development experience, the dbt Labs team continues to work on adding new features, fixing bugs, and increasing reliability ✨. Stay up-to-date with [IDE-related changes](https://docs.getdbt.com/tags/ide.md). #### New features * Lint via SQL Fluff is now available in beta (GA over the next 2-3 weeks) * Format markdown files with prettier * Leverage developer experience shortcuts, including ``Ctrl + ` ``(toggle history drawer), `CMD + Option + /` (toggle block comment), `CMD + Shift + P` (open command palette), `Option + W` (close editor tab) * Display parent folder name for files with same name in Changes section * Navigate the new IDE features quickly using [the IDE User Interface](https://docs.getdbt.com/docs/platform/studio-ide/ide-user-interface.md) help page * Use `top X` in SQL when previewing in the IDE * Opt into the new IDE backend layer over the past month (still with dbt-rpc). Ready for beta later in June! #### Product refinements * Performance-related upgrades: * Reduced cold start time by 60+% * Improved render time of modals in the IDE by 98% * Improved IDE performance with dbt Core v1.5+ (faster and snappier – highly encourage you to [upgrade your dbt version](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md)!) * Upgraded sqlfmt (which powers the Format button) to 0.18.0 * Updated Build button to change menu options based on file/model type (snapshot, macro, etc.) * Display message to disable adblocker for file contents error * Moved Format button to console bar * Made many security enhancements in the IDE #### Bug fixes * File icon sizes no longer get wonky in small screen * Toast notifications no longer take over command bar menu * Hover info inside the text editor no longer gets cut off * Transition between a file and a recently modified scratchpad no longer triggers a console error * dbt v1.5+ now can access the IDE * Confirm button on the Unsaved Changes modal now closes after clicking it * Long node names no longer overflow in the parsed logs section in history drawer * Status pill in history drawer no longer scales with longer command * Tooltip for tab name with a long file name is no longer cut off * Lint button should no longer available in main branch *  Run history improvements New usability and design improvements to the **Run History** dashboard in dbt Cloud are now available. These updates allow people to discover the information they need more easily by reducing the number of clicks, surfacing more relevant information, keeping people in flow state, and designing the look and feel to be more intuitive to use. Highlights include: * Usability improvements for CI runs with hyperlinks to the branch, PR, and commit SHA, along with more discoverable temporary schema names * Preview of runs' error messages on hover * Hyperlinks to the environment * Better iconography on run status * Clearer run trigger cause (API, scheduled, pull request, triggered by user) * More details on the schedule time on hover * Run timeout visibility dbt Labs is making a change to the metadata retrieval policy for Run History in dbt Cloud. **Beginning June 1, 2023,** developers on the dbt multi-tenant application will be able to self-serve access to their account’s run history through the dbt user interface (UI) and API for only 365 days, on a rolling basis. Older run history will be available for download by reaching out to Customer Support. We're seeking to minimize the amount of metadata we store while maximizing application performance. Specifically, all `GET` requests to the dbt Cloud [Runs endpoint](https://docs.getdbt.com/dbt-cloud/api-v2#/operations/List%20Runs) will return information on runs, artifacts, logs, and run steps only for the past 365 days. Additionally, the run history displayed in the dbt Cloud UI will only show runs for the past 365 days. [![The dbt Cloud UI displaying a Run History](/img/docs/dbt-platform/rn-run-history.jpg?v=2 "The dbt Cloud UI displaying a Run History")](#)The dbt Cloud UI displaying a Run History We will retain older run history in cold storage and can make it available to customers who reach out to our Support team. To request older run history info, contact the Support team at <support@getdbt.com> or use the dbt Cloud application chat by clicking the `?` icon in the dbt Cloud UI. *  Run details and log improvements New usability and design improvements to the run details and logs in dbt Cloud are now available. The ability to triage errors in logs is a big benefit of using dbt Cloud's job and scheduler functionality. The updates help make the process of finding the root cause much easier. Highlights include: * Surfacing a warn state on a run step * Search in logs * Easier discoverability of errors and warnings in logs * Lazy loading of logs, making the whole run details page load faster and feel more performant * Cleaner look and feel with iconography * Helpful tool tips *  Product docs updates Hello from the dbt Docs team: @mirnawong1, @matthewshaver, @nghi-ly, and @runleonarun! First, we’d like to thank the 13 new community contributors to docs.getdbt.com! Here's what's new to [docs.getdbt.com](http://docs.getdbt.com/) in May: #### 🔎 Discoverability * We made sure everyone knows that Cloud-users don’t need a [profiles.yml file](https://docs.getdbt.com/docs/local/profiles.yml.md) by adding a callout on several key pages. * Fleshed out the [model Jinja variable page](https://docs.getdbt.com/reference/dbt-jinja-functions/model.md), which originally lacked conceptual info and didn’t link to the schema page. * Added a new [Quickstarts landing page](https://docs.getdbt.com/guides.md). This new format sets up for future iterations that will include filtering! But for now, we are excited you can step through quickstarts in a focused way. #### Cloud projects * We launched [dbt Cloud IDE user interface doc](https://docs.getdbt.com/docs/platform/studio-ide/ide-user-interface.md), which provides a thorough walkthrough of the IDE UI elements and their definitions. * Launched a sparkling new [dbt Cloud Scheduler page](https://docs.getdbt.com/docs/deploy/job-scheduler.md) ✨! We went from previously having little content around the scheduler to a subsection that breaks down the awesome scheduler features and how it works. * Updated the [dbt Cloud user license page](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md#licenses) to clarify how to add or remove cloud users. * Shipped these Discovery API docs to coincide with the launch of the Discovery API: * [About the Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md) * [Use cases and examples for the Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-use-cases-and-examples.md) * [Query the Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-querying.md) #### 🎯 Core projects * See what’s coming up [in Core v 1.6](https://github.com/dbt-labs/docs.getdbt.com/issues?q=is%3Aissue+label%3A%22dbt-core+v1.6%22)! * We turned the `profiles.yml` [page](https://docs.getdbt.com/docs/local/profiles.yml.md) into a landing page, added more context to profiles.yml page, and moved the ‘About CLI’ higher up in the `Set up dbt` section. #### New 📚 Guides, ✏️ blog posts, and FAQs If you want to contribute to a blog post, we’re focusing on content * Published a blog post: [Accelerate your documentation workflow: Generate docs for whole folders at once](https://docs.getdbt.com/blog/generating-dynamic-docs-dbt) * Published a blog post: [Data engineers + dbt v1.5: Evolving the craft for scale](https://docs.getdbt.com/blog/evolving-data-engineer-craft) * Added an [FAQ](https://docs.getdbt.com/faqs/Warehouse/db-connection-dbt-compile.md) to clarify the common question users have on *Why does dbt compile needs to connect to the database?* * Published a [discourse article](https://discourse.getdbt.com/t/how-to-configure-external-user-email-notifications-in-dbt-cloud/8393) about configuring job notifications for non-dbt Cloud users #### April 2023 *  dbt Cloud IDE #### New features * New warning message suggests you invoke `dbt deps` when it's needed (as informed by `dbt-score`). * New warning message appears when you select models but don't save them before clicking **Build** or invoking dbt (like, dbt build/run/test). * Previews of Markdown and CSV files are now available in the IDE console. * The file tree menu now includes a Duplicate File option. * Display loading time when previewing a model #### Product refinements * Enhance autocomplete experience which has performed slowly for people with large projects and who implement a limit to max `manifest.json` for this feature * Introduce pagination for invocation node summary view (displaying 100 nodes at a time) * Improve rendering for the Changes / Version Control section of the IDE * Update icons to be consistent in dbt Cloud * Add table support to the Markdown preview * Add the lineage tab back to seed resources in the IDE * Implement modal priority when there are multiple warning modals * Improve a complex command's description in the command palette #### Bug fixes * File tree no longer collapses on first click when there's a project subdirectory defined * **Revert all** button now works as expected * CSV preview no longer fails with only one column * Cursor and scroll bar location are now persistent with their positions * `git diff` view now shows just change diffs and no longer shows full diff (as if file is new) until page refreshes * ToggleMinimap Command no longer runs another Command at the same time * `git diff` view no longer shows infinite spins in specific scenarios (new file, etc.) * File contents no longer get mixed up when using diff view and one file has unsaved changes * YML lineage now renders model without tests (in dbt Core v1.5 and above) * Radio buttons for **Summary** and **Details** in the logs section now consistently update to show the accurate tab selection * IDE no longer throws the console error `Error: Illegal argument` and redirects to the `Something went wrong` page *  API updates Starting May 15, 2023, we will support only the following `order_by` functionality for the List Runs endpoint: * `id` and `-id` * `created_at` and `-created_at` * `finished_at` and `-finished_at` We recommend that you change your API requests to https://\<YOUR\_ACCESS\_URL>/api/v2/accounts/{accountId}/runs/ to use a supported `order_by` before this date. Access URLs dbt Cloud is hosted in multiple regions around the world, and each region has a different access URL. Users on Enterprise plans can choose to have their account hosted in any one of these regions. For a complete list of available dbt Cloud access URLs, refer to [Regions & IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). For more info, refer to our [documentation](https://docs.getdbt.com/dbt-cloud/api-v2#/operations/List%20Runs). *  Scheduler optimization The dbt Cloud Scheduler now prevents queue clog by canceling unnecessary runs of over-scheduled jobs. The duration of a job run tends to grow over time, usually caused by growing amounts of data in the warehouse. If the run duration becomes longer than the frequency of the job’s schedule, the queue will grow faster than the scheduler can process the job’s runs, leading to a runaway queue with runs that don’t need to be processed. Previously, when a job was in this over-scheduled state, the scheduler would stop queuing runs after 50 were already in the queue. This led to a poor user experience where the scheduler canceled runs indiscriminately. You’d have to log into dbt Cloud to manually cancel all the queued runs and change the job schedule to "unclog" the scheduler queue. Now, the dbt Cloud scheduler detects when a scheduled job is set to run too frequently and appropriately cancels runs that don’t need to be processed. Specifically, scheduled jobs can only ever have one run of the job in the queue, and if a more recent run gets queued, the early queued run will get canceled with a helpful error message. Users will still need to either refactor the job so it runs faster or change the job schedule to run less often if the job often gets into an over-scheduled state. *  Starburst adapter GA The Starburst (Trino compatible) connection is now generally available in dbt Cloud. This means you can now use dbt Cloud to connect with Starburst Galaxy, Starburst Enterprise, and self-hosted Trino. This feature is powered by the [`dbt-trino`](https://github.com/starburstdata/dbt-trino) adapter. To learn more, check out our Quickstart guide for [dbt Cloud and Starburst Galaxy](https://docs.getdbt.com/guides/starburst-galaxy.md). *  Product docs updates Hello from the dbt Docs team: @mirnawong1, @matthewshaver, @nghi-ly, and @runleonarun! We want to share some highlights introduced to docs.getdbt.com in the last month: #### 🔎 Discoverability * [API docs](https://docs.getdbt.com/docs/dbt-apis/overview.md) now live in the left sidebar to improve discoverability. * [The deploy dbt jobs sidebar](https://docs.getdbt.com/docs/deploy/deployments.md) has had a glow up 💅 that splits the ‘about deployment’ into two paths (deploy w dbt cloud and deploy w other tools), adds more info about the dbt cloud scheduler, its features, and how to create a job, adds ADF deployment guidance. We hope the changes improve the user experience and provide users with guidance when deploying with other tools. #### ☁ Cloud projects * Added Starburst/Trino adapter docs, including: - [dbt Cloud quickstart guide](https://docs.getdbt.com/guides/starburst-galaxy.md),  - [connection page](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-starburst-trino.md),  - [set up page](https://docs.getdbt.com/docs/local/connect-data-platform/trino-setup.md), and [config page](https://docs.getdbt.com/reference/resource-configs/trino-configs.md). * Enhanced [dbt Cloud jobs page](https://docs.getdbt.com/docs/deploy/jobs.md) and section to include conceptual info on the queue time, improvements made around it, and about failed jobs. * Check out the April dbt [Cloud release notes](https://docs.getdbt.com/docs/dbt-versions/release-notes.md) #### 🎯 Core projects * Clearer descriptions in the [Jinja functions page](https://docs.getdbt.com/reference/dbt-jinja-functions-context-variables.md), that improve content for each card.  * [1.5 Docs](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.5.md>) have been released as a Release Candidate (RC)! * See the beautiful [work captured in Core v 1.5](https://github.com/dbt-labs/docs.getdbt.com/issues?q=is%3Aissue+label%3A%22dbt-core+v1.5%22+is%3Aclosed). #### New 📚 Guides and ✏️ blog posts * [Use Databricks workflows to run dbt Cloud jobs](https://docs.getdbt.com/guides/databricks-workflows.md) * [Refresh Tableau workbook with extracts after a job finishes](https://docs.getdbt.com/guides/zapier-refresh-tableau-workbook.md) * [dbt Python Snowpark workshop/tutorial](https://docs.getdbt.com/guides/dbt-python-snowpark.md) * [How to optimize and troubleshoot dbt Models on Databricks](https://docs.getdbt.com/guides/optimize-dbt-models-on-databricks.md) * [The missing guide to debug() in dbt](https://docs.getdbt.com/blog/guide-to-jinja-debug) * [dbt Squared: Leveraging dbt Core and dbt Cloud together at scale](https://docs.getdbt.com/blog/dbt-squared) * [Audit\_helper in dbt: Bringing data auditing to a higher level](https://docs.getdbt.com/blog/audit-helper-for-migration) #### March 2023 *  dbt v1.0 deprecation dbt Cloud now requires dbt version 1.0 or later. As of March 1, 2023, we removed all instances of older dbt versions from dbt Cloud. Any environments or jobs configured with a dbt version lower than 1.0 were automatically updated to dbt v1.4, which is the latest minor version available on dbt Cloud. For more info on dbt versions, releases, and dbt Cloud support timeline, refer to [About dbt Core versions](https://docs.getdbt.com/docs/dbt-versions.md#latest-releases). Refer to some additional info and resources to help you upgrade your dbt version: * [How to upgrade dbt without fear](https://docs.getdbt.com/blog/upgrade-dbt-without-fear) * [Upgrade Q\&A on breaking changes](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md#upgrading-legacy-versions-under-10) * [Version migration guides](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade.md) *  dbt Cloud IDE To continue improving your [Cloud IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) development experience, the dbt Labs team continue to work on adding new features, fixing bugs, and increasing reliability ✨. Read more about the [upcoming improvements to the Cloud IDE](https://www.getdbt.com/blog/improvements-to-the-dbt-cloud-ide/) and stay up-to-date with [IDE-related changes](https://docs.getdbt.com/tags/ide.md). #### New features * Commit and revert individual files under **Version Control**. * Use the [command palette](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md#studio-ide-features) to invoke common complex dbt commands, such as resuming from the last failure. * Create PRs even when there are uncommitted changes (under the **git** dropdown). * The IDE will display more autocomplete suggestions when editing a YML file, powered by [dbt-jsonschema](https://github.com/dbt-labs/dbt-jsonschema). * The file tree now has additional options in the right-click menu, such as Copy model as ref or Copy file path. * The DAG view has been adjusted to a default of `2+model+2`. * A lineage selector has been implemented in the DAG/lineage sub-tab. * Edit directly in the git diff view located in the right pane. * A warning message will now appear when users press Command-W/Control-W when there are unsaved changes. * A new onboarding flow guide is now available. #### Product refinements * The DAG selector now uses `name` instead of `file_uri` to build selectors. * The DAG is now vertically centered under the new Selector Input element * sqlfmt has been upgraded to v0.17.0. * When the Format button fails, a toast notification will display a syntax error. * The editor now has the option to toggle minimap/word-wrap via right-click. * The history drawer displays elapsed time in real-time and s/m/h increments. * When deleting development environments, the delete modal will now warn users that any uncommitted changes will be lost. * The context for the Git button has been adjusted to show that it will link to an external site (such as GitHub or GitLab) when users create a pull request. #### Bug fixes * The IDE now displays an error message when the git repository is not reachable. Previously, it failed silently. * The kebab menu is now visible when the invocation history drawer is open. Previously, it wasn't showing. * DAGs are now updated/populated consistently. Previously, it occasionally failed. * The purple highlight for DAG selection is now consistent across files. Previously, it was inconsistent. * Users can now rename files back to their original name. Previously, this wasn't possible. * The link to the IDE from the project setup page has been corrected. * The IDE no longer has issues with single-space file names. * Adding invalid characters in the sub-directory config no longer causes the IDE to fail. * YML autocomplete triggers consistently now. Previously, it occasionally didn't trigger. * Reverting single files now reloads the file contents in the tab. Previously, it didn't reload. * The file tree no longer collapses on the first click when there is a project subdirectory defined. *  API updates To make the API more scalable and reliable, we've implemented a maximum limit of `100` for all API requests to our `list` endpoints. If API requests exceed the maximum limit parameter of `100`, a user will receive an API error message. This maximum limit applies to [multi-tenant instances](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) only, and *does not* apply to single tenant instances. Refer to the [Pagination](https://docs.getdbt.com/dbt-cloud/api-v2#/) section of the overview for more information on this change. #### Feb 2023 *  Disable partial parsing in job commands You can now use the `--no-partial-parse` flag to disable partial parsing in your dbt Cloud job commands.  Previously, the [`--no-partial-parse` global config](https://docs.getdbt.com/reference/global-configs/parsing.md) was only available in dbt Core. For more information, refer to [partial parsing](https://docs.getdbt.com/reference/parsing.md#partial-parsing). *  dbt Cloud IDE To continue improving our [Cloud IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) experience, the dbt Labs team worked on fixing bugs, increasing reliability, and adding new features ✨. Learn more about the [February changes](https://getdbt.slack.com/archives/C03SAHKKG2Z/p1677605383451109). #### New features * Support for custom node colors in the IDE DAG visualization * Ref autocomplete includes models from seeds and snapshots * Prevent menus from getting cropped (git controls dropdown, file tree dropdown, build button, editor tab options) * Additional option to access the file menu by right-clicking on the files and folders in the file tree * Rename files by double-clicking on files in the file tree and the editor tabs * Right-clicking on file tabs has new options and will now open at your cursor instead of in the middle of the tab * The git branch name above **Version Control** links to the repo for specific git providers * Currently available for all [multi-tenant](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) instances using GitHub or GitLab providers #### Product refinements * Added an error modal for RPC parsing errors when users attempt to invoke dbt commands (preview, compile, or general dbt invocations) * Enabled syntax highlighting for Jinja expression and statement delimiters * Clarified and renamed the options under the **Build** button * Changed the term for RPC status from `Compiling` to `Parsing` to match dbt-core construct * Implemented a new File Tree component to improve render time by 60% * Disabled the Local Storage of File Tree to prevent users from running into max LocalStorage issue for large projects * Changed snapshot snippet template (`__snapshot`) to a select from source #### Bug fixes * You no longer have file contents carrying over when you switch to a different project that has the same file name * The preview max limit no longer allows you to override the maximum * You no longer encounter node statuses failing to update in the history drawer for those on version 1.4 core. (This is a partial fix that may be fully addressed by core version 1.5) * You can now use the **Copy File Name** option to copy up to the last dot, rather than the first dot * You can now use the `--no-partial-parse` flag to disable partial parsing in your dbt Cloud job commands.  * Previously, the [`--no-partial-parse` global config](https://docs.getdbt.com/reference/global-configs/parsing.md) was only available in dbt Core. For more information, refer to [partial parsing](https://docs.getdbt.com/reference/parsing.md#partial-parsing). #### January 2023 *  dbt Cloud IDE In the spirit of continuing to improve our [Cloud IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) experience, the dbt Labs team worked on fixing bugs, increasing reliability, and adding new features ✨. Learn more about the [January changes](https://getdbt.slack.com/archives/C03SAHKKG2Z/p1675272600286119) and what's coming soon. #### New features * Improved syntax highlighting within the IDE for better Jinja-SQL combination (double quotes now show proper syntax highlight!) * Adjusted the routing URL for the IDE page and removed the `next` from the URL * Added a *new* easter egg within the IDE 🐶🦆 #### Product refinements * Performance improvements and reduced IDE slowness. The IDE should feel faster and snappier. * Reliability improvements – Improved error handling that previously put IDE in a bad state * Corrected the list of dropdown options for the Build button * Adjusted startup page duration * Added code snippets for `unique` and `not_null` tests for YAML files * Added code snippets for metrics based on environment dbt versions * Changed “commit and push” to “commit and sync” to better reflect the action * Improved error message when saving or renaming files to duplicate names #### Bug fixes * You no longer arbitrarily encounter an `RPC server got an unknown async ID` message * You can now see the build button dropdown, which had been hidden behind the placeholder DAG screen * You can now close toast notifications for command failure when the history drawer is open * You no longer encounter a `Something went wrong` message when previewing a model * You can now see repository status in the IDE, and the IDE finds the SSH folder * Scroll bars and download CSV no longer flicker within the preview pane --- ### 2024 dbt platform release notes dbt platform | Free dbt release notes for recent and historical changes. Release notes fall into one of the following categories: * **New:** New products and features * **Enhancement:** Performance improvements and feature enhancements * **Fix:** Bug and security fixes * **Behavior change:** A change to existing behavior that doesn't fit into the other categories, such as feature deprecations or changes to default settings Release notes are grouped by month for both multi-tenant and virtual private cloud (VPC)\* environments \* The official release date for this new format of release notes is May 15th, 2024. Historical release notes for prior dates may not reflect all available features released earlier this year or their tenancy availability. #### December 2024 * **New**: Saved queries now support [tags](https://docs.getdbt.com/reference/resource-configs/tags.md), which allow you to categorize your resources and filter them. Add tags to your [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md) in the `semantic_model.yml` file or `dbt_project.yml` file. For example: dbt\_project.yml ```yml saved-queries: jaffle_shop: customer_order_metrics: +tags: order_metrics ``` * **New**: [Dimensions](https://docs.getdbt.com/reference/resource-configs/meta.md) now support the `meta` config property in [dbt Cloud **Latest** release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) and from dbt Core 1.9. You can add metadata to your dimensions to provide additional context and information about the dimension. Refer to [meta](https://docs.getdbt.com/reference/resource-configs/meta.md) for more information. * **New**: [Downstream exposures](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md) are now generally available to dbt Enterprise plans. Downstream exposures integrate natively with Tableau (Power BI coming soon) and auto-generate downstream lineage in dbt Explorer for a richer experience. * **New**: The Semantic Layer supports Sigma as a [partner integration](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md), available in Preview. Refer to [Sigma](https://help.sigmacomputing.com/docs/configure-a-dbt-semantic-layer-integration) for more information. * **New**: The Semantic Layer now supports Azure Single-tenant deployments. Refer to [Set up the Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md) for more information on how to get started. * **Fix**: Resolved intermittent issues in Single-tenant environments affecting Semantic Layer and query history. * **Fix**: [The dbt Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) now respects the BigQuery [`execution_project` attribute](https://docs.getdbt.com/docs/local/connect-data-platform/bigquery-setup.md#execution-project), including for exports. * **New**: [Model notifications](https://docs.getdbt.com/docs/deploy/model-notifications.md) are now generally available in dbt. These notifications alert model owners through email about any issues encountered by models and tests as soon as they occur while running a job. * **New**: You can now use your [Azure OpenAI key](https://docs.getdbt.com/docs/platform/account-integrations.md?ai-integration=azure#ai-integrations) (available in beta) to use dbt features like [](https://docs.getdbt.com/docs/platform/wizard-platform.md). Additionally, you can use your own [OpenAI API key](https://docs.getdbt.com/docs/platform/account-integrations.md?ai-integration=openai#ai-integrations) or use [dbt Labs-managed OpenAI](https://docs.getdbt.com/docs/platform/account-integrations.md?ai-integration=dbtlabs#ai-integrations) key. Refer to [AI integrations](https://docs.getdbt.com/docs/platform/account-integrations.md#ai-integrations) for more information. * **New**: The [`hard_deletes`](https://docs.getdbt.com/reference/resource-configs/hard-deletes.md) config gives you more control on how to handle deleted rows from the source. Supported options are `ignore` (default), `invalidate` (replaces the legacy `invalidate_hard_deletes=true`), and `new_record`. Note that `new_record` will create a new metadata column in the snapshot table. #### November 2024 * **Enhancement**: Data health signals in dbt Explorer are now available for Exposures, providing a quick view of data health while browsing resources. To view trust signal icons, go to dbt Explorer and click **Exposures** under the **Resource** tab. Refer to [Data health signals for resources](https://docs.getdbt.com/docs/explore/data-health-signals.md) for more info. * **Bug**: Identified and fixed an error with Semantic Layer queries that take longer than 10 minutes to complete. * **Fix**: Job environment variable overrides in credentials are now respected for Exports. Previously, they were ignored. * **Behavior change**: If you use a custom microbatch macro, set a [`require_batched_execution_for_custom_microbatch_strategy` behavior flag](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_batched_execution_for_custom_microbatch_strategy.md) in your `dbt_project.yml` to enable batched execution. If you don't have a custom microbatch macro, you don't need to set this flag as dbt will handle microbatching automatically for any model using the [microbatch strategy](https://docs.getdbt.com/docs/build/incremental-microbatch.md#how-microbatch-compares-to-other-incremental-strategies). * **Enhancement**: For users that have Advanced CI's [compare changes](https://docs.getdbt.com/docs/deploy/advanced-ci.md#compare-changes) feature enabled, you can optimize performance when running comparisons by using custom dbt syntax to customize deferral usage, exclude specific large models (or groups of models with tags), and more. Refer to [Compare changes custom commands](https://docs.getdbt.com/docs/deploy/job-commands.md#compare-changes-custom-commands) for examples of how to customize the comparison command. * **New**: SQL linting in CI jobs is now generally available in dbt. You can enable SQL linting in your CI jobs, using [SQLFluff](https://sqlfluff.com/), to automatically lint all SQL files in your project as a run step before your CI job builds. SQLFluff linting is available on [dbt release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) and to dbt [Team or Enterprise](https://www.getdbt.com/pricing/) accounts. Refer to [SQL linting](https://docs.getdbt.com/docs/deploy/continuous-integration.md#sql-linting) for more information. * **New**: Use the [`dbt_valid_to_current`](https://docs.getdbt.com/reference/resource-configs/dbt_valid_to_current.md) config to set a custom indicator for the value of `dbt_valid_to` in current snapshot records (like a future date). By default, this value is `NULL`. When configured, dbt will use the specified value instead of `NULL` for `dbt_valid_to` for current records in the snapshot table. This feature is available in [the dbt Cloud **Latest** release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) (formerly called `Versionless`) and dbt Core v1.9 and later. * **New**: Use the [`event_time`](https://docs.getdbt.com/reference/resource-configs/event-time.md) configuration to specify "at what time did the row occur." This configuration is required for [Incremental microbatch](https://docs.getdbt.com/docs/build/incremental-microbatch.md) and can be added to ensure you're comparing overlapping times in [Advanced CI's compare changes](https://docs.getdbt.com/docs/deploy/advanced-ci.md). Available in [the dbt Cloud **Latest** release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) (formerly called `Versionless`) and dbt Core v1.9 and higher. * **Fix**: This update improves [Semantic Layer Tableau integration](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/tableau.md) making query parsing more reliable. Some key fixes include: * Error messages for unsupported joins between saved queries and ALL tables. * Improved handling of queries when multiple tables are selected in a data source. * Fixed a bug when an IN filter contained a lot of values. * Better error messaging for queries that can't be parsed correctly. * **Enhancement**: The Semantic Layer supports creating new credentials for users who don't have permissions to create service tokens. In the **Credentials & service tokens** side panel, the **+Add Service Token** option is unavailable for those users who don't have permission. Instead, the side panel displays a message indicating that the user doesn't have permission to create a service token and should contact their administration. Refer to [Set up Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md) for more details. #### October 2024  Coalesce 2024 announcements Documentation for new features and functionality announced at Coalesce 2024: * Iceberg table support for [Snowflake](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#iceberg-table-format) * [Athena](https://docs.getdbt.com/reference/resource-configs/athena-configs.md) and [Teradata](https://docs.getdbt.com/reference/resource-configs/teradata-configs.md) adapter support in dbt Cloud * dbt Cloud now hosted on [Azure](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) * Get comfortable with [dbt Cloud Release Tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) that keep your project up-to-date, automatically — on a cadence appropriate for your team * Scalable [microbatch incremental models](https://docs.getdbt.com/docs/build/incremental-microbatch.md) * Advanced CI [features](https://docs.getdbt.com/docs/deploy/advanced-ci.md) * [Linting with CI jobs](https://docs.getdbt.com/docs/deploy/continuous-integration.md#sql-linting) * dbt Assist is now [dbt Copilot](https://docs.getdbt.com/docs/platform/wizard-platform.md) * Developer blog on [Snowflake Feature Store and dbt: A bridge between data pipelines and ML](https://docs.getdbt.com/blog/snowflake-feature-store) * [Downstream exposures with Tableau](https://docs.getdbt.com/docs/explore/view-downstream-exposures.md) * Semantic Layer integration with [Excel desktop and M365](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/excel.md) * [Data health tiles](https://docs.getdbt.com/docs/explore/data-tile.md) * [Semantic Layer and Cloud IDE integration](https://docs.getdbt.com/docs/build/metricflow-commands.md#metricflow-commands) * Query history in [Explorer](https://docs.getdbt.com/docs/explore/model-query-history.md#view-query-history-in-explorer) * Semantic Layer Metricflow improvements, including [improved granularity and custom calendar](https://docs.getdbt.com/docs/build/metricflow-time-spine.md#custom-calendar) * [Python SDK](https://docs.getdbt.com/docs/dbt-apis/sl-python.md) is now generally available - **Behavior change:** [Multi-factor authentication](https://docs.getdbt.com/docs/platform/manage-access/mfa.md) is now enforced on all users who log in with username and password credentials. - **Enhancement**: The dbt Semantic Layer JDBC now allows users to paginate `semantic_layer.metrics()` and `semantic_layer.dimensions()` for metrics and dimensions using `page_size` and `page_number` parameters. Refer to [Paginate metadata calls](https://docs.getdbt.com/docs/dbt-apis/sl-jdbc.md#querying-the-api-for-metric-metadata) for more information. - **Enhancement**: The dbt Semantic Layer JDBC now allows you to filter your metrics to include only those that contain a specific substring, using the `search` parameter. If no substring is provided, the query returns all metrics. Refer to [Fetch metrics by substring search](https://docs.getdbt.com/docs/dbt-apis/sl-jdbc.md#querying-the-api-for-metric-metadata) for more information. - **Fix**: The [Semantic Layer Excel integration](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/excel.md) now correctly surfaces errors when a query fails to execute. Previously, it was not clear why a query failed to run. - **Fix:** Previously, POST requests to the Jobs API with invalid `cron` strings would return HTTP response status code 500s but would update the underlying entity. Now, POST requests to the Jobs API with invalid `cron` strings will result in status code 400s, without the underlying entity being updated. - **Fix:** Fixed an issue where the `Source` view page in dbt Explorer did not correctly display source freshness status if older than 30 days. - **Fix:** The UI now indicates when the description of a model is inherited from a catalog comment. - **Behavior change:** User API tokens have been deprecated. Update to [personal access tokens](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) if you have any still in use. - **New**: The Cloud IDE supports signed commits for Git, available for Enterprise plans. You can sign your Git commits when pushing them to the repository to prevent impersonation and enhance security. Supported Git providers are GitHub and GitLab. Refer to [Git commit signing](https://docs.getdbt.com/docs/platform/studio-ide/git-commit-signing.md) for more information. - **New:** With Mesh, you can now enable bidirectional dependencies across your projects. Previously, dbt enforced dependencies to only go in one direction. dbt checks for cycles across projects and raises errors if any are detected. For details, refer to [Cycle detection](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#cycle-detection). There's also the [Intro to Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md) guide to help you learn more best practices. - **New**: The [Semantic Layer Python software development kit](https://docs.getdbt.com/docs/dbt-apis/sl-python.md) is now [generally available](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles.md). It provides users with easy access to the Semantic Layer with Python and enables developers to interact with the Semantic Layer APIs to query metrics/dimensions in downstream tools. - **Enhancement**: You can now add a description to a singular data test. Use the [`description` property](https://docs.getdbt.com/reference/resource-properties/description.md) to document [singular data tests](https://docs.getdbt.com/docs/build/data-tests.md#singular-data-tests). You can also use [docs block](https://docs.getdbt.com/docs/build/documentation.md#using-docs-blocks) to capture your test description. The enhancement is available now in [the **Latest** release track in dbt Cloud](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md), and it will be included in dbt Core v1.9. - **New**: Introducing the [microbatch incremental model strategy](https://docs.getdbt.com/docs/build/incremental-microbatch.md) (beta), available now in [dbt Cloud Latest](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) and will soon be supported in dbt Core v1.9. The microbatch strategy allows for efficient, batch-based processing of large time-series datasets for improved performance and resiliency, especially when you're working with data that changes over time (like new records being added daily). To enable this feature in dbt Cloud, set the `DBT_EXPERIMENTAL_MICROBATCH` environment variable to `true` in your project. - **New**: The dbt Semantic Layer supports custom calendar configurations in MetricFlow, available in [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles.md). Custom calendar configurations allow you to query data using non-standard time periods like `fiscal_year` or `retail_month`. Refer to [custom calendar](https://docs.getdbt.com/docs/build/metricflow-time-spine.md#custom-calendar) to learn how to define these custom granularities in your MetricFlow timespine YAML configuration. - **New**: In the **Latest** release track in dbt, [Snapshots](https://docs.getdbt.com/docs/build/snapshots.md) have been updated to use YAML configuration files instead of SQL snapshot blocks. This new feature simplifies snapshot management and improves performance, and will soon be released in dbt v1 version 1.9. * Who does this affect? Users of the **Latest** release track in dbt can define snapshots using the new YAML specification. Users upgrading to **Latest** who have existing snapshot definitions can keep their existing configurations, or they can choose to migrate their snapshot definitions to YAML. * Users on older versions: No action is needed; existing snapshots will continue to work as before. However, we recommend upgrading to the **Latest** release track to take advantage of the new snapshot features. - **Behavior change:** Set [`state_modified_compare_more_unrendered_values`](https://docs.getdbt.com/reference/global-configs/behavior-flags/state_modified_compare_more_unrendered_values.md) to true to reduce false positives for `state:modified` when configs differ between `dev` and `prod` environments. - **Behavior change:** Set the [`skip_nodes_if_on_run_start_fails`](https://docs.getdbt.com/reference/global-configs/behavior-flags/skip_nodes_if_on_run_start_fails.md) flag to `True` to skip all selected resources from running if there is a failure on an `on-run-start` hook. - **Enhancement**: In the **Latest** release track in dbt Cloud, snapshots defined in SQL files can now use `config` defined in `schema.yml` YAML files. This update resolves the previous limitation that required snapshot properties to be defined exclusively in `dbt_project.yml` and/or a `config()` block within the SQL file. This will also be released in dbt Core 1.9. - **New**: In the **Latest** release track in dbt Cloud, the `snapshot_meta_column_names` config allows for customizing the snapshot metadata columns. This feature allows an organization to align these automatically-generated column names with their conventions, and will be included in the upcoming dbt Core 1.9 release. - **Enhancement**: the **Latest** release track in dbt Cloud infers a model's `primary_key` based on configured data tests and/or constraints within `manifest.json`. The inferred `primary_key` is visible in dbt Explorer and utilized by the dbt Cloud [compare changes](https://docs.getdbt.com/docs/deploy/run-visibility.md#compare-tab) feature. This will also be released in dbt Core 1.9. Read about the [order dbt infers columns can be used as primary key of a model](https://github.com/dbt-labs/dbt/blob/7940ad5c7858ff11ef100260a372f2f06a86e71f/core/dbt/contracts/graph/nodes.py#L534-L541). - **New:** dbt Explorer now includes trust signal icons, which is currently available as a [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles.md). Trust signals offer a quick, at-a-glance view of data health when browsing your dbt models in dbt Explorer. These icons indicate whether a model is **Healthy**, **Caution**, **Degraded**, or **Unknown**. For accurate health data, ensure the resource is up-to-date and has had a recent job run. Refer to [Data health signals](https://docs.getdbt.com/docs/explore/data-health-signals.md) for more information. - **New:** Downstream exposures are now available in Preview in dbt. Downstream exposures helps users understand how their models are used in downstream analytics tools to inform investments and reduce incidents. It imports and auto-generates exposures based on Tableau dashboards, with user-defined curation. To learn more, refer to [Downstream exposures](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md). #### September 2024 * **Fix**: MetricFlow updated `get_and_expire` to replace the unsupported `GETEX` command with a `GET` and conditional expiration, ensuring compatibility with Azure Redis 6.0. * **Enhancement**: The [dbt Semantic Layer Python SDK](https://docs.getdbt.com/docs/dbt-apis/sl-python.md) now supports `TimeGranularity` custom grain for metrics. This feature allows you to define custom time granularities for metrics, such as `fiscal_year` or `retail_month`, to query data using non-standard time periods. * **New**: Use the AI engine to generate semantic model for your models, now available in beta. automatically generates documentation, tests, and now semantic models based on the data in your model, . To learn more, refer to [](https://docs.getdbt.com/docs/platform/wizard-platform.md). * **New**: Use the new recommended syntax for [defining `foreign_key` constraints](https://docs.getdbt.com/reference/resource-properties/constraints.md) using `refs`, available in the **Latest** release track in dbt Cloud. This will soon be released in dbt Core v1.9. This new syntax will capture dependencies and works across different environments. * **Enhancement**: You can now run [Semantic Layer commands](https://docs.getdbt.com/docs/build/metricflow-commands.md) commands in the [dbt Cloud IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md). The supported commands are `dbt sl list`, `dbt sl list metrics`, `dbt sl list dimension-values`, `dbt sl list saved-queries`, `dbt sl query`, `dbt sl list dimensions`, `dbt sl list entities`, and `dbt sl validate`. * **New**: Microsoft Excel, a Semantic Layer integration, is now generally available. The integration allows you to connect to Microsoft Excel to query metrics and collaborate with your team. Available for [Excel Desktop](https://pages.store.office.com/addinsinstallpage.aspx?assetid=WA200007100\&rs=en-US\&correlationId=4132ecd1-425d-982d-efb4-de94ebc83f26) or [Excel Online](https://pages.store.office.com/addinsinstallpage.aspx?assetid=WA200007100\&rs=en-US\&correlationid=4132ecd1-425d-982d-efb4-de94ebc83f26\&isWac=True). For more information, refer to [Microsoft Excel](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/excel.md). * **New**: [Data health tile](https://docs.getdbt.com/docs/explore/data-tile.md) is now generally available in dbt Explorer. Data health tiles provide a quick at-a-glance view of your data quality, highlighting potential issues in your data. You can embed these tiles in your dashboards to quickly identify and address data quality issues in your dbt project. * **New**: dbt Explorer's Model query history feature is now in Preview for dbt Enterprise customers. Model query history allows you to view the count of consumption queries for a model based on the data warehouse's query logs. This feature provides data teams insight, so they can focus their time and infrastructure spend on the worthwhile used data products. To learn more, refer to [Model query history](https://docs.getdbt.com/docs/explore/model-query-history.md). * **Enhancement**: You can now use [Extended Attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) and [Environment Variables](https://docs.getdbt.com/docs/build/environment-variables.md) when connecting to the Semantic Layer. If you set a value directly in the Semantic Layer Credentials, it will have a higher priority than Extended Attributes. When using environment variables, the default value for the environment will be used. If you're using exports, job environment variable overrides aren't supported yet, but they will be soon. * **New:** There are two new [environment variable defaults](https://docs.getdbt.com/docs/build/environment-variables.md#dbt-cloud-context) — `DBT_CLOUD_ENVIRONMENT_NAME` and `DBT_CLOUD_ENVIRONMENT_TYPE`. * **New:** The [Amazon Athena warehouse connection](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-amazon-athena.md) is available as a public preview for dbt accounts that have upgraded to [the **Latest** release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). #### August 2024 * **Fix:** Fixed an issue in [dbt Explorer](https://docs.getdbt.com/docs/explore/explore-projects.md) where navigating to a consumer project from a public node resulted in displaying a random public model rather than the original selection. * **New**: You can now configure metrics at granularities at finer time grains, such as hour, minute, or even by the second. This is particularly useful for more detailed analysis and for datasets where high-resolution time data is required, such as minute-by-minute event tracking. Refer to [dimensions](https://docs.getdbt.com/docs/build/dimensions.md) for more information about time granularity. * **Enhancement**: Microsoft Excel now supports [saved selections](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/excel.md#using-saved-selections) and [saved queries](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/excel.md#using-saved-queries). Use Saved selections to save your query selections within the Excel application. The application also clears stale data in [trailing rows](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/excel.md#other-settings) by default. To return your results and keep any previously selected data intact, un-select the **Clear trailing rows** option. * **Behavior change:** GitHub is no longer supported for OAuth login to dbt. Use a supported [SSO or OAuth provider](https://docs.getdbt.com/docs/platform/manage-access/sso-overview.md) to securely manage access to your dbt account. #### July 2024 * **Behavior change:** `target_schema` is no longer a required configuration for [snapshots](https://docs.getdbt.com/docs/build/snapshots.md). You can now target different schemas for snapshots across development and deployment environments using the [schema config](https://docs.getdbt.com/reference/resource-configs/schema.md). * **New:** [Connections](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md#connection-management) are now available under **Account settings** as a global setting. Previously, they were found under **Project settings**. This is being rolled out in phases over the coming weeks. * **New:** Admins can now assign [environment-level permissions](https://docs.getdbt.com/docs/platform/manage-access/environment-permissions.md) to groups for specific roles. * **New:** [Merge jobs](https://docs.getdbt.com/docs/deploy/merge-jobs.md) for implementing [continuous deployment (CD)](https://docs.getdbt.com/docs/deploy/continuous-deployment.md) workflows are now GA in dbt. Previously, you had to either set up a custom GitHub action or manually build the changes every time a pull request is merged. * **New**: The ability to lint your SQL files from the dbt CLI is now available. To learn more, refer to [Lint SQL files](https://docs.getdbt.com/docs/platform/configure-dbt-cli.md#lint-sql-files). * **Behavior change:** dbt Cloud IDE automatically adds a `--limit 100` to preview queries to avoid slow and expensive queries during development. Recently, dbt Core changed how the `limit` is applied to ensure that `order by` clauses are consistently respected. Because of this, queries that already contain a limit clause might now cause errors in the IDE previews. To address this, dbt Labs plans to provide an option soon to disable the limit from being applied. Until then, dbt Labs recommends removing the (duplicate) limit clause from your queries during previews to avoid these IDE errors. * **Enhancement**: Introducing a revamped overview page for dbt Explorer, available in beta. It includes a new design and layout for the dbt Explorer homepage. The new layout provides a more intuitive experience for users to navigate their dbt projects, as well as a new **Latest updates** section to view the latest changes or issues related to project resources. To learn more, refer to [Overview page](https://docs.getdbt.com/docs/explore/explore-projects.md#overview-page). ###### dbt Semantic Layer * **New**: Introduced the [`dbt-sl-sdk` Python software development kit (SDK)](https://github.com/dbt-labs/semantic-layer-sdk-python) Python library, which provides you with easy access to the dbt Semantic Layer with Python. It allows developers to interact with the dbt Semantic Layer APIs and query metrics and dimensions in downstream tools. Refer to the [dbt Semantic Layer Python SDK](https://docs.getdbt.com/docs/dbt-apis/sl-python.md) for more information. * **New**: Introduced Semantic validations in CI pipelines. Automatically test your semantic nodes (metrics, semantic models, and saved queries) during code reviews by adding warehouse validation checks in your CI job using the `dbt sl validate` command. You can also validate modified semantic nodes to guarantee code changes made to dbt models don't break these metrics. Refer to [Semantic validations in CI](https://docs.getdbt.com/docs/deploy/ci-jobs.md#semantic-validations-in-ci) to learn about the additional commands and use cases. * **New**: We now expose the `meta` field within the [config property](https://docs.getdbt.com/reference/resource-configs/meta.md) for dbt Semantic Layer metrics in the [JDBC and GraphQL APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md) under the `meta` field. * **New**: Added a new command in the dbt CLI called `export-all`, which allows you to export multiple or all of your saved queries. Previously, you had to explicitly specify the [list of saved queries](https://docs.getdbt.com/docs/build/metricflow-commands.md#list-saved-queries). * **Enhancement**: The Semantic Layer now offers more granular control by supporting multiple data platform credentials, which can represent different roles or service accounts. Available for dbt Enterprise plans, you can map credentials to service tokens for secure authentication. Refer to [Set up Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md#set-up-dbt-semantic-layer) for more details. * **Fix**: Addressed a bug where unicode query filters (such as Chinese characters) were not working correctly in the Semantic Layer Tableau integration. * **Fix**: Resolved a bug with parsing certain private keys for BigQuery when running an export. * **Fix**: Addressed a bug that caused a "closed connection" error to be returned when querying or running an Export. * **Fix**: Resolved an issue in dbt v1 where, during partial parsing, all generated metrics in a file were incorrectly deleted instead of just those related to the changed semantic model. Now, only the metrics associated with the modified model are affected. #### June 2024 * **New:** Introduced new granularity support for cumulative metrics in MetricFlow. Granularity options for cumulative metrics are slightly different than granularity for other metric types. For other metrics, we use the `date_trunc` function to implement granularity. However, because cumulative metrics are non-additive (values can't be added up), we can't use the `date_trunc` function to change their time grain granularity. Instead, we use the `first()`, `last()`, and `avg()` aggregation functions to aggregate cumulative metrics over the requested period. By default, we take the first value of the period. You can change this behavior by using the `period_agg` parameter. For more information, refer to [Granularity options for cumulative metrics](https://docs.getdbt.com/docs/build/cumulative.md#granularity-options). ###### dbt Semantic Layer * **New:** Added support for Predicate pushdown SQL optimization in MetricFlow. We will now push down categorical dimension filters to the metric source table. Previously filters were applied after we selected from the metric source table. This change helps reduce full table scans on certain query engines. * **New:** Enabled `where` filters on dimensions (included in saved queries) to use the cache during query time. This means you can now dynamically filter your dashboards without losing the performance benefits of caching. Refer to [caching](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-cache.md#result-caching) for more information. * **Enhancement:** In [Google Sheets](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/gsheets.md), we added information icons and descriptions to metrics and dimensions options in the Query Builder menu. Click on the **Info** icon button to view a description of the metric or dimension. Available in the following Query Builder menu sections: metric, group by, where, saved selections, and saved queries. * **Enhancement:** In [Google Sheets](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/gsheets.md), you can now apply granularity to all time dimensions, not just metric time. This update uses our [APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md) to support granularity selection on any chosen time dimension. * **Enhancement**: MetricFlow time spine warnings now prompt users to configure missing or small-grain-time spines. An error message is displayed for multiple time spines per granularity. * **Enhancement**: Errors now display if no time spine is configured at the requested or smaller granularity. * **Enhancement:** Improved querying error message when no semantic layer credentials were set. * **Enhancement:** Querying grains for cumulative metrics now returns multiple granularity options (day, week, month, quarter, year) like all other metric types. Previously, you could only query one grain option for cumulative metrics. * **Fix:** Removed errors that prevented querying cumulative metrics with other granularities. * **Fix:** Fixed various Tableau errors when querying certain metrics or when using calculated fields. * **Fix:** In Tableau, we relaxed naming field expectations to better identify calculated fields. * **Fix:** Fixed an error when refreshing database metadata for columns that we can't convert to Arrow. These columns will now be skipped. This mainly affected Redshift users with custom types. * **Fix:** Fixed Private Link connections for Databricks. ###### Also available this month: * **Enhancement:** Updates to the UI when [creating merge jobs](https://docs.getdbt.com/docs/deploy/merge-jobs.md) are now available. The updates include improvements to helper text, new deferral settings, and performance improvements. * **New**: The Semantic Layer now offers a seamless integration with Microsoft Excel, available in [preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles.md). Build semantic layer queries and return data on metrics directly within Excel, through a custom menu. To learn more and install the add-on, check out [Microsoft Excel](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/excel.md). * **New:** [Job warnings](https://docs.getdbt.com/docs/deploy/job-notifications.md) are now GA. Previously, you could receive email or Slack alerts about your jobs when they succeeded, failed, or were canceled. Now with the new **Warns** option, you can also receive alerts when jobs have encountered warnings from tests or source freshness checks during their run. This gives you more flexibility on *when* to be notified. * **New:** A [preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles.md) of the dbt Snowflake Native App is now available. With this app, you can access dbt Explorer, the **Ask dbt** chatbot, and orchestration observability features, extending your dbt experience into the Snowflake UI. #### May 2024 * **Enhancement:** We've now introduced a new **Prune branches** [Git button](https://docs.getdbt.com/docs/platform/studio-ide/ide-user-interface.md#prune-branches-modal) in the IDE. This button allows you to delete local branches that have been deleted from the remote repository, keeping your branch management tidy. Available in all regions now and will be released to single tenant accounts during the next release cycle. ###### dbt Cloud Launch Showcase event The following features are new or enhanced as part of our [dbt Launch Showcase](https://www.getdbt.com/resources/webinars/dbt-cloud-launch-showcase) event on May 14th, 2024: * **New:** [](https://docs.getdbt.com/docs/platform/wizard-platform.md)is a powerful AI engine helping you generate documentation, tests, and semantic models, saving you time as you deliver high-quality data. Available in private beta for a subset of dbt Enterprise users and in the IDE. [Register your interest](https://docs.google.com/forms/d/e/1FAIpQLScPjRGyrtgfmdY919Pf3kgqI5E95xxPXz-8JoVruw-L9jVtxg/viewform) to join the private beta. * **New:** The new low-code editor, now in private beta, enables less SQL-savvy analysts to create or edit dbt models through a visual, drag-and-drop experience inside of dbt. These models compile directly to SQL and are indistinguishable from other dbt models in your projects: they are version-controlled, can be accessed across projects in Mesh, and integrate with dbt Explorer and the Cloud IDE. [Register your interest](https://docs.google.com/forms/d/e/1FAIpQLScPjRGyrtgfmdY919Pf3kgqI5E95xxPXz-8JoVruw-L9jVtxg/viewform) to join the private beta. * **New:** [dbt CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) is now Generally Available (GA) to all users. The dbt CLI is a command-line interface that allows you to interact with dbt, use automatic deferral, leverage Mesh, and more! * **New:** [Unit tests](https://docs.getdbt.com/docs/build/unit-tests.md) are now GA in dbt. Unit tests enable you to test your SQL model logic against a set of static inputs. *  New: Native support for Azure Synapse Analytics[Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Native support in dbt Cloud for Azure Synapse Analytics is now available as a [preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles.md)! To learn more, refer to [Connect Azure Synapse Analytics](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-azure-synapse-analytics.md) and [Microsoft Azure Synapse DWH configurations](https://docs.getdbt.com/reference/resource-configs/azuresynapse-configs.md). Also, check out the [Quickstart for dbt Cloud and Azure Synapse Analytics](https://docs.getdbt.com/guides/azure-synapse-analytics.md?step=1). The guide walks you through: * Loading the Jaffle Shop sample data (provided by dbt Labs) into Azure Synapse Analytics. * Connecting dbt Cloud to Azure Synapse Analytics. * Turning a sample query into a model in your dbt project. A model in dbt is a SELECT statement. * Adding tests to your models. * Documenting your models. * Scheduling a job to run. * **New:** MetricFlow enables you to now add metrics as dimensions to your metric filters to create more complex metrics and gain more insights. Available for all Semantic Layer users. * **New:** [Staging environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md#staging-environment) is now GA. Use staging environments to grant developers access to deployment workflows and tools while controlling access to production data. Available to all dbt users. * **New:** Oauth login support via [Databricks](https://docs.getdbt.com/docs/platform/manage-access/set-up-databricks-oauth.md) is now GA to Enterprise customers. *  New: GA of dbt Explorer's features dbt Explorer's current capabilities — including column-level lineage, model performance analysis, and project recommendations — are now Generally Available for dbt Cloud Enterprise and Teams plans. With Explorer, you can more easily navigate your dbt Cloud project – including models, sources, and their columns – to gain a better understanding of its latest production or staging state. To learn more about its features, check out: * [Explore projects](https://docs.getdbt.com/docs/explore/explore-projects.md) * [Explore multiple projects](https://docs.getdbt.com/docs/explore/explore-multiple-projects.md) * [Column-level lineage](https://docs.getdbt.com/docs/explore/column-level-lineage.md) * [Model performance](https://docs.getdbt.com/docs/explore/model-performance.md) * [Project recommendations](https://docs.getdbt.com/docs/explore/project-recommendations.md) * **New:** Native support for Microsoft Fabric in dbt is now GA. This feature is powered by the [dbt-fabric](https://github.com/Microsoft/dbt-fabric) adapter. To learn more, refer to [Connect Microsoft Fabric](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-microsoft-fabric.md) and [Microsoft Fabric DWH configurations](https://docs.getdbt.com/reference/resource-configs/fabric-configs.md). There's also a [quickstart guide](https://docs.getdbt.com/guides/microsoft-fabric.md?step=1) to help you get started. * **New:** Mesh is now GA to dbt Enterprise users. Mesh is a framework that helps organizations scale their teams and data assets effectively. It promotes governance best practices and breaks large projects into manageable sections. Get started with Mesh by reading the [Mesh quickstart guide](https://docs.getdbt.com/guides/mesh-qs.md?step=1). * **New:** The Semantic Layer [Tableau Desktop, Tableau Server](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/tableau.md), and [Google Sheets integration](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/gsheets.md) is now GA to dbt Team or Enterprise accounts. These first-class integrations allow you to query and unlock valuable insights from your data ecosystem. * **Enhancement:** As part of our ongoing commitment to improving the [IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md#considerations), the filesystem now comes with improvements to speed up dbt development, such as introducing a Git repository limit of 10GB. ###### Also available this month: * **Update**: The [dbt CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) is now available for Azure single tenant and is accessible in all [deployment regions](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for both multi-tenant and single-tenant accounts. * **New**: The [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) introduces [declarative caching](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-cache.md), allowing you to cache common queries to speed up performance and reduce query compute costs. Available for dbt Team or Enterprise accounts. *  New: Latest Release Track The **Latest** Release Track is now Generally Available (previously Public Preview). On this release track, you get automatic upgrades of dbt, including early access to the latest features, fixes, and performance improvements for your dbt project. dbt Labs will handle upgrades behind-the-scenes, as part of testing and redeploying the dbt Cloud application — just like other dbt Cloud capabilities and other SaaS tools that you're using. No more manual upgrades and no more need for *a second sandbox project* just to try out new features in development. To learn more about the new setting, refer to [Release Tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) for details. [![Example of the Latest setting](/img/docs/dbt-platform/platform-configuring-dbt-platform/choosing-dbt-version/example-environment-settings.png?v=2 "Example of the Latest setting")](#)Example of the Latest setting * **Behavior change:** Introduced the `require_resource_names_without_spaces` flag, opt-in and disabled by default. If set to `True`, dbt will raise an exception if it finds a resource name containing a space in your project or an installed package. This will become the default in a future version of dbt. Read [No spaces in resource names](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_resource_names_without_spaces.md) for more information. #### April 2024 *  New: Merge jobs[Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") You can now set up a continuous deployment (CD) workflow for your projects natively in dbt Cloud. You can now access a beta release of [Merge jobs](https://docs.getdbt.com/docs/deploy/merge-jobs.md), which is a new [job type](https://docs.getdbt.com/docs/deploy/jobs.md), that enables you to trigger dbt job runs as soon as changes (via Git pull requests) merge into production. [![Example of creating a merge job](/img/docs/dbt-platform/using-dbt-platform/example-create-merge-job.png?v=2 "Example of creating a merge job")](#)Example of creating a merge job * **Behavior change:** Introduced the `require_explicit_package_overrides_for_builtin_materializations` flag, opt-in and disabled by default. If set to `True`, dbt will only use built-in materializations defined in the root project or within dbt, rather than implementations in packages. This will become the default in May 2024 (dbt Core v1.8 and dbt Cloud release tracks). Read [Package override for built-in materialization](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_explicit_package_overrides_for_builtin_materializations.md) for more information. **Semantic Layer** * **New**: Use Saved selections to [save your query selections](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/gsheets.md#using-saved-selections) within the [Google Sheets application](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/gsheets.md). They can be made private or public and refresh upon loading. * **New**: Metrics are now displayed by their labels as `metric_name`. * **Enhancement**: [Metrics](https://docs.getdbt.com/docs/build/metrics-overview.md) now supports the [`meta` option](https://docs.getdbt.com/reference/resource-configs/meta.md) under the [config](https://docs.getdbt.com/reference/resource-properties/config.md) property. Previously, we only supported the now deprecated `meta` tag. * **Enhancement**: In the Google Sheets application, we added [support](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/gsheets.md#using-saved-queries) to allow jumping off from or exploring MetricFlow-defined saved queries directly. * **Enhancement**: In the Google Sheets application, we added support to query dimensions without metrics. Previously, you needed a dimension. * **Enhancement**: In the Google Sheets application, we added support for time presets and complex time range filters such as "between", "after", and "before". * **Enhancement**: In the Google Sheets application, we added supported to automatically populate dimension values when you select a "where" filter, removing the need to manually type them. Previously, you needed to manually type the dimension values. * **Enhancement**: In the Google Sheets application, we added support to directly query entities, expanding the flexibility of data requests. * **Enhancement**: In the Google Sheets application, we added an option to exclude column headers, which is useful for populating templates with only the required data. * **Deprecation**: For the Tableau integration, the [`METRICS_AND_DIMENSIONS` data source](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/tableau.md#using-the-integration) has been deprecated for all accounts not actively using it. We encourage users to transition to the "ALL" data source for future integrations. #### March 2024 * **New:** The Semantic Layer services now support using Privatelink for customers who have it enabled. * **New:** You can now develop against and test your Semantic Layer in the dbt CLI if your user credential uses SSO. * **Enhancement:** You can select entities to Group By, Filter By, and Order By. * **Fix:** `dbt parse` no longer shows an error when you use a list of filters (instead of just a string filter) on a metric. * **Fix:** `join_to_timespine` now properly gets applied to conversion metric input measures. * **Fix:** Fixed an issue where exports in Redshift were not always committing to the DWH, which also had the side-effect of leaving table locks open. * **Behavior change:** Introduced the `source_freshness_run_project_hooks` flag, opt-in and disabled by default. If set to `True`, dbt will include `on-run-*` project hooks in the `source freshness` command. This will become the default in a future version of dbt. Read [Project hooks with source freshness](https://docs.getdbt.com/reference/global-configs/behavior-flags/source_freshness_run_project_hooks.md) for more information. #### February 2024 * **New:** [Exports](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md#define-exports) allow you to materialize a saved query as a table or view in your data platform. By using exports, you can unify metric definitions in your data platform and query them as you would any other table or view. * **New:** You can access a list of your [exports](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md) with the new list saved-queries command by adding `--show-exports` * **New:** The Semantic Layer and [Tableau Connector](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/tableau.md) now supports relative date filters in Tableau. *  New: Use exports to write saved queries You can now use the [exports](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md) feature with [dbt Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md), allowing you to query reliable metrics and fast data reporting. Exports enhance the saved queries feature, allowing you to write commonly used queries directly within your data platform using dbt Cloud's job scheduler. By exposing tables of metrics and dimensions, exports enable you to integrate with additional tools that don't natively connect with the dbt Semantic Layer, such as PowerBI. Exports are available for dbt Cloud multi-tenant [Team or Enterprise](https://www.getdbt.com/pricing/) plans on dbt versions 1.7 or newer. Refer to the [exports blog](https://www.getdbt.com/blog/announcing-exports-for-the-dbt-semantic-layer) for more details. [![Add an environment variable to run exports in your production run.](/img/docs/dbt-platform/semantic-layer/deploy_exports.png?v=2 "Add an environment variable to run exports in your production run.")](#)Add an environment variable to run exports in your production run. *  New: Trigger on job completion teamenterprise Now available for dbt Cloud Team and Enterprise plans is the ability to trigger deploy jobs when other deploy jobs are complete. You can enable this feature [in the UI](https://docs.getdbt.com/docs/deploy/deploy-jobs.md) with the **Run when another job finishes** option in the **Triggers** section of your job or with the [Create Job API endpoint](https://docs.getdbt.com/dbt-cloud/api-v2#/operations/Create%20Job). When enabled, your job will run after the specified upstream job completes. You can configure which run status(es) will trigger your job. It can be just on `Success` or on all statuses. If you have dependencies between your dbt projects, this allows you to *natively* orchestrate your jobs within dbt Cloud — no need to set up a third-party tool. An example of the **Triggers** section when creating the job: [![Example of Triggers on the Deploy Job page](/img/docs/dbt-platform/using-dbt-platform/example-triggers-section.png?v=2 "Example of Triggers on the Deploy Job page")](#)Example of Triggers on the Deploy Job page *  New: Latest Release Track[Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") *Now available in the dbt version dropdown in dbt Cloud — starting with select customers, rolling out to wider availability through February and March.* On this release track, you get automatic upgrades of dbt, including early access to the latest features, fixes, and performance improvements for your dbt project. dbt Labs will handle upgrades behind-the-scenes, as part of testing and redeploying the dbt Cloud application — just like other dbt Cloud capabilities and other SaaS tools that you're using. No more manual upgrades and no more need for *a second sandbox project* just to try out new features in development. To learn more about the new setting, refer to [Release Tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) for details. [![Example of the Latest setting](/img/docs/dbt-platform/platform-configuring-dbt-platform/choosing-dbt-version/example-environment-settings.png?v=2 "Example of the Latest setting")](#)Example of the Latest setting *  New: Override dbt version with new User development settings You can now [override the dbt version](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md#override-dbt-version) that's configured for the development environment within your project and use a different version — affecting only your user account. This lets you test new dbt features without impacting other people working on the same project. And when you're satisfied with the test results, you can safely upgrade the dbt version for your project(s). Use the **dbt version** dropdown to specify the version to override with. It's available on your project's credentials page in the **User development settings** section. For example: [![Example of overriding the dbt version on your user account](/img/docs/dbt-platform/platform-configuring-dbt-platform/choosing-dbt-version/example-override-version.png?v=2 "Example of overriding the dbt version on your user account")](#)Example of overriding the dbt version on your user account *  Enhancement: Edit in primary git branch in IDE You can now edit, format, or lint files and execute dbt commands directly in your primary git branch in the [dbt Cloud IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md). This enhancement is available across various repositories, including native integrations, imported git URLs, and managed repos. This enhancement is currently available to all dbt Cloud multi-tenant regions and will soon be available to single-tenant accounts. The primary branch of the connected git repo has traditionally been *read-only* in the IDE. This update changes the branch to *protected* and allows direct edits. When a commit is made, dbt Cloud will prompt you to create a new branch. dbt Cloud will pre-populate the new branch name with the GIT\_USERNAME-patch-#; however, you can edit the field with a custom branch name. Previously, the primary branch was displayed as read-only, but now the branch is displayed with a lock icon to identify it as protected: [![Previous read-only experience](/img/docs/dbt-platform/using-dbt-platform/read-only.png?v=2 "Previous read-only experience")](#)Previous read-only experience [![New protected experience](/img/docs/dbt-platform/using-dbt-platform/protected.png?v=2 "New protected experience")](#)New protected experience When you make a commit while on the primary branch, a modal window will open prompting you to create a new branch and enter a commit message: [![Create new branch window](/img/docs/dbt-platform/using-dbt-platform/create-new-branch.png?v=2 "Create new branch window")](#)Create new branch window * **Enhancement:** The Semantic Layer [Google Sheets integration](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/gsheets.md) now exposes a note on the cell where the data was requested, indicating clearer data requests. The integration also now exposes a new **Time Range** option, which allows you to quickly select date ranges. * **Enhancement:** The [GraphQL API](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md) includes a `requiresMetricTime` parameter to better handle metrics that must be grouped by time. (Certain metrics defined in MetricFlow can't be looked at without a time dimension). * **Enhancement:** Enable querying metrics with offset and cumulative metrics with the time dimension name, instead of `metric_time`. [Issue #1000](https://github.com/dbt-labs/metricflow/issues/1000) * Enable querying `metric_time` without metrics. [Issue #928](https://github.com/dbt-labs/metricflow/issues/928) * **Enhancement:** Added support for consistent SQL query generation, which enables ID generation consistency between otherwise identical MF queries. Previously, the SQL generated by `MetricFlowEngine` was not completely consistent between identical queries. [Issue 1020](https://github.com/dbt-labs/metricflow/issues/1020) * **Fix:** The Tableau Connector returns a date filter when filtering by dates. Previously it was erroneously returning a timestamp filter. * **Fix:** MetricFlow now validates if there are `metrics`, `group by`, or `saved_query` items in each query. Previously, there was no validation. [Issue 1002](https://github.com/dbt-labs/metricflow/issues/1002) * **Fix:** Measures using `join_to_timespine` in MetricFlow now have filters applied correctly after time spine join. * **Fix:** Querying multiple granularities with offset metrics: * If you query a time offset metric with multiple instances of `metric_time`/`agg_time_dimension`, only one of the instances will be offset. All of them should be. * If you query a time offset metric with one instance of `metric_time`/`agg_time_dimension` but filter by a different one, the query will fail. * **Fix:** MetricFlow prioritizes a candidate join type over the default type when evaluating nodes to join. For example, the default join type for distinct values queries is `FULL OUTER JOIN`, however, time spine joins require `CROSS JOIN`, which is more appropriate. * **Fix:** Fixed a bug that previously caused errors when entities were referenced in `where` filters. #### January 2024 *  January docs updates Hello from the dbt Docs team: @mirnawong1, @matthewshaver, @nghi-ly, and @runleonarun! First, we’d like to thank the 10 new community contributors to docs.getdbt.com 🙏 What a busy start to the year! We merged 110 PRs in January. Here's how we improved the [docs.getdbt.com](http://docs.getdbt.com/) experience: * Added new hover behavior for images * Added new expandables for FAQs * Pruned outdated notices and snippets as part of the docs site maintenance January saw some great new content: * New [dbt Mesh FAQs](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-5-faqs.md) page * Beta launch of [Explorer’s column-level lineage](https://docs.getdbt.com/docs/explore/column-level-lineage.md) feature * Developer blog posts: * [More time coding, less time waiting: Mastering defer in dbt](https://docs.getdbt.com/blog/defer-to-prod) * [Deprecation of dbt Server](https://docs.getdbt.com/blog/deprecation-of-dbt-server) * From the community: [Serverless, free-tier data stack with dlt + dbt core](https://docs.getdbt.com/blog/serverless-dlt-dbt-stack) * The Extrica team added docs for the [dbt-extrica community adapter](https://docs.getdbt.com/docs/local/connect-data-platform/extrica-setup.md) * Semantic Layer: New [conversion metrics docs](https://docs.getdbt.com/docs/build/conversion.md) and added the parameter `fill_nulls_with` to all metric types (launched the week of January 12, 2024) * New [dbt environment command](https://docs.getdbt.com/reference/commands/dbt-environment.md?version=2.0) and its flags for the dbt CLI January also saw some refreshed content, either aligning with new product features or requests from the community: * Native support for [partial parsing in dbt Cloud](https://docs.getdbt.com/docs/platform/account-settings.md#partial-parsing) * Updated guidance on using dots or underscores in the [Best practice guide for models](https://docs.getdbt.com/best-practices/how-we-style/1-how-we-style-our-dbt-models.md) * Updated [PrivateLink for VCS docs](https://docs.getdbt.com/docs/platform/secure/private-connectivity/aws/aws-self-hosted.md) * Added a new `job_runner` role in our [Enterprise project role permissions docs](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#project-role-permissions) * Added saved queries to [Metricflow commands](https://docs.getdbt.com/docs/build/metricflow-commands.md#list-saved-queries) * Removed [as\_text docs](https://github.com/dbt-labs/docs.getdbt.com/pull/4726) that were wildly outdated * **New:** New metric type that allows you to measure conversion events. For example, users who viewed a web page and then filled out a form. For more details, refer to [Conversion metrics](https://docs.getdbt.com/docs/build/conversion.md). * **New:** Instead of specifying the fully qualified dimension name (for example, `order__user__country`) in the group by or filter expression, you now only need to provide the primary entity and dimensions name, like `user__county`. * **New:** You can now query the [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md) you've defined in the Semantic Layer using [Tableau](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/tableau.md), [GraphQL API](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md), [JDBC API](https://docs.getdbt.com/docs/dbt-apis/sl-jdbc.md), and the [dbt CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md). *  New: Native support for partial parsing By default, dbt parses all the files in your project at the beginning of every dbt invocation. Depending on the size of your project, this operation can take a long time to complete. With the new partial parsing feature in dbt Cloud, you can reduce the time it takes for dbt to parse your project. When enabled, dbt Cloud parses only the changed files in your project instead of parsing all the project files. As a result, your dbt invocations will take less time to run. To learn more, refer to [Partial parsing](https://docs.getdbt.com/docs/platform/account-settings.md#partial-parsing). [![Example of the Partial parsing option](/img/docs/deploy/account-settings-partial-parsing.png?v=2 "Example of the Partial parsing option")](#)Example of the Partial parsing option * **Enhancement:** The YAML spec parameter `label` is now available for Semantic Layer metrics in [JDBC and GraphQL APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md). This means you can conveniently use `label` as a display name for your metrics when exposing them. * **Enhancement:** Added support for `create_metric: true` for a measure, which is a shorthand to quickly create metrics. This is useful in cases when metrics are only used to build other metrics. * **Enhancement:** Added support for Tableau parameter filters. You can use the [Tableau connector](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/tableau.md) to create and use parameters with your Semantic Layer data. * **Enhancement:** Added support to expose `expr` and `agg` for [Measures](https://docs.getdbt.com/docs/build/measures.md) in the [GraphQL API](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md). * **Enhancement:** You have improved error messages in the command line interface when querying a dimension that is not reachable for a given metric. * **Enhancement:** You can now query entities using our Tableau integration (similar to querying dimensions). * **Enhancement:** A new data source is available in our Tableau integration called "ALL", which contains all semantic objects defined. This has the same information as "METRICS\_AND\_DIMENSIONS". In the future, we will deprecate "METRICS\_AND\_DIMENSIONS" in favor of "ALL" for clarity. * **Fix:** Support for numeric types with precision greater than 38 (like `BIGDECIMAL`) in BigQuery is now available. Previously, it was unsupported so would return an error. * **Fix:** In some instances, large numeric dimensions were being interpreted by Tableau in scientific notation, making them hard to use. These should now be displayed as numbers as expected. * **Fix:** We now preserve dimension values accurately instead of being inadvertently converted into strings. * **Fix:** Resolved issues with naming collisions in queries involving multiple derived metrics using the same metric input. Previously, this could cause a naming collision. Input metrics are now deduplicated, ensuring each is referenced only once. * **Fix:** Resolved warnings related to using two duplicate input measures in a derived metric. Previously, this would trigger a warning. Input measures are now deduplicated, enhancing query processing and clarity. * **Fix:** Resolved an error where referencing an entity in a filter using the object syntax would fail. For example, `{{Entity('entity_name')}}` would fail to resolve. --- ### 2025 dbt platform release notes dbt platform | Free dbt release notes for recent and historical changes. Release notes fall into one of the following categories: * **New:** New products and features * **Enhancement:** Performance improvements and feature enhancements * **Fix:** Bug and security fixes * **Behavior change:** A change to existing behavior that doesn't fit into the other categories, such as feature deprecations or changes to default settings Release notes are grouped by month for both multi-tenant and virtual private cloud (VPC) environments. #### December 2025 * **New**: [Global navigation](https://docs.getdbt.com/docs/explore/global-navigation.md) is now the default experience for Catalog, providing a unified search experience that lets you find dbt resources across all your projects, as well as non-dbt resources in Snowflake. Global navigation is now generally available to all users. You can access Catalog by clicking **Catalog** in the top-level navigation. * **Enhancement**: dbt SSO slugs are now system-generated during SSO setup and aren't customizable. SSO slug configurations currently in use will remain valid; they will be read-only and cannot be changed. If you delete your existing SSO configuration and create a new one, you'll be provided with a new system-generated SSO slug. This change enhances security and prevents accounts from setting slugs that "impersonate" other organizations. * **Enhancement**: For users in the default region (`US1`) that previously created a dbt account in the past, the dbt VS Code extension now supports registering with OAuth . This makes it easier to register the extension for users who may have forgotten their password or are locked out of their account. For more information, see [Sign in or register](https://docs.getdbt.com/docs/sign-in-dbt-extension.md). * **New and enhancements:** The dbt [Studio IDE user interface](https://docs.getdbt.com/docs/platform/studio-ide/ide-user-interface.md) has been enhanced to bring more powerful development features to your fingertips: * A newly designed toolbar that groups all of your action and project insight tabs for easy access. * A dedicated inline **Commands** tab for history and logs. * When you upgrade your development environment to dbt v2, the environment includes a new **Problems** tab that gives you live error detection on issues that could block your project from running successfully. #### November 2025 * **Behavior change**: [dbt Copilot](https://docs.getdbt.com/docs/platform/wizard-platform.md) now requires all input files to use UTF-8 encoding. Files that use other encodings will return an error. If you're working with legacy files that use a different encoding, convert them to UTF-8 before using Copilot. * **Enhancement**: dbt Copilot now has improved reliability when working with OpenAI. This includes longer timeouts, better retry behavior, and improved handling of reasoning messages for long code generations, resulting in fewer failures and more successful completions. * **New**: The Snowflake adapter now supports basic table materialization on Iceberg tables registered in a Glue catalog through a [catalog-linked database](https://docs.snowflake.com/en/user-guide/tables-iceberg-catalog-linked-database#label-catalog-linked-db-create). For more information, see [Glue Data Catalog](https://docs.getdbt.com/docs/build/iceberg/adapters/snowflake-iceberg-support.md#external-catalogs). * **New**: You can use the `platform_detection_timeout_seconds` parameter to control how long the Snowflake connector waits when detecting the cloud platform where the connection is being made. For more information, see [Snowflake setup](https://docs.getdbt.com/docs/local/connect-data-platform/snowflake-setup.md#platform_detection_timeout_seconds). * **New**: The `cluster_by` configuration is supported in dynamic tables. For more information, see [Dynamic table clustering](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#dynamic-table-clustering). * **New**: When jobs exceed their configured timeout, the BigQuery adapter sends a cancellation request to the BigQuery job. For more information, see [Connect BigQuery](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-bigquery.md#job-creation-timeout-seconds). #### October 2025 * **Enhancement**: dbt enforces cumulative log size limits on run endpoints. If logs exceed this limit, dbt omits them and displays a banner. For more information, see [Run visibility](https://docs.getdbt.com/docs/deploy/run-visibility.md#log-size-limits). * **New**: The [docs.getdbt.com](http://docs.getdbt.com/) documentation site has introduced an LLM Context menu on all product documentation and guide pages. This menu provides users with quick options to interact with the current page using LLMs. You can can now: * Copy the page as raw Markdown — This makes it easier to reference or reuse documentation content. * Open the page directly in ChatGPT or Claude — This redirects you to a chat with the LLM and automatically loads a message asking it to read the page, helping you start a conversation with context from the page. [![LLM Context menu on documentation pages](/img/llm-menu.png?v=2 "LLM Context menu on documentation pages")](#)LLM Context menu on documentation pages * **Enhancement**: The CodeGenCodeLen feature has been re-introduced to the Studio IDE. This feature was [temporarily](#pre-coalesce) removed in the previous release due to compatibility issues. ##### Coalesce 2025 announcements The following features are new or enhanced as part of [dbt's Coalesce analytics engineering conference](https://coalesce.getdbt.com/event/21662b38-2c17-4c10-9dd7-964fd652ab44/summary) from October 13-16, 2025: * **New**: The [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md) is now generally available (GA). For more information on the dbt MCP server and dbt Agents, refer to the [Announcing dbt Agents and the remote dbt MCP Server: Trusted AI for analytics](https://www.getdbt.com/blog/dbt-agents-remote-dbt-mcp-server-trusted-ai-for-analytics) blog post. * **Private preview**: The [dbt platform (powered by Fusion)](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md#dbt-v2) is now in private preview. If you have any questions, please reach out to your account manager. * [About data platform connections](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md) lists all available dbt platform connections on Fusion and the supported authentication methods per connection. * **New**: Fusion‑specific configuration is now available for BigQuery, Databricks, Redshift, and Snowflake. For more information, see [Connect Fusion to your data platform](https://docs.getdbt.com/docs/local/profiles.yml.md). * **Alpha**: The `dbt-salesforce` adapter is available via dbt v2 CLI. Note that this connection is in the Alpha product stage and is not production-ready. For more information, see [Salesforce Data Cloud setup](https://docs.getdbt.com/docs/local/connect-data-platform/salesforce-data-cloud-setup.md). * **Private preview**: [State-aware orchestration](https://docs.getdbt.com/docs/deploy/state-aware-about.md) is now in private preview! * **New**: You can now [enable state-aware orchestration](https://docs.getdbt.com/docs/deploy/state-aware-setup.md) by selecting **Enable Fusion cost optimization features** in your job settings. Previously, you had to disable **Force node selection** to enable state-aware orchestration. * **Private beta**: The [Efficient Testing feature](https://docs.getdbt.com/docs/deploy/state-aware-about.md#efficient-testing-in-state-aware-orchestration) is now available in private beta. This feature reduces warehouse costs by avoiding redundant data tests and combining multiple tests in a single query. * **New**: To improve visibility into state‑aware orchestration and provide better control when you need to reset cached state, the following [UI enhancements](https://docs.getdbt.com/docs/deploy/state-aware-interface.md) are introduced: * **Models built and reused chart** on your **Account home** * New charts in the **Overview** section of your job that display **Recent runs**, **Total run duration**, **Models built**, and **Models reused** * A new structure to view logs grouped by models, with a **Reused** tab to quickly find reused models * **Reused** tag in **Latest status** lineage lens to see reused models in your DAG * **Clear cache** button on the **Environments** page to reset cached state when needed * **New**: [dbt Insights](https://docs.getdbt.com/docs/explore/dbt-insights.md) is now generally available (GA)! * **Private beta**: The [Analyst agent](https://docs.getdbt.com/docs/explore/navigate-dbt-insights.md#dbt-copilot) is now available in dbt Insights. The Analyst agent is a conversational AI feature where you can ask natural language prompts and receive analysis in real-time. For more information, see [Analyze data with the Analyst agent](https://docs.getdbt.com/docs/dbt-ai/analyst-agent.md). * **Beta**: The [Semantic Layer querying](https://docs.getdbt.com/docs/explore/navigate-dbt-insights.md#semantic-layer-querying) within dbt Insights is now available in beta. With this feature, you can build SQL queries against the Semantic Layer without writing SQL code. It guides you in creating queries based on available metrics, dimensions, and entities. * **Enhancement**: In [dbt Insights](https://docs.getdbt.com/docs/explore/dbt-insights.md), projects upgraded to the [dbt v2](https://docs.getdbt.com/docs/introduction.md) get [Language Server Protocol (LSP) features](https://docs.getdbt.com/docs/explore/navigate-dbt-insights.md#lsp-features) and their compilation running on dbt v2. * **New**: [MetricFlow](https://docs.getdbt.com/docs/build/about-metricflow.md) is now developed and maintained as part of the [Open Semantic Interchange (OSI)](https://www.snowflake.com/en/blog/open-semantic-interchange-ai-standard/) initiative, and is distributed under the [Apache 2.0 license](https://github.com/dbt-labs/metricflow/blob/main/LICENSE). For more information, see the blog post about [Open sourcing MetricFlow](https://www.getdbt.com/blog/open-source-metricflow-governed-metrics). ##### Pre-Coalesce * **Behavior change**: dbt platform [access URLs](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for accounts in the US multi-tenant (US MT) region are transitioning from `cloud.getdbt.com` to dedicated domains on `dbt.com` (for example, `us1.dbt.com`). Users will be automatically redirected, which means no action is required. EMEA and APAC MT accounts are not impacted by this change and will be updated by the end of November 2025. Organizations that use network allow-listing should add `YOUR_ACCESS_URL.dbt.com` to their allow list (for example, if your access URL is `ab123.us1.dbt.com`, add the entire domain `ab123.us1.dbt.com` to your allow list). All OAuth, Git, and public API integrations will continue to work with the previous domain. View the updated access URL in dbt platform's **Account settings** page. For questions, contact <support@getdbt.com>. * **Enhancement**: * **dbt v2 MCP tools** — Added dbt v2 tools that support `compile_sql` and `get_column_lineage` (Fusion-exclusive) for both [Remote](https://docs.getdbt.com/docs/dbt-ai/mcp-available-tools.md#fusion-tools-remote) and [Local](https://docs.getdbt.com/docs/dbt-ai/mcp-available-tools.md#fusion-tools-local) usage. Remote dbt v2 tools defer to your prod environment by default (set with `x-dbt-prod-environment-id`); you can disable deferral with `x-dbt-fusion-disable-defer=true`. Refer to [set up remote MCP](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md) for more info. * **Self-hosted MCP server OAuth** — You can now authenticate the self-hosted dbt MCP server to the dbt platform with OAuth (supported docs for [Claude](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-claude.md), [Cursor](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-cursor.md), and [VS Code](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-vscode.md)), reducing local secret management and standardizing setup. Refer to [dbt platform authentication](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#dbt-platform-authentication) for more information. * **Behavior change**: The CodeGenCodeLens feature for creating models from your sources with a click of a button has been temporarily removed from the Studio IDE due to compatibility issues. We plan to reintroduce this feature in the near future for both the IDE and the VS Code extension. #### September 2025 * **Fix**: Improved how [MetricFlow](https://docs.getdbt.com/docs/build/about-metricflow.md) handles [offset metrics](https://docs.getdbt.com/docs/build/derived.md) for more accurate results when querying time-based data. MetricFlow now joins data *after* aggregation when the query grain matches the offset grain. Previously, when querying offset metrics, the offset join was applied *before* aggregation, which could exclude some values from the total time period. #### August 2025 * **Fix**: Resolved a bug that caused [saved query](https://docs.getdbt.com/docs/build/saved-queries.md) exports to fail during `dbt build` with `Unable to get saved_query` errors. * **New**: The Semantic Layer GraphQL API now has a [`queryRecords`](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md#query-records) endpoint. With this endpoint, you can view the query history both for Insights and Semantic Layer queries. * **Fix**: Resolved a bug that caused Semantic Layer queries with a trailing whitespace to produce an error. This issue mostly affected [Push.ai](https://docs.push.ai/data-sources/semantic-layers/dbt) users and is fixed now. * **New**: You can now use [personal access tokens (PATs)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) to authenticate in the Semantic Layer. This enables user-level authentication and reduces the need for sharing tokens between users. When you authenticate using PATs, queries are run using your personal user credentials. For more information, see [Set up the dbt Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md). #### July 2025 * **New**: The [Tableau Cloud](https://www.tableau.com/products/cloud-bi) integration with Semantic Layer is now available. For more information, see [Tableau](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/tableau.md). * **Preview**: The [Semantic Layer Power BI integration](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/power-bi.md) is now available in Preview. * **Enhancement:** You can now use `limit` and `order_by` parameters when creating [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md). * **Enhancement:** Users assigned IT [licenses](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md) can now edit and manage [global connections settings](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md#connection-management). * **New**: Paginated [GraphQL](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md) endpoints for metadata queries in Semantic Layer are now available. This improves integration load times for large manifests. For more information, see [Metadata calls](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md#metadata-calls). #### June 2025 * **New**: [System for Cross-Domain Identity Management](https://docs.getdbt.com/docs/platform/manage-access/scim.md#scim-configuration-for-entra-id) (SCIM) through Microsoft Entra ID is now GA. Also available on legacy Enterprise plans. * **Enhancement:** You can now set the [compilation environment](https://docs.getdbt.com/docs/explore/access-dbt-insights.md#set-jinja-environment) to control how Jinja functions are rendered in dbt Insights. * **Beta**: The dbt Fusion engine supports the BigQuery adapter in beta. * **New:** You can now view the history of settings changes for [projects](https://docs.getdbt.com/docs/platform/account-settings.md), [environments](https://docs.getdbt.com/docs/dbt-platform-environments.md), and [jobs](https://docs.getdbt.com/docs/deploy/deploy-jobs.md). * **New:** Added support for the latest version of BigQuery credentials in Semantic Layer and MetricFlow. * **New:** Snowflake External OAuth is now supported for Semantic Layer queries. Snowflake connections that use External OAuth for user credentials can now emit queries for Insights, dbt platform CLI, and Studio IDE through the Semantic Layer Gateway. This enables secure, identity-aware access via providers like Okta or Microsoft Entra ID. * **New:** You can now [download your managed Git repo](https://docs.getdbt.com/docs/platform/git/managed-repository.md#download-managed-repository) from the dbt platform. * **New**: The Semantic Layer now supports Trino as a data platform. For more details, see [Set up the Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md). * **New**: The dbt Fusion engine supports Databricks in beta. * **Enhancement**: Group owners can now specify multiple email addresses for model-level notifications, enabling broader team alerts. Previously, only a single email address was supported. Check out the [Configure groups](https://docs.getdbt.com/docs/deploy/model-notifications.md#configure-groups) section to learn more. * **New**: The Semantic Layer GraphQL API now has a [`List a saved query`](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md#list-a-saved-query) endpoint. #### May 2025 ##### 2025 dbt Launch Showcase The following features are new or enhanced as part of our [dbt Launch Showcase](https://www.getdbt.com/resources/webinars/2025-dbt-cloud-launch-showcase) on May 28th, 2025: * **New**: The dbt Fusion engine is the brand new dbt engine re-written from the ground up to provide incredible speed, cost-savings tools, and comprehensive SQL language tools. The dbt Fusion engine is now available in beta for Snowflake users. * Read more [about Fusion](https://docs.getdbt.com/docs/introduction.md). * Understand what actions you need to take to get your projects Fusion-ready with the [upgrade guide](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md). * Begin testing today with the [quickstart guide](https://docs.getdbt.com/guides/dbt.md). * Know [where we're headed with the dbt Fusion engine](https://getdbt.com/blog/where-we-re-headed-with-the-dbt-fusion-engine). * **New**: The dbt VS Code extension is a powerful new tool that brings the speed and productivity of the dbt Fusion engine into your Visual Studio Code editor. This is a free download that will forever change your dbt development workflows. The dbt VS Code extension is now available as beta [alongside Fusion](https://getdbt.com/blog/get-to-know-the-new-dbt-fusion-engine-and-vs-code-extension). Check out the [installation instructions](https://docs.getdbt.com/docs/install-dbt-extension.md) and read more [about the features](https://docs.getdbt.com/docs/about-dbt-extension.md) to get started enhancing your dbt workflows today! * **New**: dbt Explorer is now Catalog! Learn more about the change [here](https://getdbt.com/blog/updated-names-for-dbt-platform-and-features). * dbt's Catalog, global navigation provides a search experience that lets you find dbt resources across all your projects, as well as non-dbt resources in Snowflake. * External metadata ingestion allows you to connect directly to your data warehouse, giving you visibility into tables, views, and other resources that aren't defined in dbt. * **New**: [dbt Canvas is now generally available](https://getdbt.com/blog/dbt-canvas-is-ga) (GA). Canvas is the intuitive visual editing tool that enables anyone to create dbt models with an easy to understand drag-and-drop interface. Read more [about Canvas](https://docs.getdbt.com/docs/platform/canvas.md) to begin empowering your teams to build more, faster! * **New**: [State-aware orchestration](https://docs.getdbt.com/docs/deploy/state-aware-about.md) is now in beta! Every time a new job in Fusion runs, state-aware orchestration automatically determines which models to build by detecting changes in code or data. * **New**: With Hybrid Projects, your organization can adopt complementary dbt Core and dbt Cloud workflows and seamlessly integrate these workflows by automatically uploading dbt Core artifacts into dbt Cloud. [Hybrid Projects](https://docs.getdbt.com/docs/deploy/hybrid-projects.md) is now available as a preview to [dbt Enterprise accounts](https://www.getdbt.com/pricing). * **New**: [System for Cross-Domain Identity Management (SCIM)](https://docs.getdbt.com/docs/platform/manage-access/scim.md) through Okta is now GA. * **New**: dbt now acts as a [Model Context Protocol](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md) (MCP) server, allowing seamless integration of AI tools with data warehouses through a standardized framework. * **New**: The [quickstart guide for data analysts](https://docs.getdbt.com/guides/analyze-your-data.md) is now available. With dbt, data analysts can use built-in, AI-powered tools to build governed data models, explore how they’re built, and run their own analysis. * **New**: You can view your [usage metering and limiting in dbt Copilot](https://docs.getdbt.com/docs/platform/billing/dbt-ai-usage.md) on the billing page of your dbt Cloud account. * **New**: You can use Copilot to create a `dbt-styleguide.md` for dbt projects. The generated style guide template includes SQL style guidelines, model organization and naming conventions, model configurations and testing practices, and recommendations to enforce style rules. For more information, see [dbt Wizard skills](https://docs.getdbt.com/docs/dbt-ai/wizard-skills.md). * **New**: Copilot chat is an interactive interface within the Studio IDE where you can generate SQL code from natural language prompts and ask analytics-related questions. It integrates contextual understanding of your dbt project and assists in streamlining SQL development. For more information, see [dbt Wizard in Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md). * **New**: Leverage dbt Copilot to generate SQL queries in [Insights](https://docs.getdbt.com/docs/explore/dbt-insights.md) from natural language prompts, enabling efficient data exploration within a context-aware interface. * **New**: The dbt platform Cost management dashboard was available as a preview for Snowflake users on Enterprise and Enterprise Plus plans. On November 25, 2025, we retired the cost management dashboard to focus on building a more scalable and integrated cost-insights experience, expected in early 2026. * **New**: Apache Iceberg catalog integration support is now available on Snowflake and BigQuery! This is essential to making your dbt Mesh interoperable across platforms, built on Iceberg. Read more about [Iceberg](https://docs.getdbt.com/docs/build/iceberg/apache-iceberg-support.md) to begin creating Iceberg tables. * **Update**: Product renaming and other changes. For more information, refer to [Updated names for dbt platform and features](https://getdbt.com/blog/updated-names-for-dbt-platform-and-features).  Product names key * Canvas (previously Visual Editor) * Catalog (previously Explorer) * Copilot * Cost Management * dbt Fusion engine * Insights * Mesh * Orchestrator * Studio IDE (previously Cloud IDE) * Semantic Layer * Pricing plan changes. For more information, refer to [One dbt](https://www.getdbt.com/product/one-dbt). #### April 2025 * **Enhancement**: The [Python SDK](https://docs.getdbt.com/docs/dbt-apis/sl-python.md) now supports lazy loading for large fields for `dimensions`, `entities`, and `measures` on `Metric` objects. For more information, see [Lazy loading for large fields](https://docs.getdbt.com/docs/dbt-apis/sl-python.md#lazy-loading-for-large-fields). * **Enhancement**: The Semantic Layer now supports SSH tunneling for [Postgres](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-postgresql-alloydb.md#connecting-using-an-ssh-tunnel) or [Redshift](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-redshift.md#connecting-using-an-ssh-tunnel) connections. Refer to [Set up the Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md) for more information. * **Behavior change**: Users assigned the [`job admin` permission set](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#job-admin) now have access to set up integrations for projects, including the [Tableau](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md) integration to populate downstream exposures. #### March 2025 * **Behavior change**: As of March 31st, 2025, dbt v1 versions 1.0, 1.1, and 1.2 have been deprecated from dbt. They are no longer available to select as versions for dbt projects. Workloads currently on these versions will be automatically upgraded to v1.3, which may cause new failures. * **Enhancement**: [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) users on single-tenant configurations no longer need to contact their account representative to enable this feature. Setup is now self-service and available across all tenant configurations. * **New**: The Semantic Layer now supports Postgres as a data platform. For more details on how to set up the Semantic Layer for Postgres, see [Set up the Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md). * **New**: New [environment variable default](https://docs.getdbt.com/docs/build/environment-variables.md#dbt-cloud-context) `DBT_CLOUD_INVOCATION_CONTEXT`. * **Enhancement**: Users assigned [read-only licenses](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#licenses) are now able to view the [Deploy](https://docs.getdbt.com/docs/deploy/deployments.md) section of their dbt account and click into the individual sections but not edit or otherwise make any changes. ###### dbt Developer day The following features are new or enhanced as part of our [dbt Developer day](https://www.getdbt.com/resources/webinars/dbt-developer-day) on March 19th and 20th, 2025: * **New**: The [`--sample` flag](https://docs.getdbt.com/docs/build/sample-flag.md), now available for the `run` and `build` commands, helps reduce build times and warehouse costs by running dbt in sample mode. It generates filtered refs and sources using time-based sampling, allowing developers to validate outputs without building entire models. * **New**: dbt Wizard, an AI-powered assistant, is now generally available in the Cloud IDE for all dbt Enterprise accounts. Check out [dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-platform.md) for more information. ###### Also available this month * **New**: Bringing your own [Azure OpenAI key](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md#manage-ai-features) for [dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-platform.md) is now generally available. Your organization can configure dbt Wizard to use your own Azure OpenAI keys, giving you more control over data governance and billing. * **New**: The Semantic Layer supports Power BI as a [partner integration](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md), available in private beta. To join the private beta, please reach out to your account representative. Check out the [Power BI](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/power-bi.md) integration for more information. * **New**: [dbt release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) are Generally Available. Depending on their plan, customers may select among the Latest, Compatible, or Extended tracks to manage the update cadences for development and deployment environments. * **New:** The dbt-native integration with Azure DevOps now supports [Entra ID service principals](https://docs.getdbt.com/docs/platform/git/setup-service-principal.md). Unlike a services user, which represents a real user object in Entra ID, the service principal is a secure identity associated with your dbt app to access resources in Azure unattended. Please [migrate your service user](https://docs.getdbt.com/docs/platform/git/setup-service-principal.md#migrate-to-service-principal) to a service principal for Azure DevOps as soon as possible. #### February 2025 * **Enhancement**: The [Python SDK](https://docs.getdbt.com/docs/dbt-apis/sl-python.md) added a new timeout parameter to Semantic Layer client and to underlying GraphQL clients to specify timeouts. Set a timeout number or use the `total_timeout` parameter in the global `TimeoutOptions` to control connect, execute, and close timeouts granularly. `ExponentialBackoff.timeout_ms` is now deprecated. * **New**: The [Azure DevOps](https://docs.getdbt.com/docs/platform/git/connect-azure-devops.md) integration for Git now supports [Entra service principal apps](https://docs.getdbt.com/docs/platform/git/setup-service-principal.md) on dbt Enterprise accounts. Microsoft is enforcing MFA across user accounts, including service users, which will impact existing app integrations. This is a phased rollout, and dbt Labs recommends [migrating to a service principal](https://docs.getdbt.com/docs/platform/git/setup-service-principal.md#migrate-to-service-principal) on existing integrations once the option becomes available in your account. * **New**: Added the `dbt invocation` command to the [dbt CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md). This command allows you to view and manage active invocations, which are long-running sessions in the dbt CLI. For more information, see [dbt invocation](https://docs.getdbt.com/reference/commands/invocation.md?version=2.0). * **New**: Users can now switch themes directly from the user menu, available [in Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles.md). We have added support for **Light mode** (default), **Dark mode**, and automatic theme switching based on system preferences. The selected theme is stored in the user profile and will follow users across all devices. * Dark mode is currently available on the Developer plan and will be available for all [plans](https://www.getdbt.com/pricing) in the future. We’ll be rolling it out gradually, so stay tuned for updates. For more information, refer to [Change your dbt theme](https://docs.getdbt.com/docs/platform/about-platform/change-your-dbt-theme.md). * **Fix**: Semantic Layer errors in the Cloud IDE are now displayed with proper formatting, fixing an issue where newlines appeared broken or difficult to read. This fix ensures error messages are more user-friendly and easier to parse. * **Fix**: Fixed an issue where [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md) with no [exports](https://docs.getdbt.com/docs/build/saved-queries.md#configure-exports) would fail with an `UnboundLocalError`. Previously, attempting to process a saved query without any exports would cause an error due to an undefined relation variable. Exports are optional, and this fix ensures saved queries without exports don't fail. * **New**: You can now query metric alias in Semantic Layer [GraphQL](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md) and [JDBC](https://docs.getdbt.com/docs/dbt-apis/sl-jdbc.md) APIs. * For the JDBC API, refer to [Query metric alias](https://docs.getdbt.com/docs/dbt-apis/sl-jdbc.md#query-metric-alias) for more information. * For the GraphQL API, refer to [Query metric alias](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md#query-metric-alias) for more information. * **Enhancement**: Added support to automatically refresh access tokens when Snowflake's SSO connection expires. Previously, users would get the following error: `Connection is not available, request timed out after 30000ms` and would have to wait 10 minutes to try again. * **Enhancement**: The [`dbt_version` format](https://docs.getdbt.com/reference/commands/version.md#versioning) in dbt Cloud now better aligns with [semantic versioning rules](https://semver.org/). Leading zeroes have been removed from the month and day (`YYYY.M.D+<suffix>`). For example: * New format: `2024.10.8+996c6a8` * Previous format: `2024.10.08+996c6a8` --- ### About AI and dbt Analytics engineering requires more than code generation — it requires understanding your data, how it connects, and what breaks when something changes. dbt brings purpose-built AI to that workflow, grounded in your project's lineage, tests, contracts, and metric definitions. dbt Wizard is the recommended agent for dbt work dbt Wizard is the recommended AI agent for governed data development in dbt. It handles the full development lifecycle — investigation, building, validation, and shipping — grounded in your dbt project's lineage, tests, contracts, and metric definitions. Refer to [dbt AI FAQs](https://docs.getdbt.com/docs/dbt-ai/dbt-ai-faqs.md#is-dbt-wizard-the-same-as-dbt-copilot), [Billing](https://docs.getdbt.com/docs/dbt-ai/wizard-billing-faqs.md), and [dbt's Terms of Use](https://www.getdbt.com/terms-of-use) for more information. #### Explore dbt Wizard and AI [![](/img/icons/wizard.svg)](https://docs.getdbt.com/docs/platform/wizard-overview.md) ###### [About dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-overview.md) [Purpose-built AI for governed data development in the dbt platform and CLI.](https://docs.getdbt.com/docs/platform/wizard-overview.md) [![](/img/icons/wizard.svg)](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) ###### [Use dbt Wizard locally](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) [Install dbt Wizard locally and start a terminal session.](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) [![](/img/icons/wizard.svg)](https://docs.getdbt.com/docs/platform/wizard-platform.md) ###### [dbt Wizard in dbt platform](https://docs.getdbt.com/docs/platform/wizard-platform.md) [AI is on by default in your dbt platform account. Use dbt Wizard for governed dbt data development and dbt Copilot for inline AI assistance.](https://docs.getdbt.com/docs/platform/wizard-platform.md) [![](/img/icons/wizard.svg)](https://docs.getdbt.com/docs/dbt-ai/wizard-migrate.md) ###### [Move to dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-migrate.md) [Move from Claude Code, Cursor, or another AI agent to dbt Wizard CLI with native project context.](https://docs.getdbt.com/docs/dbt-ai/wizard-migrate.md) #### Explore dbt Copilot [![](/img/icons/dbt-copilot.svg)](https://docs.getdbt.com/docs/dbt-ai/copilot-overview.md) ###### [dbt Copilot](https://docs.getdbt.com/docs/dbt-ai/copilot-overview.md) [dbt's inline AI assistance for single-click generation of SQL, docs, tests, and semantic models.](https://docs.getdbt.com/docs/dbt-ai/copilot-overview.md) [![](/img/icons/dbt-copilot.svg)](https://docs.getdbt.com/docs/dbt-ai/analyst-agent.md) ###### [dbt Copilot in Insights](https://docs.getdbt.com/docs/dbt-ai/analyst-agent.md) [Chat with your data using dbt Copilot in Insights, powered by the Semantic Layer.](https://docs.getdbt.com/docs/dbt-ai/analyst-agent.md) #### Explore the dbt MCP server [![](/img/icons/integration.svg)](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md) ###### [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md) [Bring dbt structured context into any MCP-compatible AI tool.](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md) --- ### About continuous integration (CI) in dbt dbt platform Use [CI jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md) in dbt to set up automation for testing code changes before merging to production. Additionally, [enable Advanced CI features](https://docs.getdbt.com/docs/platform/account-settings.md#account-access-to-advanced-ci-features) for these jobs to evaluate whether the code changes are producing the appropriate data changes you want by reviewing the comparison differences dbt provides. Refer to the guide [Get started with continuous integration tests](https://docs.getdbt.com/guides/set-up-ci.md?step=1) for more information. [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/continuous-integration.md) ###### [Continuous integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md) [Set up CI checks to test every single change prior to deploying the code to production.](https://docs.getdbt.com/docs/deploy/continuous-integration.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/advanced-ci.md) ###### [Advanced CI](https://docs.getdbt.com/docs/deploy/advanced-ci.md) [Compare the differences between what's in the production environment and the pull request before merging those changes, ensuring that you're always shipping trusted data products.](https://docs.getdbt.com/docs/deploy/advanced-ci.md) <br /> --- ### About dbt Insights Beta ### About dbt Insights [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt platform | Enterprise, Enterprise+ Learn how to query data with Insights and view documentation in Catalog. Insights in dbt empowers users to seamlessly explore and query data with an intuitive, context-rich interface. It bridges technical and business users by combining metadata, documentation, AI-assisted tools, and powerful querying capabilities into one unified experience. Insights in dbt integrates with [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md), [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md), [Canvas](https://docs.getdbt.com/docs/platform/canvas.md), [dbt Copilot in Insights](https://docs.getdbt.com/docs/explore/access-dbt-insights.md), and [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) to make it easier for you to perform exploratory data analysis, leverage AI-assisted tools, make faster decisions, and collaborate across teams. [![Overview of the dbt Insights and its features](/img/docs/dbt-insights/insights-main.gif?v=2 "Overview of the dbt Insights and its features")](#)Overview of the dbt Insights and its features #### Prerequisites * Be on a dbt [Enterprise or Enterprise+](https://www.getdbt.com/pricing) plan — [book a demo](https://www.getdbt.com/contact) to learn more about Insights. * Available on all [tenant](https://docs.getdbt.com/docs/platform/about-platform/tenancy.md) configurations. * Have a dbt [developer license](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md) with access to Insights. * Configured [user credentials](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md#get-started-with-the-studio-ide). * Your production and development [environments](https://docs.getdbt.com/docs/dbt-platform-environments.md) are on dbt’s **v1 Latest** [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) or a supported dbt version. * Use a supported data platform: Snowflake, BigQuery, Databricks, Redshift, or Postgres. * Single sign-on (SSO) for development user accounts is supported. Deployment environments will be queried leveraging the user's user credentials configured in **Account settings**. * (Optional) — To query [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) metrics from the Insights, you must also: * [Configure](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md) the Semantic Layer for your dbt project. * Have a successful job run in the environment where you configured the Semantic Layer. * (Optional) To enable [Language Server Protocol (LSP) features](https://docs.getdbt.com/docs/explore/navigate-dbt-insights.md#lsp-features-in-dbt-insights) in Insights and run your compilations on dbt v2, set your development environment to use the **v2 Stable** dbt version. #### Key benefits Key benefits include: * Quickly write, run, and iterate on SQL queries with tools like syntax highlighting, tabbed editors, and query history. * Leverage dbt metadata, trust signals, and lineage from Catalog for informed query construction. * Make data accessible to users of varied technical skill levels with SQL, Semantic Layer queries, and visual tools. * Use dbt Copilot in Insights to generate or edit SQL queries, descriptions, and more. Some example use cases include: * Analysts can quickly construct queries to analyze sales performance metrics across regions and view results. * All users have a rich development experience powered by Catalog's end-to-end exploration experience. dbt Wizard is the recommended agent for dbt work dbt Wizard is the recommended AI agent for governed data development in dbt. It handles the full development lifecycle — investigation, building, validation, and shipping — grounded in your dbt project's lineage, tests, contracts, and metric definitions. Refer to [dbt AI FAQs](https://docs.getdbt.com/docs/dbt-ai/dbt-ai-faqs.md#is-dbt-wizard-the-same-as-dbt-copilot), [Billing](https://docs.getdbt.com/docs/dbt-ai/wizard-billing-faqs.md), and [dbt's Terms of Use](https://www.getdbt.com/terms-of-use) for more information. --- ### About dbt integrations Many data applications integrate with dbt, enabling you to leverage the power of dbt for a variety of use cases and workflows. #### Integrations with dbt [![](/img/icons/vsce.svg)](https://docs.getdbt.com/docs/about-dbt-extension.md) ###### [dbt VS Code extension](https://docs.getdbt.com/docs/about-dbt-extension.md) [The dbt extension brings a hyper-fast, intelligent, and cost-efficient dbt development experience to VS Code. The best way to experience all the power of the new dbt v2 while developing locally.](https://docs.getdbt.com/docs/about-dbt-extension.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures.md) ###### [Visualize and orchestrate downstream exposures](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures.md) [Configure downstream exposures automatically from dashboards and understand how models are used in downstream tools. Proactively refresh the underlying data sources during scheduled dbt jobs.](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) ###### [dbt Semantic layer integrations](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) [Review a wide range of partners you can integrate and query with the dbt Semantic Layer.](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) --- ### About dbt LSP dbt v2 offers benefits beyond the speed and power of the framework. The dbt VS Code extension, Studio IDE, and Insights all contain a powerful set of features backed by our Language Server Protocol (LSP) that enable fast, efficient development workflows. The following features are supported across these tools: | | VS Code extension | Studio IDE | Insights | | ---------------------------- | ----------------- | ---------- | -------- | | Autocomplete function names | ✅ | ✅ | ❌ | | Autocomplete ref/source args | ✅ | ✅ | ✅ | | CTE Preview | ✅ | ✅ | ✅ | | Column-level lineage | ✅ | ❌ | ❌ | | Compare changes locally | ✅ | ❌ | ❌ | | Command palette | ✅ | N/A | ❌ | | Error detection | ✅ | ✅ | ✅ | | Go-to definition | ✅ | ✅ | ❌ | | Go-to reference | ✅ | ✅ | ❌ | | Incremental compilation | ✅ | ✅ | ❌ | | Lazy compilation | ✅ | ✅ | ❌ | | Preview query results | ✅ | N/A | ❌ | | Problems tab | ✅ | ✅ | ❌ | | Propagate column renames | ✅ | ❌ | ❌ | | Propagate model renames | ✅ | ❌ | ❌ | | Show column type on hover | ✅ | ✅ | ✅ | | Show compiled SQL | ✅ | ✅ | ❌ | | View table lineage | ✅ | N/A | ❌ | | Warning detection | ✅ | ✅ | ❌ | dbt v1 and v2 both support the dbt VS Code extension. Features that depend on built-in SQL parsing, such as column-level lineage, column go-to definition and find references, and `SELECT *` expansion, require [`static_analysis: strict`](https://docs.getdbt.com/reference/resource-configs/static-analysis.md?version=2). #### Lazy compilation The dbt language server uses on-demand compilation, also called lazy compilation. Lazy compilation starts automatically when you open a model file, you don't need to run `dbt compile` to trigger it. It compiles only the nodes it needs to answer questions about the file you are working in, instead of blocking on a full project compile first. That improves performance because you get editor features for your active file much sooner. ##### What compiles first When you open or focus on a model, the server determines a minimal set of nodes to compile so it can produce up-to-date LSP results for that model. That set includes the current model and its upstream dependencies (ancestors in the DAG), because rendered SQL and analysis depend on `ref`, sources, and inherited context from parents. Nodes you are not actively working on remain `not compiled` until the background compilation pass reaches them. How long that takes depends on the size of your project. Until a node is compiled, LSP results for that node are not available. When you switch to another file, the server reuses results from any compilations that already finished. If a compilation was still in progress when you switched files, it is cancelled and that partial work is discarded; the server then schedules a fresh compile for the newly focused model and its dependencies. ##### Background compilation After the minimal compile for your active file, the server continues with a background compile of the rest of the project. That pass fills in project-wide state without preventing you from using tooling on models that already finished compiling. Background compilation enables full project analysis once it completes. Until then, some features that need the full graph may be limited. You can monitor compilation progress in your editor's status bar. When the progress notifications clear, the background compile is complete. dbt and the language server run independently. Running a command like `dbt run` or `dbt compile` from the terminal does not interrupt or affect LSP compilation. #### Versioning The v2 CLI and the language server ship in a single binary, so they always share the same version and can't be mismatched. The dbt VS Code extension declares which dbt versions it supports and checks the installed binary at startup. For the compatibility matrix and how to verify binaries for manual or air-gapped installs, refer to [Version compatibility](https://docs.getdbt.com/docs/dbt-versions/dbt-version-compatibility.md). --- ### About dbt Mesh Organizations of all sizes rely upon dbt to manage their data transformations, from small startups to large enterprises. At scale, it can be challenging to coordinate all the organizational and technical requirements demanded by your stakeholders within the scope of a single dbt project. To date, there also hasn't been a first-class way to effectively manage the dependencies, governance, and workflows between multiple dbt projects. That's where **Mesh** comes in - empowering data teams to work *independently and collaboratively*; sharing data, code, and best practices without sacrificing security or autonomy. Mesh is not a single product - it is a pattern enabled by a convergence of several features in dbt: * **[Cross-project references](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#how-to-write-cross-project-ref)** - this is the foundational feature that enables the multi-project deployments. `{{ ref() }}`s now work across dbt projects on Enterprise and Enterprise+ plans. * **[Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md)** - dbt's metadata-powered documentation platform, complete with full, cross-project lineage. * **Governance** - dbt's governance features allow you to manage access to your dbt models both within and across projects. * **[Groups](https://docs.getdbt.com/docs/mesh/govern/model-access.md#groups)** - With groups, you can organize nodes in your dbt DAG that share a logical connection (for example, by functional area) and assign an owner to the entire group. * **[Access](https://docs.getdbt.com/docs/mesh/govern/model-access.md#access-modifiers)** - access configs allow you to control who can reference models. * **[Model Versions](https://docs.getdbt.com/docs/mesh/govern/model-versions.md)** - when coordinating across projects and teams, we recommend treating your data models as stable APIs. Model versioning is the mechanism to allow graceful adoption and deprecation of models as they evolve. * **[Model Contracts](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md)** - data contracts set explicit expectations on the shape of the data to ensure data changes upstream of dbt or within a project's logic don't break downstream consumers' data products. #### When is the right time to use dbt Mesh? The multi-project architecture helps organizations with mature, complex transformation workflows in dbt increase the flexibility and performance of their dbt projects. If you're already using dbt and your project has started to experience any of the following, you're likely ready to start exploring this paradigm: * The **number of models** in your project is degrading performance and slowing down development. * Teams have developed **separate workflows** and need to decouple development from each other. * Teams are experiencing **communication challenges**, and the reliability of some of your data products has started to deteriorate. * **Security and governance** requirements are increasing and would benefit from increased isolation. dbt is designed to coordinate the features above and simplify the complexity to solve for these problems. If you're just starting your dbt journey, don't worry about building a multi-project architecture right away. You can *incrementally* adopt the features as you scale. The collection of features work effectively as independent tools. Familiarizing yourself with the tooling and features that make up a multi-project architecture, and how they can apply to your organization will help you make better decisions as you grow. For additional information, refer to the [Mesh FAQs](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-5-faqs.md). --- ### About dbt Model Context Protocol (MCP) server The [dbt MCP server](https://github.com/dbt-labs/dbt-mcp) provides a standardized framework that lets you integrate AI applications with dbt‑managed data assets across different data platforms. This ensures consistent, governed access to models, metrics, lineage, and freshness across your AI tools. To help with dbt, assistants need your project metadata and, when you allow it, supported actions such as CLI runs, platform APIs, and Semantic Layer queries. The dbt MCP server exposes those to MCP clients and supports use cases such as conversational access to data, agentic automation for dbt workflows, and AI-assisted development. This page covers local and remote setups, available tools, and how to get started. The MCP server provides access to [dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-overview.md), dbt platform CLI, [API](https://docs.getdbt.com/docs/dbt-apis/overview.md), the [Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md), and [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md). It provides access to private APIs, text-to-SQL, and SQL execution. For more information on MCP, have a look at [Get started with the Model Context Protocol (MCP)](https://modelcontextprotocol.io/introduction). The dbt MCP server comes in two flavors: self-hosted and remote. * [Self-hosted MCP server](#local-mcp-server): runs on your machine and requires installing `uvx` (which installs dbt-mcp locally). * [Remote MCP server](#remote-mcp-server): uses an HTTP connection and makes calls to dbt-mcp remotely on the managed dbt platform. This is a dbt-hosted solution and is ideal for data consumption use cases. For more details on the server types, refer to [Server access](#server-access). #### Get started To get started, choose the quickstart that matches your setup: | I want to... | Quickstart | Tool access | | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | ------------------------------------------------- | | Query data and run dbt platform CLI commands locally while connected to my dbt platform account (Semantic Layer, Discovery API, Admin API, SQL, Codegen). | [Connect to dbt platform](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-oauth.md) | Uses [self-hosted MCP server](#local-mcp-server). | | Run dbt platform CLI commands locally, with or without a dbt platform account; with an account, also query data and explore metadata through the same server. | [Run dbt locally](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-cli.md) | Uses [self-hosted MCP server](#local-mcp-server). | | Use MCP with zero installation (query data only through dbt-hosted tools; no dbt platform CLI commands). | [Connect to the remote dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-remote.md) | Uses [remote MCP server](#remote-mcp-server). | To configure or disable specific tools (self-hosted MCP), see the [Environment variables reference](https://docs.getdbt.com/docs/dbt-ai/mcp-environment-variables.md). #### Server access You can use the dbt MCP server in the following ways: * [Self-hosted MCP server](#local-mcp-server) — runs on your machine and requires installing `uvx` (which installs dbt-mcp locally) and then running `uvx dbt-mcp` to start the server. No need to clone the repo unless you want to contribute to [dbt MCP server](https://github.com/dbt-labs/dbt-mcp). * [Remote MCP server](#remote-mcp-server) — uses an HTTP connection and makes calls to dbt-mcp hosted on the managed dbt platform. This setup requires no self-hosted installation and is ideal for data consumption use cases. ##### Self-hosted MCP server The self-hosted MCP server provides the best experience for development workflows, like authoring dbt models, tests, and documentation. The [self-hosted MCP server](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md) runs on your machine and requires installing `uvx` (which installs dbt-mcp locally) and then running `uvx dbt-mcp` to start the server. You don't need to clone the repository unless you want to contribute to dbt MCP. The self-hosted MCP server provides: * Full access to dbt commands (`dbt run`, `dbt build`, `dbt test`, and more) * Support for dbt v1, dbt platform CLI, and dbt v2 * Ability to work with local dbt projects with or without a dbt platform account * Optional integration with dbt platform APIs for metadata discovery and Semantic Layer access ##### Remote MCP server The remote MCP server from dbt offers data consumption use cases without a self-hosted setup. It doesn't support local development or dbt platform CLI commands; use the [self-hosted MCP server](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md) for those workflows. The [remote MCP server](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md) connects to the dbt platform via HTTP and requires no installation. This option is useful when: * You either don’t want to install, or are restricted from installing, additional software on your system. * Your use case is primarily consumption-based (for example, querying metrics, exploring metadata, viewing lineage). The remote MCP server is available on all dbt platform [plans](https://www.getdbt.com/pricing). However, the underlying [dbt APIs](https://docs.getdbt.com/docs/dbt-apis/overview.md) that the server's tools rely on vary by plan type. For example, the Discovery API and Semantic Layer APIs. As a result, the tools available to you through the remote MCP server depend on your plan. Remote MCP uses the same default global API rate limit as other dbt APIs (5,000 requests per minute per IP). Self-hosted MCP is subject to the public Administrative and Discovery API limits. Refer to [API rate limits](https://docs.getdbt.com/docs/dbt-apis/rate-limits.md) for details. info Only [`text_to_sql`](https://docs.getdbt.com/docs/dbt-ai/mcp-available-tools.md) consumes your dbt Copilot action allotment. Other MCP tools do not. When your account runs out of dbt Copilot actions, the remote MCP server blocks every tool that runs through it, including tools invoked from a self-hosted MCP server and [proxied](https://github.com/dbt-labs/dbt-mcp/blob/main/src/dbt_mcp/tools/toolsets.py#L24) to remote MCP, such as SQL and remote dbt v2 tools. If you reach your dbt Copilot actions limit, remote MCP tools remain unavailable until the limit resets. If you need help, contact your account manager. ##### Supported tools by MCP server type The dbt MCP server has access to many parts of the dbt experience related to development, deployment, and discovery. Here are the categories of tools supported based on what form of the MCP server you connect to as well as detailed information on exact commands or queries available to the LLM. Self-hosted MCP is required for dbt platform CLI commands, Codegen, and Administrative API; remote MCP supports Semantic Layer, SQL, Discovery, Administrative API, and dbt v2 tools only. Note that access to the [dbt APIs](https://docs.getdbt.com/docs/dbt-apis/overview.md) is limited depending on your [plan type](https://www.getdbt.com/pricing). | Tools | Self-hosted | Remote | | ------------------------- | ----------- | ------ | | dbt platform CLI commands | ✅ | ❌ | | Semantic Layer | ✅ | ✅ | | SQL | ✅ | ✅ | | Metadata Discovery | ✅ | ✅ | | Administrative API | ✅ | ✅ | | Codegen Tools | ✅ | ❌ | | dbt v2 Tools | ✅ | ✅ | | Product Docs Tools | ✅ | ✅ | | MCP Server Metadata Tools | ✅ | ✅ | #### Available tools The dbt MCP server has access to many tools related to development, deployment, and discovery — like CLI A full list of tools is available for your MCP server and is auto-fetched from the [dbt MCP server README on GitHub](https://github.com/dbt-labs/dbt-mcp#tools) when the docs are built, so it stays in sync with each release. To view the full list of tools, see [Available tools](https://docs.getdbt.com/docs/dbt-ai/mcp-available-tools.md). #### MCP integrations The dbt MCP server integrates with any [MCP client](https://modelcontextprotocol.io/clients) that supports OAuth or token authentication and tool use capabilities, depending on your setup. We have created integration guides for the following clients: * [Claude](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-claude.md) * [Cursor](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-cursor.md) * [VS Code](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-vscode.md) * [Snowflake Cortex](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-snowflake-cortex.md) #### Data retention The dbt MCP server doesn't store or retain any production data or job run results. It's a read-only access layer that reads metadata, job results, and Semantic Layer data from the dbt platform in real time when a tool is called. Your [dbt platform data retention policy](https://www.getdbt.com/security) determines how long job runs and artifacts are available, not the MCP server. As long as a job or artifact exists in dbt platform, the MCP server can read it. #### Resources * [Environment variables reference](https://docs.getdbt.com/docs/dbt-ai/mcp-environment-variables.md) — full list of variables and tool configuration for self-hosted MCP * For more information, refer to our blog on [Introducing the dbt MCP Server](https://docs.getdbt.com/blog/introducing-dbt-mcp-server#getting-started). --- ### About dbt models Self-hosted and cloud-hosted dbt are composed of different moving parts working harmoniously. All of them are important to what dbt does — transforming data—the 'T' in ELT. When you execute `dbt run`, you are running a model that will transform your data without that data ever leaving your warehouse. Models are where your developers spend most of their time within a dbt environment. Models are primarily written as a `select` statement and saved as a `.sql` file. While the definition is straightforward, the complexity of the execution will vary from environment to environment. Models will be written and rewritten as needs evolve and your organization finds new ways to maximize efficiency. SQL is the language most dbt users will utilize, but it is not the only one for building models. Starting in version 1.3, dbt v1 and dbt support Python models. Python models are useful for training or deploying data science models, complex transformations, or where a specific Python package meets a need — such as using the `dateutil` library to parse dates. ##### Models and modern workflows The top level of a dbt workflow is the project. A project is a directory of a `.yml` file (the project configuration) and either `.sql` or `.py` files (the models). The project file tells dbt the project context, and the models let dbt know how to build a specific data set. For more details on projects, refer to [About dbt projects](https://docs.getdbt.com/docs/build/projects.md). Your organization may need only a few models, but more likely you’ll need a complex structure of nested models to transform the required data. A model is a single file containing a final `select` statement, and a project can have multiple models, and models can even reference each other. Add to that, numerous projects and the level of effort required for transforming complex data sets can improve drastically compared to older methods. Learn more about models in [SQL models](https://docs.getdbt.com/docs/build/sql-models.md) and [Python models](https://docs.getdbt.com/docs/build/python-models.md) pages. If you'd like to begin with a bit of practice, visit our [Getting Started Guide](https://docs.getdbt.com/guides.md) for instructions on setting up the Jaffle\_Shop sample data so you can get hands-on with the power of dbt. --- ### About dbt projects A dbt project informs dbt about the context of your project and how to transform your data (build your data sets). By design, dbt enforces the top-level structure of a dbt project such as the `dbt_project.yml` file, the `models` directory, the `snapshots` directory, and so on. Within the directories of the top-level, you can organize your project in any way that meets the needs of your organization and data pipeline. At a minimum, all a project needs is the `dbt_project.yml` project configuration file. dbt supports a number of different resources, so a project may also include: | Resource | Description | | ------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [models](https://docs.getdbt.com/docs/build/models.md) | Each model lives in a single file and contains logic that either transforms raw data into a dataset that is ready for analytics or, more often, is an intermediate step in such a transformation. | | [snapshots](https://docs.getdbt.com/docs/build/snapshots.md) | A way to capture the state of your mutable tables so you can refer to it later. | | [seeds](https://docs.getdbt.com/docs/build/seeds.md) | CSV files with static data that you can load into your data platform with dbt. | | [data tests](https://docs.getdbt.com/docs/build/data-tests.md) | SQL queries that you can write to test the models and resources in your project. | | [macros](https://docs.getdbt.com/docs/build/jinja-macros.md) | Blocks of code that you can reuse multiple times. | | [docs](https://docs.getdbt.com/docs/build/documentation.md) | Docs for your project that you can build. | | [sources](https://docs.getdbt.com/docs/build/sources.md) | A way to name and describe the data loaded into your warehouse by your Extract and Load tools. | | [exposures](https://docs.getdbt.com/docs/build/exposures.md) | A way to define and describe a downstream use of your project. | | [metrics](https://docs.getdbt.com/docs/build/build-metrics-intro.md) | A way for you to define metrics for your project. | | [groups](https://docs.getdbt.com/docs/build/groups.md) | Groups enable collaborative node organization in restricted collections. | | [analysis](https://docs.getdbt.com/docs/build/analyses.md) | A way to organize analytical SQL queries in your project such as the general ledger from your QuickBooks. | | [semantic models](https://docs.getdbt.com/docs/build/semantic-models.md) | Semantic models define the foundational data relationships in [MetricFlow](https://docs.getdbt.com/docs/build/about-metricflow.md) and the [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md), enabling you to query metrics using a semantic graph. | | [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md) | Saved queries organize reusable queries by grouping metrics, dimensions, and filters into nodes visible in the dbt DAG. | | [user-defined functions](https://docs.getdbt.com/docs/build/udfs.md) | User-defined functions (UDFs) let you create reusable custom functions in your warehouse, shareable across dbt, BI tools, data science workflows, and more. | When building out the structure of your project, you should consider these impacts on your organization's workflow: * **How would people run dbt commands** — Selecting a path * **How would people navigate within the project** — Whether as developers in the Studio IDE or stakeholders from the docs * **How would people configure the models** — Some bulk configurations are easier done at the directory level so people don't have to remember to do everything in a config block with each new model #### Project configuration Every dbt project includes a project configuration file called `dbt_project.yml`. It defines the directory of the dbt project and other project configurations. Edit `dbt_project.yml` to set up common project configurations such as: | YAML key | Value description | | ----------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | | [name](https://docs.getdbt.com/reference/project-configs/name.md) | Your project’s name in [snake case](https://en.wikipedia.org/wiki/Snake_case) | | [version](https://docs.getdbt.com/reference/project-configs/version.md) | Version of your project | | [require-dbt-version](https://docs.getdbt.com/reference/project-configs/require-dbt-version.md) | Restrict your project to only work with a range of [dbt v1 versions](https://docs.getdbt.com/docs/dbt-versions.md) | | [profile](https://docs.getdbt.com/reference/project-configs/profile.md) | The profile dbt uses to connect to your data platform | | [model-paths](https://docs.getdbt.com/reference/project-configs/model-paths.md) | Directories to where your model and source files live | | [seed-paths](https://docs.getdbt.com/reference/project-configs/seed-paths.md) | Directories to where your seed files live | | [test-paths](https://docs.getdbt.com/reference/project-configs/test-paths.md) | Directories to where your test files live | | [analysis-paths](https://docs.getdbt.com/reference/project-configs/analysis-paths.md) | Directories to where your analyses live | | [macro-paths](https://docs.getdbt.com/reference/project-configs/macro-paths.md) | Directories to where your macros live | | [snapshot-paths](https://docs.getdbt.com/reference/project-configs/snapshot-paths.md) | Directories to where your snapshots live | | [docs-paths](https://docs.getdbt.com/reference/project-configs/docs-paths.md) | Directories to where your docs blocks live | | [vars](https://docs.getdbt.com/docs/build/project-variables.md) | Project variables you want to use for data compilation | For complete details on project configurations, see [dbt\_project.yml](https://docs.getdbt.com/reference/dbt_project.yml.md). #### Project subdirectories You can use the Project subdirectory option in dbt to specify a subdirectory in your git repository that dbt should use as the root directory for your project. This is helpful when you have multiple dbt projects in one repository or when you want to organize your dbt project files into subdirectories for easier management. To use the Project subdirectory option in dbt, follow these steps: 1. Click your account name in the bottom left and select **Your profile**. 2. Under **Projects**, select the project you want to configure as a project subdirectory. 3. Select **Edit** on the lower right-hand corner of the page. 4. In the **Project subdirectory** field, add the name of the subdirectory. For example, if your project YAML files are located in a subdirectory called `<repository>/finance`, you would enter `finance` as the subdirectory. * You can also reference nested subdirectories. For example, if your project YAML files are located in `<repository>/teams/finance`, you would enter `teams/finance` as the subdirectory. **Note**: You do not need a leading or trailing `/` in the Project subdirectory field. 5. Click **Save** when you've finished. After configuring the Project subdirectory option, dbt will use it as the root directory for your dbt project. This means that dbt commands, such as `dbt run` or `dbt test`, will operate on files within the specified subdirectory. If there is no `dbt_project.yml` file in the Project subdirectory, you will be prompted to initialize the dbt project. Project support in dbt plans Some [plans](https://www.getdbt.com/pricing) support only one dbt project, while [Enterprise-tier plans](https://www.getdbt.com/contact) allow multiple projects and [cross-project references](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md) with Mesh. #### New projects You can create new projects and [share them](https://docs.getdbt.com/docs/platform/git/git-version-control.md) with other people by making them available on a hosted git repository like GitHub, GitLab, and BitBucket. After you set up a connection with your data platform, you can [initialize your new project in dbt](https://docs.getdbt.com/guides.md) and start developing. Or, run [dbt init from the command line](https://docs.getdbt.com/reference/commands/init.md) to set up your new project. During project initialization, dbt creates sample model files in your project directory to help you start developing quickly. #### Sample projects If you want to explore dbt projects more in-depth, refer to [Clone the Jaffle Shop sample project](https://docs.getdbt.com/guides/clone-jaffle-shop.md). It's a runnable project that contains sample configurations and helpful notes. If you want to see what a mature, production project looks like, check out the [GitLab Data Team public repo](https://gitlab.com/gitlab-data/analytics/-/tree/master/transform/snowflake-dbt). #### Related docs * [Best practices: How we structure our dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md) * [Quickstarts for dbt](https://docs.getdbt.com/guides.md) * [Quickstart for dbt v1](https://docs.getdbt.com/guides/manual-install.md) --- ### About dbt self-hosted installations Local development You can run dbt locally from your terminal with the dbt CLI, or from your code editor with the dbt VS Code extension. Local development lets you build, test, and run dbt projects from your own machine while connecting to your data platform. (Applies to dbt v2.0 and later) #### Install dbt Install dbt locally to run dbt commands from your terminal. You can install it with `pip`: ```shell python -m pip install --pre dbt ``` Or with `curl`: ```shell curl -fsSL https://public.cdn.getdbt.com/fs/install/install.sh | sh ``` For full installation instructions, including adapter-specific setup, refer to [Install dbt](https://docs.getdbt.com/docs/local/install-dbt.md). #### dbt VS Code extension The [dbt VS Code extension](https://docs.getdbt.com/docs/about-dbt-extension.md) lets you develop dbt projects from VS Code, Cursor, or Windsurf. Use the extension if you want an editor-based local development experience. For installation and setup, refer to the [extension docs](https://docs.getdbt.com/docs/about-dbt-extension.md). #### dbt Wizard [dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) is a natural next step for local dbt development. It works with dbt and adds an AI agent that understands your full project through dbt's [native metadata engine](https://docs.getdbt.com/docs/dbt-ai/about-dbt-ai.md), a structured index of your [lineage](https://docs.getdbt.com/docs/explore/explore-projects.md), model health, test coverage, and semantic definitions. * **Build and refactor from natural language:** Describe the change, get a reviewable diff, approve before anything is written. * **Validate in a tight loop:** Every proposed change compiles and runs against your warehouse, catching issues before production. * **Navigate with full project context:** Traverse the [DAG](https://docs.getdbt.com/docs/explore/explore-projects.md), surface downstream impact, and keep tests and YAML in sync as models evolve. For data practitioners, dbt Wizard adds an AI layer that knows your project, not just your code. Refer to the [dbt Wizard quickstart](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) to get started. #### dbt MCP server The dbt MCP server connects your local dbt project to AI assistants using the [Model Context Protocol](https://modelcontextprotocol.io/). It works with dbt and requires no repository clone. * **dbt platform CLI tools:** Run `dbt run`, `build`, `test`, `compile`, `list`, `parse`, and `show` directly from your AI assistant's chat interface. * **Local project context:** Surface model lineage, node details, and dependency graphs from your local `manifest.json` without leaving your editor. * **Code generation:** Auto-generate model YAML, source definitions, and staging SQL from your warehouse schema (requires the codegen toolset to be enabled). * **Zero-clone install:** Install [uv](https://docs.astral.sh/uv/getting-started/installation/) and run `uvx dbt-mcp`. No repository clone needed! [Connect dbt MCP server to your local project](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-cli.md). #### Licensing info dbt framework has two distributions which can both be installed locally for free, powered by a single engine: * The Apache 2.0 licensed open-source distribution. Both v1 and v2 are available as open source installations. * dbt v2 extends the dbt OSS offering with additional advanced capabilities — all free to use! Refer to [licensing](https://docs.getdbt.com/docs/dbt-licensing.md) for more info. --- ### About dbt State Preview ### About dbt State [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Login required | Usage-based dbt State makes dbt smarter about what to build. Instead of rebuilding every node on every run, dbt reuses nodes by cloning from another location or skipping a rebuild when the logic and data haven't changed. With dbt State, dbt first compares the logic and data of each node to previous builds across multiple environments on every run — whether orchestrated in the dbt platform, through your own orchestrator, or in development. If the logic is the same and the data is still fresh, dbt reuses an existing object. It will either clone an existing node from elsewhere, or skip executing a model that already exists, rather than building it anew. Additionally, it will automatically defer to production state without the need to manually set the `--defer` or `--state` flags. dbt State can reuse all node types that create relations in the database (such as SQL models, snapshots, seeds) and data tests. Note that the following models are not eligible for reuse: * **Python models**: dbt State builds Python models on every run, even if their code and upstream data have not changed. * **Models with custom materializations**: dbt State builds these models on every run because custom materializations may have side effects (for example, modifying table properties or writing to other schemas), and dbt State cannot safely determine whether skipping the run would produce the same result. dbt State works with dbt (v1 and v2) and the dbt platform, across all environments and orchestrators, making it a flexible approach regardless of how you run dbt. It requires authentication through a dbt platform account. For pricing details, refer to [dbt State usage and pricing](https://docs.getdbt.com/docs/platform/billing/dbt-state-usage.md). #### Benefits dbt State delivers efficiency gains across both production and development environments: * **Fresher data, lower costs**: Nodes only rebuild when the result would be different (new data or code changes), reducing warehouse compute while keeping production data fresh. * **Faster iteration cycles**: In development, dbt automatically clones selected nodes from production whenever possible, so you spend less time waiting for builds and more time writing code. * **Smarter than standard deferral**: Unlike standard deferral, which always builds selected nodes and only defers unselected upstream references, dbt State decides whether transformations need to run at all, or whether an existing table can simply be cloned. * **Model-level freshness threshold**: The [`lag_tolerance`](https://docs.getdbt.com/reference/resource-configs/lag-tolerance.md) config sets how much time must pass since the last upstream data change before dbt triggers a rebuild. It decouples downstream models from high-frequency upstream changes, and prevents costly rebuilds on stagnant data when an upstream dependency misses its freshness [Service Level Agreement (SLA)](https://www.getdbt.com/blog/data-slas-best-practices). #### How dbt State works When you run a command like `dbt build --select +my_model`, dbt State evaluates each selected node and applies the most efficient approach it can: * **Reuse node from same schema (skip)** — dbt checks whether the object already exists in the target schema, its logic hasn't changed, and its upstream parents haven't received fresh data beyond the configured [`lag_tolerance`](https://docs.getdbt.com/reference/resource-configs/lag-tolerance.md). If all conditions are met, dbt skips the node entirely, as if it was never selected. For data tests, if the nodes being tested haven't changed since the last run, the previous test result is reused without re-executing the test query. For views, if the view's logic is unchanged, dbt State reuses it even if new data has arrived upstream. Because views don't store data, new upstream data is automatically reflected when the view is queried, even without a rebuild. Note that views using `select *` on an upstream node may behave differently — refer to [Views with `select *`](https://docs.getdbt.com/faqs/State/views-rebuilt.md#views-with-select) for more information. * **Reuse node from different schema (clone)** — dbt State looks across all environments and jobs for a matching object with identical logic and fresh data. This includes schemas where a model was built before it ever ran in production. When multiple candidates exist, dbt State clones from the one with the freshest data, regardless of which environment it came from. For example, if a CI schema has fresher data than production and identical logic, dbt State clones from there. The node is marked as **Reused** at a fraction of the compute cost. If you want to prevent cloning into a specific target (for example, in regulated environments), set [`allow_clones: false`](https://docs.getdbt.com/reference/resource-configs/allow-clones.md) on that target in `profiles.yml` or as an [extended attribute](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) in the dbt platform. * **Normal build** — If reuse is not possible, dbt builds the node as normal, automatically deferring any unselected upstream nodes. dbt State fetches table metadata (for example, last-modified timestamps) in the background at the start of each run. Any node ready to skip, clone, or execute proceeds immediately; nodes with an undetermined action wait for the fetch to complete. Without dbt State, every selected node rebuilds on every run regardless of whether anything has changed. To see which decision dbt State made for each node after a run and why, you can run the (Applies to dbt v2.0 and later) [`dbt state explain`](https://docs.getdbt.com/reference/commands/state-explain.md) command. If you use the dbt platform, the same information is available without running a command — go to the [**Explain** tab](https://docs.getdbt.com/docs/deploy/dbt-state-interface.md#explain-tab) on the job run details page to see the full decision breakdown for each node. For the full list of available configs, see [dbt State configs](https://docs.getdbt.com/reference/resource-configs/dbt-state-configs.md).  How dbt State decides whether to rebuild, clone, or reuse The following decision tree shows how dbt State chooses the most efficient valid action for each node. [![Decision tree showing how dbt State chooses whether to rebuild, clone, or reuse a node based on state bypasses, volatile SQL handling, execution hashes, freshness, schema matches, clone eligibility, and whether fresh upstream data can still be cloned from time travel or another schema](/img/docs/deploy/run-cache-decision-tree.png?v=2 "dbt State decision tree for rebuild, clone, and reuse")](#)dbt State decision tree for rebuild, clone, and reuse The key idea is that dbt State only skips work when it can prove the existing object is sufficiently equivalent for the current run. If the SQL logic, relevant config, schema, or upstream freshness means the result might be different, dbt rebuilds instead. #### Signing up for dbt State dbt State is connected to your existing dbt platform account. Your dbt State credentials are the same as your platform credentials, and dbt State has access to your platform environments and jobs. dbt State app retirement The standalone dbt State app (`app.state.dbt.com`) is being retired and is no longer accepting new users. If you have an existing dbt State app account, [create a free dbt platform account](http://us1.dbt.com/register?_dbtsrc=dbt-state) to continue using dbt State — your free trial will be extended upon account creation. #### FAQs What happened to state-aware orchestration? On June 1, 2026, dbt Labs and Fivetran announced **[dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md)**[Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") as a new and improved version of state-aware orchestration. A key feature is [`lag_tolerance`](https://docs.getdbt.com/reference/resource-configs/lag-tolerance.md), which controls how much time must pass since the last upstream data change before a node is eligible for a rebuild. dbt State improves upon state-aware orchestration in a few key ways: * **Works everywhere** — dbt State works with dbt v1, dbt v2, and dbt platform, as well as external orchestrators, across both development and deployment environments. * **Smarter data freshness tracking** — dbt State tracks data freshness across the DAG and automatically propagates it through models materialized as views. Unlike state-aware orchestration's `build_after` config which compares against the model's last successful execution, dbt State's `lag_tolerance` compares against the freshness of the underlying data. * **Advanced change detection** — dbt State can detect and ignore file modifications that don't change actual transformation logic, such as adding a comment or cleaning up whitespace. If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. For details on billing after the trial ends, refer to [dbt State usage and pricing](https://docs.getdbt.com/docs/platform/billing.md#dbt-state-usage). While dbt State is in preview, there is no required migration timeline — dbt Labs will communicate a timeline when dbt State reaches general availability. To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). How is dbt State different from using state:modified? `state:modified` in dbt v1 requires manual management of `manifest.json`, which is cumbersome and error-prone. dbt State is completely managed with almost zero setup and no workflow changes. `state:modified` only checks if a file has changed. dbt State has semantic understanding of SQL, so meaningless changes like whitespace or table aliases are not counted as a change — making dbt State smarter about what actually needs to rebuild. `state:modified` does not consider upstream data changes. dbt State checks all sources to see if there is any new data or if the schema has been modified. This enables dbt State to skip running models if the result of the run would be the same as before. For example, if you run `dbt run` with `state:modified` twice, it runs all modified models both times. dbt State only reruns models the second time if upstream sources have changed. dbt State also has the ability to auto-defer refs and automatically clone tables when the result of the clone would have been the same as a full model run. `state:modified` has a limitation on seed files over 1MB, while dbt State does not. Does dbt State support incremental models? dbt State works with incremental models. When you make a change to an incremental model and run it in development, dbt State automatically clones the model from production if it exists, then runs the new model logic on top of the clone. If you want to revert to the original dbt behavior and fully refresh the incremental model, pass the [`--full-refresh` flag](https://docs.getdbt.com/reference/commands/run.md#refresh-incremental-models). Does dbt State support Python models? dbt State builds Python models but does not reuse them. It executes Python models on every run, even when their code and upstream data have not changed. How is data stored in dbt State? dbt State sends the following metadata to dbt Labs servers: * **Last-modified timestamps**: Used to determine whether upstream data has changed since the last run * **SQL statement hashes**: SQL statements are processed to detect and classify changes, then hashed. Only the hash is persisted for future comparisons. No actual data from your warehouse is transmitted. The dbt State service runs in a single US multi-tenant (MT) instance. The service never connects to your data warehouse. No actual data from your warehouse is transmitted. The only connection is to your running dbt process (CLI or platform) in order to exchange the metadata described above. For data retention details, refer to the [dbt Labs privacy policy](https://www.getdbt.com/cloud/privacy-policy). Where does the metadata about last updated timestamp come from? Last updated timestamps come directly from the data warehouse, for example from `INFORMATION_SCHEMA` tables. How does dbt State calculate that a model has changed? dbt State only considers substantial changes to a model. Because dbt State understands the entire lineage of your models, it can see through things like whitespace and aliases to determine whether a model is the same or different across environments. By default, dbt State compares rendered SQL to detect code changes. Any change to the rendered SQL — including from non-deterministic macros or environment variables — triggers a rebuild. You can enable [`compare_unrendered_code`](https://docs.getdbt.com/reference/resource-configs/compare-unrendered-code.md) to also check the Jinja template (unrendered code). When enabled, a rebuild only occurs when both the template *and* the rendered SQL have changed. Non-deterministic values (for example, `{{ env_var('AIRFLOW_RUN_ID') }}` or a macro that calls `uuid()`) don't trigger rebuilds as long as the template itself is unchanged, which helps avoid unnecessary warehouse compute costs. Why is my model being rebuilt instead of reused? dbt State decides whether to reuse a model by parsing the rendered SQL into a syntax tree and comparing the hash. If the hash has changed (implying the model's logic has changed), dbt State rebuilds the model. dbt State prioritizes safety and precision; if it can't guarantee skipping a node is safe, then it rebuilds the node to be sure. A few patterns that commonly cause overeager rebuilds are listed on this page, along with recommendations to increase reuse rate. The following patterns commonly cause unexpected rebuilds: * [Views with `select *`](#views-with-select) * [Non-deterministic Jinja templating](#non-deterministic-jinja-templating) * [Models with external sources in BigQuery](#models-with-external-sources-in-bigquery) * [Models with custom materializations](#models-with-custom-materializations) #### Views with `select *` dbt State reuses a model when its compiled SQL matches the stored hash. When a view uses `select *` directly on a `ref()` or `source()`, dbt can't determine the column list at parse time — the upstream model or source table might have gained or lost columns since the last run. To be safe, dbt State forces a rebuild. For example, this view will be rebuilt even if `stg_orders` hasn't changed because dbt can't know at parse time whether `stg_orders` has the same columns as before: ```sql -- stg_orders_view.sql (materialized: view) select * from {{ ref('stg_orders') }} ``` However, if you use `select *` on a CTE, dbt can resolve the columns from the CTE definition and safely reuse the view: ```sql with renamed as ( select order_id, customer_id, order_total from {{ ref('stg_orders') }} ) select * from renamed ``` If a CTE explicitly names its columns, a `select *` that reads from that CTE won't force a rebuild even if an earlier CTE used `select *` on a `ref()` or `source()`. The typical staging pattern is reused: ```sql with source as ( select * from {{ source('jaffle_shop', 'orders') }} ), renamed as ( select id as order_id, user_id as customer_id, amount as order_total from source ) select * from renamed ``` tip To avoid forced rebuilds, use explicit column names when selecting directly from a `ref()` or `source()`. You can also exclude views from execution using `--exclude config.materialized:view`. #### Non-deterministic Jinja templating Some macros and environment variables can cause unexpected rebuilds. For example, `dbt_utils.get_relations_by_pattern` (an introspective macro) combined with `dbt_utils.union_relations` can return relations in a different order on each run, producing different rendered SQL even when your project logic hasn't changed. Similarly, environment variables that change between runs produce different rendered SQL on every run: ```sql select '{{ env_var("AIRFLOW_RUN_ID") }}' as airflow_run_id, ... ``` Because the query result order or the environment variable's value changes, the rendered SQL differs from the stored hash on every run. dbt State treats this as a code change and rebuilds the model, even though the underlying project logic hasn't changed. This pattern can affect any model type, not just views; if a base or staging model rebuilds on every run, all of its downstream models rebuild, too. To avoid these unnecessary rebuilds, enable [`compare_unrendered_code`](https://docs.getdbt.com/reference/resource-configs/compare-unrendered-code.md). When enabled, dbt State checks both the Jinja template and rendered SQL; non-deterministic values that don't change the template don't trigger a rebuild. For example: ```sql {{ config(state={"compare_unrendered_code": true}) }} select '{{ env_var("AIRFLOW_RUN_ID") }}' as airflow_run_id, ... ``` #### Models with external sources on BigQuery On BigQuery, models that use external sources (such as Google Sheets) always rebuild because BigQuery doesn't expose modification timestamps for external sources, so dbt State can't determine freshness. tip To prevent external sources from always being considered stale, configure [`loaded_at_field`](https://docs.getdbt.com/reference/resource-properties/freshness.md#loaded_at_field) or [`loaded_at_query`](https://docs.getdbt.com/reference/resource-properties/freshness.md#loaded_at_query) in your source definition to point to a timestamp field. This lets dbt State query a timestamp field directly to determine freshness, rather than relying on warehouse metadata. #### Models with custom materializations Models using custom materializations are always built and are never reused. Custom materializations may have side effects (for example, modifying table properties or writing to other schemas), and dbt State cannot safely determine whether skipping the run would produce the same result. #### How to diagnose After a run, use (Applies to dbt v2.0 and later) [`dbt state explain`](https://docs.getdbt.com/reference/commands/state-explain.md) to see why dbt State rebuilt, reused, or cloned a specific model. For a detailed breakdown, use the `--verbose` flag with `-s` to select your model: note The command name differs by version: dbt v2 uses `dbt state explain` (with a space), while dbt v1 uses `dbt-state explain` (with a hyphen). (Applies to dbt v2.0 and later) ```bash dbt state explain --verbose -s my_model_name ``` If you use the dbt platform, the same information is available without running a command — go to the [**Explain** tab](https://docs.getdbt.com/docs/deploy/dbt-state-interface.md#explain-tab) on the job run details page to see the full decision breakdown for each node. What happens if dbt State servers fail? If dbt State servers are unavailable, dbt gracefully falls back to normal dbt behavior. What if I work on multiple projects that each use their own dbt State? You can specify your org ID in `dbt_project.yml`: ```yaml dbt-cloud: state-org-id: <your-org-id> ``` What if my prod environment isn't named prod? You can specify the defer-to environment using the [`defer_to_target`](https://docs.getdbt.com/reference/resource-configs/defer-to-target.md) config in `profiles.yml`: ```yaml my_project: outputs: prod: type: snowflake defer_to_target: production ``` `defer_to_target` only applies to self-managed deployments. If you're using the dbt platform, deferral is configured through your environment settings in the UI. For more details, refer to [Configuring deferral](https://docs.getdbt.com/docs/deploy/dbt-state-deferral.md). #### Related docs * [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) * [Monitor dbt State activity](https://docs.getdbt.com/docs/deploy/dbt-state-interface.md) * [Non-interactive environment setup](https://docs.getdbt.com/docs/deploy/dbt-state-cicd.md) * [dbt State configs](https://docs.getdbt.com/reference/resource-configs/dbt-state-configs.md) * [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md) * [dbt State trial and billing](https://docs.getdbt.com/docs/deploy/dbt-state-trial.md) * [dbt State usage and pricing](https://docs.getdbt.com/docs/platform/billing/dbt-state-usage.md) --- ### About dbt versions dbt v2 is the current generation of dbt and uses the 2.x release series. dbt v1 is the Python-based generation and stays on the 1.x series. Both follow semantic versioning. This page covers versioning for local dbt installations. v2 ships in two distributions. Install dbt v2 for the full, recommended experience. If your organization has a strict requirement for an Apache 2.0 licensed runtime, install dbt OSS instead. For more information, refer to [dbt licensing](https://github.com/docs/dbt-licensing). If you're using the dbt platform (including the dbt platform CLI), you don't need to manage dbt versions yourself. [Release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) automatically keep you up to date and provide early access to new features. #### dbt v2 versioning dbt v2 uses semantic versioning starting with version 2.0. To install or update dbt v2, see [Install dbt](https://docs.getdbt.com/docs/local/install-dbt.md?version=2). ##### Semantic versioning dbt v2 follows [semantic versioning](https://semver.org/): * **Major versions** (for example, v2 to v3) may include breaking changes. Deprecated functionality will stop working. * **Minor versions** (for example, v2.0 to v2.1) add features and are backwards compatible. They will not break project code that relies on documented functionality. * **Patch versions** (for example, v2.0.0 to v2.0.1) include fixes only: bug fixes, security fixes, or installation fixes. ##### Release channels dbt v2 is distributed through release channels during the preview period: | Channel | Description | Stability | | -------- | -------------------------------------- | ------------------------------------------------------------------- | | `latest` | The stable, "known good" version | ✅ Recommended for most users | | `canary` | The latest officially released version | ⚠️ Most recent stable version but still undergoing thorough testing | | `dev` | The latest development build | ❌ May not have passed all tests | Run `dbt system update` to get the latest stable release, or specify a channel with `dbt system update --version canary`. For current versions and release history, see [dbt v2 releases](https://docs.getdbt.com/docs/dbt/dbt-releases.md). ##### Checking your version Run `dbt --version` to check your installed version: ```text $ dbt --version dbt Fusion 2.0.0-preview.126 ``` ##### Further reading * [Install dbt v2](https://docs.getdbt.com/docs/local/install-dbt.md?version=2): Install or update dbt v2. * [dbt v2 releases](https://docs.getdbt.com/docs/dbt/dbt-releases.md): View current versions and release history. * [Get started with dbt v2](https://docs.getdbt.com/docs/dbt/get-started-dbt.md): Learn about dbt v2 features and migration. #### dbt v1 versioning The dbt v1 engine uses semantic versioning for the 1.x release series. To install or update dbt v1, see [Install dbt](https://docs.getdbt.com/docs/local/install-dbt.md?version=1). * **[Active](https://docs.getdbt.com/docs/dbt-versions.md#current-version-support)**: In the first few months after a minor version's initial release, we patch it with bugfix releases. These include fixes for regressions, new bugs, and older bugs / quality-of-life improvements. We implement these changes when we have high confidence that they're narrowly scoped and won't cause unintended side effects. * **[Critical](https://docs.getdbt.com/docs/dbt-versions.md#current-version-support)**: When a newer minor version ships, the previous one transitions to "Critical Support" for the remainder of its one-year window. Patches during this period are limited to critical security and installation fixes. After the one-year window ends, the version reaches end of life. * **[End of Life](https://docs.getdbt.com/docs/dbt-versions.md#end-of-life-versions)**: Minor versions that have reached EOL no longer receive new patch releases. * **Deprecated**: dbt v1 versions that are no longer maintained by dbt Labs, nor supported in the dbt platform. ##### Latest releases | dbt v1 | Initial release | Support level and end date | | ------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ | ------------------------------------------------------ | | [**v2.0**](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md) | Currently in [beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles.md) | TBD | | [**v1.12**](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.12.md) | Jul 16, 2026 | **Active support — July 15, 2027** | | [**v1.11**](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.11.md) | Dec 19, 2025 | **Critical support — Dec 18, 2026** | | [**v1.10**](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.10.md) | Jun 16, 2025 | Deprecated ⛔️ | | [**v1.9**](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.9.md) | Dec 9, 2024 | Deprecated ⛔️ | | [**v1.8**](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.8.md) | May 9, 2024 | Deprecated ⛔️ | | [**v1.7**](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.7.md) | Nov 2, 2023 | End of Life ⚠️<br />Deprecation date: January 31, 2027 | | [**v1.6**](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.6.md>) | Jul 31, 2023 | End of Life ⚠️<br />Deprecation date: January 31, 2027 | | [**v1.5**](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.5.md>) | Apr 27, 2023 | End of Life ⚠️<br />Deprecation date: January 31, 2027 | | [**v1.4**](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.4.md>) | Jan 25, 2023 | End of Life ⚠️<br />Deprecation date: January 31, 2027 | | [**v1.3**](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.3.md>) | Oct 12, 2022 | End of Life ⚠️<br />Deprecation date: January 31, 2027 | | [**v1.2**](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.2.md>) | Jul 26, 2022 | Deprecated ⛔️ | | [**v1.1**](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.1.md>) | Apr 28, 2022 | Deprecated ⛔️ | | [**v1.0**](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.0.md>) | Dec 3, 2021 | Deprecated ⛔️ | | **v0.X** ⛔️ | (Various dates) | Deprecated ⛔️ | All functionality in v1 since the v1.7 release is available in [dbt release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md), which provide automated upgrades at a cadence appropriate for your team. 1 Release tracks are required for the Developer and Starter plans on dbt. Accounts using older dbt versions will be migrated to the **v1 Latest** release track. For customers of dbt: dbt Labs strongly recommends migrating environments on older and unsupported versions to [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) or a supported version. On January 31, 2027, dbt Labs will deprecate v1 versions v1.3 through v1.7, removing them from availability in dbt platform. ##### How dbt v1 uses semantic versioning dbt follows [semantic versioning](https://semver.org/): * **Major versions** (for example, v1 to v2) may include breaking changes. Deprecated functionality will stop working. * **Minor versions** (for example, v1.8 to v1.9) add features and are backwards compatible. They will not break project code that relies on documented functionality. * **Patch versions** (for example, v1.8.0 to v1.8.1) include fixes only: bug fixes, security fixes, or installation fixes. We are committed to avoiding breaking changes in minor versions for end users of dbt. There are two types of breaking changes that may be included in minor versions: * Changes to the Python interface for adapter plugins. These changes are relevant only to adapter maintainers, and they will be clearly communicated in documentation and release notes. For more information, refer to [Build, test, document, and promote adapters guide](https://docs.getdbt.com/guides/adapter-creation.md). * Changes to metadata interfaces, including [artifacts](https://docs.getdbt.com/docs/deploy/artifacts.md) and [logging](https://docs.getdbt.com/reference/events-logging.md), signalled by a version bump. Those version upgrades may require you to update external code that depends on these interfaces, or to coordinate upgrades between dbt orchestrations that share metadata, such as [state-powered selection](https://docs.getdbt.com/reference/node-selection/syntax.md#about-node-selection). ##### Adapter plugin versions dbt releases `dbt-core` and adapter plugins (such as `dbt-snowflake`) independently. Their minor and patch version numbers may not match, but they coordinate through the `dbt-adapters` interface so you won't get a broken experience. For example, `dbt-core==1.8.0` can work with `dbt-snowflake==1.9.0`. If you're building or maintaining an adapter, refer to the [adapter creation guide](https://docs.getdbt.com/guides/adapter-creation.md) for details on the `dbt-adapters` interface. Run `dbt --version` to check your installed versions: ```text $ dbt --version Core: - installed: 1.8.0 - latest: 1.8.0 - Up to date! Plugins: - snowflake: 1.9.0 - Up to date! ``` You can also find the registered adapter version in [logs](https://docs.getdbt.com/reference/global-configs/logs.md). For example, in `logs/dbt.log`: ```text [0m13:13:48.572182 [info ] [MainThread]: Registered adapter: snowflake=1.9.0 ``` Refer to [Supported data platforms](https://docs.getdbt.com/docs/supported-data-platforms.md) for the full list of adapters. ##### Further reading * [Choosing a dbt v1 version in dbt](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md): Learn how to use dbt v1 versions in dbt. * [Install dbt v1](https://docs.getdbt.com/docs/local/install-dbt.md?version=1): Install or update dbt v1. * [`require-dbt-version`](https://docs.getdbt.com/reference/project-configs/require-dbt-version.md) and [`dbt_version`](https://docs.getdbt.com/reference/dbt-jinja-functions/dbt_version.md): Restrict your project to work with a specific range of versions. #### End-of-life versions info On January 31, 2027, dbt versions v1.3-v1.7 will be deprecated and removed from availability in dbt platform. Upgrade any environments still on these versions to a supported version or a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) before then. Once a dbt version reaches end-of-life (EOL), it no longer receives patches, including for known bugs. We recommend upgrading to a newer version in [dbt](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md), [dbt v2](https://docs.getdbt.com/docs/local/install-dbt.md?version=2), or [dbt v1](https://docs.getdbt.com/docs/local/install-dbt.md). All versions prior to v1.0 have been deprecated. #### Current version support dbt supports each minor version (for example, v1.8) for *one year* from its initial release. During that window, we release patches with bug fixes and security updates. When we refer to a minor version, we mean its latest available patch (v1.8.x). After a newer minor version ships, the previous one transitions to **critical support** (security and installation fixes only) for the remainder of its one-year window. After the one-year window ends, the version reaches **end of life** and no longer receives patches. While a minor version is officially supported: * You can use it in dbt. For more on dbt versioning, see [Choosing a dbt version](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md). * You can select it from the version dropdown on this website to see documentation that is accurate for use with that minor version. #### Upgrading Upgrade to new patch versions as soon as they're available. Upgrade to new minor versions when you're ready because you can only get some features and fixes on the latest minor version. dbt makes all versions available as prereleases before the final release. For minor versions, we aim to release one or more betas 4+ weeks before the final release so you can try new features and share feedback. Release candidates are available about two weeks before the final release for testing in production-like environments. Refer to the [`dbt-labs/dbt` milestones](https://github.com/dbt-labs/dbt/milestones) for details. --- ### About documentation Good documentation for your dbt models will help downstream consumers discover and understand the datasets you curate for them. dbt provides a way to generate documentation for your dbt project and render it as a website. Tip Use [dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-overview.md) to generate documentation in the dbt platform or locally in the CLI. #### Related documentation * [Declaring properties](https://docs.getdbt.com/reference/configs-and-properties.md) * [`dbt docs` command](https://docs.getdbt.com/reference/commands/cmd-docs.md) * [`doc` Jinja function](https://docs.getdbt.com/reference/dbt-jinja-functions/doc.md) * If you're new to dbt, we recommend that you check out our [quickstart guide](https://docs.getdbt.com/guides.md) to build your first dbt project, complete with documentation. #### Assumed knowledge * [Data tests](https://docs.getdbt.com/docs/build/data-tests.md) #### Overview dbt provides a scalable way to [generate](#generating-documentation) documentation for your dbt project using descriptions and commands. The documentation for your project includes: * **Information about your project**: including model code, a DAG of your project, any tests you've added to a column, and more. * **Information about your data warehouse**: including column data types, and table sizes. This information is generated by running queries against the information schema. * Importantly, dbt also provides a way to add **descriptions** to models, columns, sources, and more, to further enhance your documentation. The following sections describe how to [add descriptions](#adding-descriptions-to-your-project) to your project, [generate documentation](#generating-documentation), how to use [docs blocks](#using-docs-blocks), and set a [custom overview](#setting-a-custom-overview) for your documentation. #### Adding descriptions to your project Before generating documentation, add [descriptions](https://docs.getdbt.com/reference/resource-properties/description.md) to your project resources. Add the `description:` key to the same YAML files where you declare [data tests](https://docs.getdbt.com/docs/build/data-tests.md). For example: models/\<filename>.yml ```yaml models: - name: events description: This table contains clickstream events from the marketing website columns: - name: event_id description: This is a unique identifier for the event data_tests: - unique - not_null - name: user-id quote: true description: The user who performed the event data_tests: - not_null ``` ##### FAQs Are there any example dbt documentation sites? Yes! * **Quickstart Tutorial:** You can build your own example dbt project in the [quickstart guide](https://docs.getdbt.com/docs/get-started-dbt.md) * **Jaffle Shop:** A demonstration project (closely related to the tutorial) for a fictional e-commerce store. Refer to [Clone the Jaffle Shop sample project](https://docs.getdbt.com/guides/clone-jaffle-shop.md) for clone instructions, or view the [main source code](https://github.com/dbt-labs/jaffle-shop) and [source code using DuckDB](https://github.com/dbt-labs/jaffle_shop_duckdb) on GitHub. * **GitLab:** Gitlab's internal dbt project is open source and is a great example of how to use dbt at scale ([source code](https://gitlab.com/gitlab-com/content-sites/handbook/blob/main/content/handbook/enterprise-data/platform/dbt-guide.md)) * **dummy-dbt:** A containerized dbt project that populates the Sakila database in Postgres and populates dbt seeds, models, snapshots, and tests. The project can be used for testing and experimentation purposes ([source code](https://github.com/gmyrianthous/dbt-dummy)) * **Google Analytics 4:** A demonstration project that transforms the Google Analytics 4 BigQuery exports to various models ([source code](https://github.com/stacktonic-com/stacktonic-dbt-example-project), [docs](https://stacktonic.com/article/google-analytics-big-query-and-dbt-a-dbt-example-project)) * **Make Open Data:** A production-grade ELT with tests, documentation, and CI/CD (GHA) about French open data (housing, demography, geography, etc). It can be used to learn with voluminous and ambiguous data. Contributions are welcome ([source code](https://github.com/make-open-data/make-open-data), [docs](https://make-open-data.fr/)) If you have an example project to add to this list, suggest an edit by clicking **Edit this page** below. Do I need to add a YAML entry for column for it to appear in the docs site? Fortunately, no! dbt will introspect your warehouse to generate a list of columns in each relation, and match it with the list of columns in your `.yml` files. As such, any undocumented columns will still appear in your documentation! How do I write long-form explanations in my descriptions? If you need more than a sentence to explain a model, you can: 1. Split your description over multiple lines using `>`. Interior line breaks are removed and Markdown can be used. This method is recommended for simple, single-paragraph descriptions: ```yml models: - name: customers description: > Lorem ipsum **dolor** sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. ``` 2. Split your description over multiple lines using `|`. Interior line breaks are maintained and Markdown can be used. This method is recommended for more complex descriptions: ```yml models: - name: customers description: | ### Lorem ipsum * dolor sit amet, consectetur adipisicing elit, sed do eiusmod * tempor incididunt ut labore et dolore magna aliqua. ``` 3. Use a [docs block](https://docs.getdbt.com/docs/build/documentation.md#using-docs-blocks) to write the description in a separate Markdown file. How do I access documentation in dbt Catalog? If you're using dbt to deploy your project and have a [Starter, Enterprise, or Enterprise+ plan](https://www.getdbt.com/pricing/), you can use Catalog to view your project's [resources](https://docs.getdbt.com/docs/build/projects.md) (such as models, tests, and metrics) and their lineage to gain a better understanding of its latest production state. Access Catalog in dbt by clicking the **Catalog** link in the navigation. Read-only users can access the documentation for your project — for license availability by plan, refer to [dbt pricing](https://www.getdbt.com/pricing). dbt developer plan and dbt v1 users can use [dbt Docs](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md#dbt-docs), which generates basic documentation but it doesn't offer the same speed, metadata, or visibility as Catalog. Can I document things other than models, like sources, seeds, and snapshots? Yes! You can document almost everything in your project using the `description:` key. Check out the reference docs on [descriptions](https://docs.getdbt.com/reference/resource-properties/description.md) for more info! #### Generating documentation (Applies to dbt v2.0 and later) Using dbt v2, dbt Docs v2 replaces the v1 static site with a modern, performant catalog. `dbt docs generate` compiles your project, produces the v2 Parquet artifacts, and writes a static site that the browser queries directly with DuckDB-WASM (WebAssembly), so you don't need a server to view it. To generate and serve documentation: 1. Run `dbt docs generate` to compile your project, write the index, and export the documentation site in a single command. 2. Run `dbt docs serve` to preview the site locally. Refer to [dbt docs commands](https://docs.getdbt.com/reference/commands/cmd-docs.md) for full usage details. dbt provides three complementary ways to [view documentation](https://docs.getdbt.com/docs/build/view-documentation.md) after descriptions are generated: * [**dbt Docs (Legacy)**](https://docs.getdbt.com/docs/build/view-documentation.md#dbt-docs): A static documentation site with model lineage, metadata, and documentation that can be hosted on your web server (like S3 or Netlify). Available for dbt v1 or dbt Developer plans. * [**dbt Docs v2**](https://docs.getdbt.com/docs/build/view-documentation.md#dbt-docs-v2): A modern, performant open-source catalog with a redesigned UI, Semantic Layer metadata, and column-level lineage, served as a static site you can host anywhere. Available with dbt v2. * [**Catalog**](https://docs.getdbt.com/docs/explore/explore-projects.md): Builds upon dbt Docs to provide a dynamic, real-time interface with enhanced metadata, customizable views, deeper project insights, and collaboration tools. Available on dbt [Starter, Enterprise, or Enterprise+ plans](https://www.getdbt.com/pricing). Refer to [View documentation](https://docs.getdbt.com/docs/build/view-documentation.md) to get the most out of your dbt project's documentation. #### Using docs blocks Docs blocks provide a robust method for documenting models and other resources using Jinja and markdown. Docs block files can contain arbitrary markdown, but they must be uniquely named. ##### Syntax To declare a docs block, use the Jinja `docs` tag. The name of a docs block can't start with a digit and may contain: * Uppercase and lowercase letters (A-Z, a-z) * Digits (0-9) * Underscores (\_) events.md ```markdown {% docs table_events %} This table contains clickstream events from the marketing website. The events in this table are recorded by Snowplow and piped into the warehouse on an hourly basis. The following pages of the marketing site are tracked: - / - /about - /team - /contact-us {% enddocs %} ``` In this example, a docs block named `table_events` is defined with some descriptive markdown contents. There is nothing significant about the name `table_events` — docs blocks can be named however you like, as long as the name only contains alphanumeric and underscore characters and doesn't start with a numeric character. ##### Placement (Applies to dbt v1.12 and later) Place docs blocks in `.md` files. You can also use Jinja-style extensions (`.md.j2`, `.md.jinja`, `.md.jinja2`), however these require setting [`allow_jinja_file_extensions: true`](https://docs.getdbt.com/reference/global-configs/behavior-flags/allow_jinja_file_extensions.md) in your `dbt_project.yml`. This enables Jinja-aware syntax highlighting in IDEs that associate these suffixes with Jinja templating. By default, dbt searches in all resource paths for docs blocks (for example, the combined list of [model-paths](https://docs.getdbt.com/reference/project-configs/model-paths.md), [seed-paths](https://docs.getdbt.com/reference/project-configs/seed-paths.md), [analysis-paths](https://docs.getdbt.com/reference/project-configs/analysis-paths.md), [test-paths](https://docs.getdbt.com/reference/project-configs/test-paths.md), [macro-paths](https://docs.getdbt.com/reference/project-configs/macro-paths.md), and [snapshot-paths](https://docs.getdbt.com/reference/project-configs/snapshot-paths.md)). You can adjust this behavior using the [docs-paths](https://docs.getdbt.com/reference/project-configs/docs-paths.md) config. ##### Usage To use a docs block, reference it from your `schema.yml` file with the [doc()](https://docs.getdbt.com/reference/dbt-jinja-functions/doc.md) function in place of a markdown string. Using the examples above, the `table_events` docs can be included in the `schema.yml` file as shown here: schema.yml ```yaml models: - name: events description: '{{ doc("table_events") }}' columns: - name: event_id description: This is a unique identifier for the event data_tests: - unique - not_null ``` In the resulting documentation, `'{{ doc("table_events") }}'` will be expanded to the markdown defined in the `table_events` docs block. #### Setting a custom overview Availability This feature is available only in [dbt Docs](https://docs.getdbt.com/docs/build/documentation.md), the generated documentation site for your dbt project. The "overview" shown in the dbt Docs website can be overridden by supplying your own docs block called `__overview__`. * By default, dbt supplies an overview with helpful information about the docs site itself. * Depending on your needs, it may be a good idea to override this docs block with specific information about your company style guide, links to reports, or information about who to contact for help. * To override the default overview, create a docs block that looks like this: models/overview.md ```markdown {% docs __overview__ %} # Monthly Recurring Revenue (MRR) playbook. This dbt project is a worked example to demonstrate how to model subscription revenue. **Check out the full write-up [here](https://blog.getdbt.com/modeling-subscription-revenue/), as well as the repo for this project [here](https://github.com/dbt-labs/mrr-playbook/).** ... {% enddocs %} ``` ##### Custom project-level overviews You can set different overviews for each dbt project/package included in your documentation site by creating a docs block named `__[project_name]__`. For example, in order to define custom overview pages that appear when a viewer navigates inside the `dbt_utils` or `snowplow` package: models/overview.md ```markdown {% docs __dbt_utils__ %} # Utility macros Our dbt project heavily uses this suite of utility macros, especially: - `surrogate_key` - `test_equality` - `pivot` {% enddocs %} {% docs __snowplow__ %} # Snowplow sessionization Our organization uses this package of transformations to roll Snowplow events up to page views and sessions. {% enddocs %} ``` --- ### About Hybrid projects dbt platform | Enterprise+ With Hybrid projects, your organization can adopt complementary dbt v1 and dbt workflows (where some teams deploy projects in dbt v1 and others in dbt) and seamlessly integrate these workflows by automatically uploading dbt v1 [artifacts](https://docs.getdbt.com/reference/artifacts/dbt-artifacts.md) into dbt. Available in public preview Hybrid projects is available in public preview to [dbt Enterprise accounts](https://www.getdbt.com/pricing). dbt v1 users can seamlessly upload [artifacts](https://docs.getdbt.com/reference/artifacts/dbt-artifacts.md) like [run results.json](https://docs.getdbt.com/reference/artifacts/run-results-json.md), [manifest.json](https://docs.getdbt.com/reference/artifacts/manifest-json.md), [catalog.json](https://docs.getdbt.com/reference/artifacts/catalog-json.md), [sources.json](https://docs.getdbt.com/reference/artifacts/sources-json.md), and so on — into dbt after executing a run in the dbt v1 command line interface (CLI), which helps: * Collaborate with dbt + dbt v1 users by enabling them to visualize and perform [cross-project references](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#how-to-write-cross-project-ref) to dbt models that live in dbt v1 projects. * (Coming soon) New users interested in the [Canvas](https://docs.getdbt.com/docs/platform/canvas.md) can build off of dbt models already created by a central data team in dbt v1 rather than having to start from scratch. * dbt v1 and dbt users can navigate to [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) and view their models and assets. To view Catalog, you must have a [read-only seat](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md). #### Prerequisites To upload artifacts, make sure you meet these prerequisites: * Your organization is on a [dbt Enterprise+ plan](https://www.getdbt.com/pricing) * You're on [dbt's release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) and your dbt v1 project is on dbt v1.10 or higher * [Configured](https://docs.getdbt.com/docs/deploy/hybrid-setup.md#connect-project-in-dbt-cloud) a hybrid project in dbt. * Updated your existing dbt v1 project with latest changes and [configured it with model access](https://docs.getdbt.com/docs/deploy/hybrid-setup.md#make-dbt-models-public): * Ensure models that you want to share with other dbt projects use `access: public` in their model configuration. This makes the models more discoverable and shareable * Learn more about [access modifier](https://docs.getdbt.com/docs/mesh/govern/model-access.md#access-modifiers) and how to set the [`access` config](https://docs.getdbt.com/reference/resource-configs/access.md) * Update [dbt permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) to create a new project in dbt **Note:** Uploading artifacts doesn't count against dbt run slots. --- ### About Iceberg catalogs A technical data catalog is a metadata management layer that enables users and tools to programmatically discover, understand, and govern data assets that are available across multiple data platforms and query engines. #### Background In the early 2010s, with the introduction of [Hadoop](https://hadoop.apache.org/) and data lakes, [Hive Metastore](https://hive.apache.org/) became the standard for managing schema metadata in Hadoop ecosystems. While effective in supporting early distributed query engines (separated compute and storage), such as Apache Spark and Trino, the Hive Metastore is limited in the kinds of structural metadata it can support, and it's generally slower and costlier than modern technical catalogs. In recent years, the emergence and consolidation around the [Iceberg standard](https://iceberg.apache.org/terms/) has led to the evolution of existing data catalogs (such as [Unity Catalog](https://www.unitycatalog.io/)) or the creation of new open source catalogs (including [Polaris](https://polaris.apache.org/) and [Lakekeeper](https://docs.lakekeeper.io/)). It's important to note that **technical data catalogs** serve a different purpose from **business catalogs**, although both benefit data teams: * **Technical data catalogs:** Focus on structural metadata, including information about data like table and column names, data types, storage locations (particularly important for open table formats), and access controls. They can be built into an existing data platform (no setup needed) — such as Databricks Unity Catalog or Snowflake Horizon Catalog — or externally managed and integrated with each query engine. Those engines use the catalog to locate, read, and write data. * **Business data catalogs:** Serve broader organizational users (such as BI analysts and product managers). They enrich technical metadata with business context in the form of metrics, business definitions, data quality indicators, usage patterns, and ownership. #### Why data catalogs are important to dbt For dbt users working in a lakehouse or multi-engine architecture, data catalogs can serve two purposes: * **Table discovery:** dbt models are materialized in an open table format (often Iceberg) and registered in a catalog. Understanding the catalog structure is critical for managing datasets and informing dbt about what has already been built and where it resides. * **Cross-engine interoperability:** Iceberg catalogs allow datasets created by one compute engine to be read by another, without replication. This is the foundation for [cross-platform dbt Mesh](https://docs.getdbt.com/docs/mesh/cross-platform-mesh.md). Without a catalog, each Iceberg table's metadata needs to be registered with the query engine individually, requiring many additional `create iceberg table` and `alter ... refresh` queries. With a catalog, each table's metadata is registered and refreshed automatically, and there's a single endpoint to ask about every table. Over the past year, data platform vendors have been adding support for catalog "linking" or "federation," which is when the data platform manages the synchronization of metadata between external data catalogs and its managed data catalog. This means that an Iceberg table written by one query engine into one catalog is automatically available for reading by another query engine. See: [Snowflake catalog-linked databases](https://docs.snowflake.com/en/sql-reference/sql/create-database-catalog-linked), [Databricks catalog federation](https://docs.databricks.com/aws/en/query-federation/catalog-federation), [AWS Glue catalog federation](https://docs.aws.amazon.com/lake-formation/latest/dg/federated-catalog-data-connection.html), [BigQuery catalog federation](https://docs.cloud.google.com/lakehouse/docs/use-catalog-federation). #### How dbt works with Iceberg catalogs dbt uses [Iceberg catalogs defined in `catalogs.yml`](https://docs.getdbt.com/docs/build/iceberg/catalogs-yml.md) in order to: * **Materialize models:** When dbt materializes a model as a table or view, if the catalog integration is declared, the underlying adapter (such as Spark, Trino, or Snowflake) creates an Iceberg table entry in the specified catalog, both built-in or external. * **Resolve references:** When dbt reads a model materialized to a catalog, it uses the "physical" location defined by the active adapter (new spec) or write integration (old spec). This enables referencing (reading) a model (Iceberg table) in one query engine that was originally materialized (written) by another one, so long as both engines are integrated to the same catalog. #### Limitations Many data platforms, query engines, and data catalogs have added substantially more support for Iceberg standards over the past few years — but it's still not a guarantee that any given data warehouse can interoperate with any given data catalog. When it does exist, that support can be limited. For example, most Iceberg catalogs only support tables (`table` + `incremental` materialization in dbt), not views. They may not support atomic `create table as` statements, meaning that dbt must update tables using multiple statements without ACID guarantees. Benoit (DX advocate at dbt Labs) created a [useful visualization](https://b-per.github.io/iceberg-roulette/#matrix) (also available as a [fun interactive game](https://b-per.github.io/iceberg-roulette/)) to show the current state of warehouse / catalog interoperability. (See something that looks wrong or out-of-date? Open an issue or PR in [the repo](https://github.com/b-per/iceberg-roulette).) --- ### About incremental models This is an introduction on incremental models, when to use them, and how they work in dbt. Incremental models in dbt is a [materialization](https://docs.getdbt.com/docs/build/materializations.md) strategy designed to efficiently update your data warehouse tables by only transforming and loading new or changed data since the last run. Instead of processing your entire dataset every time, incremental models append or update only the new rows, significantly reducing the time and resources required for your data transformations. This page will provide you with a brief overview of incremental models, their importance in data transformations, and the core concepts of incremental materializations in dbt. [![A visual representation of how incremental models work. Source: Materialization best practices guide (/best-practices/materializations/1-guide-overview)](/img/docs/building-a-dbt-project/incremental-diagram.jpg?v=2 "A visual representation of how incremental models work. Source: Materialization best practices guide (/best-practices/materializations/1-guide-overview)")](#)A visual representation of how incremental models work. Source: Materialization best practices guide (/best-practices/materializations/1-guide-overview) Learn by video! For video tutorials on Incremental models, go to dbt Learn and check out the [Incremental models course](https://learn.getdbt.com/courses/incremental-models). Build idempotent incremental models Incremental models are stateful, so they're the easiest place to accidentally break [idempotence](https://docs.getdbt.com/best-practices/idempotence.md) — the expectation that re-running a model produces the same result. Before you configure one, review [Idempotence in dbt](https://docs.getdbt.com/best-practices/idempotence.md#idempotence-and-incremental-models). #### Understand incremental models Incremental models enable you to significantly reduce the build time by just transforming new records. This is particularly useful for large datasets, where the cost of processing the entire dataset is high. Incremental models [require extra configuration](https://docs.getdbt.com/docs/build/incremental-models.md) and are an advanced usage of dbt. We recommend using them when your dbt runs are becoming too slow. ##### When to use an incremental model Building models as tables in your data warehouse is often preferred for better query performance. However, using `table` materialization can be computationally intensive, especially when: * Source data has millions or billions of rows. * Data transformations on the source data are computationally expensive (take a long time to execute) and complex, like when using Regex or UDFs. Incremental models offer a balance between complexity and improved performance compared to `view` and `table` materializations and offer better performance of your dbt runs. In addition to these considerations for incremental models, it's important to understand their limitations and challenges, particularly with large datasets. For more insights into efficient strategies, performance considerations, and the handling of late-arriving data in incremental models, refer to the [On the Limits of Incrementality](https://discourse.getdbt.com/t/on-the-limits-of-incrementality/303) discourse discussion or to our [Materialization best practices](https://docs.getdbt.com/best-practices/materializations/2-available-materializations.md) page. ##### How incremental models work in dbt dbt's [incremental materialization strategy](https://docs.getdbt.com/docs/build/incremental-strategy.md) works differently on different databases. Where supported, a `merge` statement is used to insert new records and update existing records. On warehouses that do not support `merge` statements, a merge is implemented by first using a `delete` statement to delete records in the target table that are to be updated, and then an `insert` statement. Transaction management, a process used in certain data platforms, ensures that a set of actions is treated as a single unit of work (or task). If any part of the unit of work fails, dbt will roll back open transactions and restore the database to a good state. #### Related docs * [Incremental models](https://docs.getdbt.com/docs/build/incremental-models.md) to learn how to configure incremental models in dbt. * [Incremental strategies](https://docs.getdbt.com/docs/build/incremental-strategy.md) to understand how dbt implements incremental models on different databases. * [Microbatch](https://docs.getdbt.com/docs/build/incremental-microbatch.md) to understand a new incremental strategy intended for efficient and resilient processing of very large time-series datasets. * [Materializations best practices](https://docs.getdbt.com/best-practices/materializations/1-guide-overview.md) to learn about the best practices for using materializations in dbt. --- ### About incremental strategy Incremental strategies for materializations optimize performance by defining how to handle new and changed data. There are various strategies to implement the concept of incremental materializations. The value of each strategy depends on: * The volume of data. * The reliability of your `unique_key`. * The support of certain features in your data platform. An optional `incremental_strategy` config is provided in some adapters that controls the code that dbt uses to build incremental models. Microbatch The [`microbatch` incremental strategy](https://docs.getdbt.com/docs/build/incremental-microbatch.md) is intended for large time-series datasets. dbt will process the incremental model in multiple queries (or "batches") based on a configured `event_time` column. Depending on the volume and nature of your data, this can be more efficient and resilient than using a single query for adding new data. ##### Supported incremental strategies by adapter This table shows the support of each incremental strategy across adapters available on dbt v2 or dbt's [**v1 Latest** release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). Some strategies may be unavailable if you're not on **v1 Latest** and the feature hasn't been released to the **v1 Compatible** track. If you're interested in incremental strategies for additional adapters, check out the [adapter's individual configuration page](https://docs.getdbt.com/reference/resource-configs.md) for more details. Click the name of the adapter in the following table for more information about supported incremental strategies: | Data platform adapter | `append` | `merge` | `delete+insert` | `insert_overwrite` | `microbatch` | | ----------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------- | --------------- | ------------------ | ------------ | | [dbt-postgres](https://docs.getdbt.com/reference/resource-configs/postgres-configs.md#incremental-materialization-strategies) | ✅ | ✅ | ✅ | | ✅ | | [dbt-redshift](https://docs.getdbt.com/reference/resource-configs/redshift-configs.md#incremental-materialization-strategies) | ✅ | ✅ | ✅ | | ✅ | | [dbt-bigquery](https://docs.getdbt.com/reference/resource-configs/bigquery-configs.md#merge-behavior-incremental-models) | | ✅ | | ✅ | ✅ | | [dbt-spark](https://docs.getdbt.com/reference/resource-configs/spark-configs.md#incremental-models) | ✅ | ✅ | | ✅ | ✅ | | [dbt-databricks](https://docs.getdbt.com/reference/resource-configs/databricks-configs.md#incremental-models) | ✅ | ✅ | ✅ | ✅ | ✅ | | [dbt-snowflake](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#merge-behavior-incremental-models) | ✅ | ✅ | ✅ | ✅ | ✅ | | [dbt-trino](https://docs.getdbt.com/reference/resource-configs/trino-configs.md#incremental) | ✅ | ✅ | ✅ | | ✅ | | [dbt-fabric](https://docs.getdbt.com/reference/resource-configs/fabric-configs.md#incremental) | ✅ | ✅ | ✅ | | ✅ | | [dbt-athena](https://docs.getdbt.com/reference/resource-configs/athena-configs.md#incremental-models) | ✅ | ✅ | | ✅ | ✅ | | [dbt-teradata](https://docs.getdbt.com/reference/resource-configs/teradata-configs.md#valid_history-incremental-materialization-strategy) | ✅ | ✅ | ✅ | | ✅ | | [dbt-duckdb](https://docs.getdbt.com/reference/resource-configs/duckdb-configs.md#incremental) | ✅ | ✅ | ✅ | | ✅ | ##### Configuring incremental strategy The `incremental_strategy` config can either be defined in specific models or for all models in your `dbt_project.yml` file: dbt\_project.yml ```yaml models: +incremental_strategy: "insert_overwrite" ``` or: models/my\_model.sql ```sql {{ config( materialized='incremental', unique_key='date_day', incremental_strategy='delete+insert', ... ) }} select ... ``` ##### Strategy-specific configs If you use the `merge` strategy and specify a `unique_key`, by default, dbt will entirely overwrite matched rows with new values. On adapters which support the `merge` strategy, you may optionally pass a list of column names to a `merge_update_columns` config. In that case, dbt will update *only* the columns specified by the config, and keep the previous values of other columns. models/my\_model.sql ```sql {{ config( materialized = 'incremental', unique_key = 'id', merge_update_columns = ['email', 'ip_address'], ... ) }} select ... ``` Alternatively, you can specify a list of columns to exclude from being updated by passing a list of column names to a `merge_exclude_columns` config. models/my\_model.sql ```sql {{ config( materialized = 'incremental', unique_key = 'id', merge_exclude_columns = ['created_at'], ... ) }} select ... ``` ##### About incremental\_predicates `incremental_predicates` is an advanced use of incremental models, where data volume is large enough to justify additional investments in performance. This config accepts a list of any valid SQL expression(s). dbt does not check the syntax of the SQL statements. This an example of a model configuration in a `yml` file you might expect to see on Snowflake: ```yml models: - name: my_incremental_model config: materialized: incremental unique_key: id # this will affect how the data is stored on disk, and indexed to limit scans cluster_by: ['session_start'] incremental_strategy: merge # this limits the scan of the existing table to the last 7 days of data incremental_predicates: ["DBT_INTERNAL_DEST.session_start > dateadd(day, -7, current_date)"] # `incremental_predicates` accepts a list of SQL statements. # `DBT_INTERNAL_DEST` and `DBT_INTERNAL_SOURCE` are the standard aliases for the target table and temporary table, respectively, during an incremental run using the merge strategy. ``` Alternatively, here are the same configurations configured within a model file: ```sql -- in models/my_incremental_model.sql {{ config( materialized = 'incremental', unique_key = 'id', cluster_by = ['session_start'], incremental_strategy = 'merge', incremental_predicates = [ "DBT_INTERNAL_DEST.session_start > dateadd(day, -7, current_date)" ] ) }} ... ``` This will template (in the `dbt.log` file) a `merge` statement like: ```sql merge into <existing_table> DBT_INTERNAL_DEST from <temp_table_with_new_records> DBT_INTERNAL_SOURCE on -- unique key DBT_INTERNAL_DEST.id = DBT_INTERNAL_SOURCE.id and -- custom predicate: limits data scan in the "old" data / existing table DBT_INTERNAL_DEST.session_start > dateadd(day, -7, current_date) when matched then update ... when not matched then insert ... ``` Limit the data scan of *upstream* tables within the body of their incremental model SQL, which will limit the amount of "new" data processed/transformed. ```sql with large_source_table as ( select * from {{ ref('large_source_table') }} {% if is_incremental() %} where session_start >= dateadd(day, -3, current_date) {% endif %} ), ... ``` info The syntax depends on how you configure your `incremental_strategy`: * If using the `merge` strategy, you may need to explicitly alias any columns with either `DBT_INTERNAL_DEST` ("old" data) or `DBT_INTERNAL_SOURCE` ("new" data). * There's a decent amount of conceptual overlap with the `insert_overwrite` incremental strategy. ##### Built-in strategies Before diving into [custom strategies](#custom-strategies), it's important to understand the built-in incremental strategies in dbt and their corresponding macros: | `incremental_strategy` | Corresponding macro | | ------------------------------------------------------------------------------------------------- | -------------------------------------- | | [`append`](https://docs.getdbt.com/docs/build/incremental-strategy.md#append) | `get_incremental_append_sql` | | [`delete+insert`](https://docs.getdbt.com/docs/build/incremental-strategy.md#deleteinsert) | `get_incremental_delete_insert_sql` | | [`merge`](https://docs.getdbt.com/docs/build/incremental-strategy.md#merge) | `get_incremental_merge_sql` | | [`insert_overwrite`](https://docs.getdbt.com/docs/build/incremental-strategy.md#insert_overwrite) | `get_incremental_insert_overwrite_sql` | | [`microbatch`](https://docs.getdbt.com/docs/build/incremental-strategy.md#microbatch) | `get_incremental_microbatch_sql` | For example, a built-in strategy for the `append` can be defined and used with the following files: macros/append.sql ```sql {% macro get_incremental_append_sql(arg_dict) %} {% do return(some_custom_macro_with_sql(arg_dict["target_relation"], arg_dict["temp_relation"], arg_dict["unique_key"], arg_dict["dest_columns"], arg_dict["incremental_predicates"])) %} {% endmacro %} {% macro some_custom_macro_with_sql(target_relation, temp_relation, unique_key, dest_columns, incremental_predicates) %} {%- set dest_cols_csv = get_quoted_csv(dest_columns | map(attribute="name")) -%} insert into {{ target_relation }} ({{ dest_cols_csv }}) ( select {{ dest_cols_csv }} from {{ temp_relation }} ) {% endmacro %} ``` Define a model models/my\_model.sql: ```sql {{ config( materialized="incremental", incremental_strategy="append", ) }} select * from {{ ref("some_model") }} ``` ###### About built-in incremental strategies ###### `append` The `append` strategy is simple to implement and has low processing costs. It inserts selected records into the destination table without updating or deleting existing data. This strategy doesn’t align directly with type 1 or type 2 [slowly changing dimensions](https://www.thoughtspot.com/data-trends/data-modeling/slowly-changing-dimensions-in-data-warehouse) (SCD). It differs from SCD1, which overwrites existing records, and only loosely resembles SCD2. While it adds new rows (like SCD2), it doesn’t manage versioning or track historical changes explicitly. Importantly, `append` doesn't check for duplicates or verify whether a record already exists in the destination. If the same record appears multiple times in the source, it will be inserted again, potentially resulting in duplicate rows. This may not be an issue depending on your use case and data quality requirements. ###### `delete+insert` The `delete+insert` strategy deletes the data for the `unique_key` from the target table and then inserts the data for those with a `unique_key`, which may be less efficient for larger datasets. It ensures updated records are fully replaced, avoiding partial updates and can be useful when a `unique_key` isn't truly unique or when `merge` is unsupported. `delete+insert` doesn't map directly to SCD logic (type 1 or 2) because it overwrites data at the row level and while it can add new rows, it does not track history. For SCD2, use [dbt snapshots](https://docs.getdbt.com/docs/build/snapshots.md#what-are-snapshots), not `delete+insert`. ###### `merge` `merge` inserts records with a `unique_key` that don’t exist yet in the destination table and updates records with keys that do exist — mirroring the logic of SCD1, where changes are overwritten rather than historically tracked. This strategy shouldn't be confused with `delete+insert` which deletes matching records before inserting new ones. By specifying a `unique_key` (which can be composed of one or more columns), `merge` can also help resolve duplicates. If the `unique_key` already exists in the destination table, `merge` will update the record, so you won't have duplicates. If the records don’t exist, `merge` will insert them. Note, if you use `merge` without specifying a `unique_key`, it behaves like the `append` strategy. While the `merge` strategy is useful for keeping tables current, it's best suited for smaller tables or incremental datasets. It can be expensive for large tables because it scans the entire destination table to determine what to update or insert. ###### `insert_overwrite` The [`insert_overwrite`](https://downloads.apache.org/spark/docs/3.1.1/sql-ref-syntax-dml-insert-overwrite-table.html) strategy is used to efficiently update partitioned tables by replacing entire partitions with new data, rather than merging or updating individual rows. It overwrites only the affected partitions, not the whole table. Because it is designed for partitioned data and replaces entire partitions wholesale, it does not align with typical SCD logic, which tracks row-level history or changes. It's ideal for tables partitioned by date or another key and useful for refreshing recent or corrected data without full table rebuilds. ###### `microbatch` [`microbatch`](https://docs.getdbt.com/docs/build/incremental-microbatch.md#what-is-microbatch-in-dbt) is an incremental strategy designed for processing large time-series datasets by splitting the data into time-based batches (for example, daily or hourly). It supports [parallel batch execution](https://docs.getdbt.com/docs/build/parallel-batch-execution.md#how-parallel-batch-execution-works) for faster runs. For details on which incremental strategies are supported by each adapter, refer to the section [Supported incremental strategies by adapter](https://docs.getdbt.com/docs/build/incremental-strategy.md#supported-incremental-strategies-by-adapter). ##### Custom strategies limited support Custom strategies are not currently supported on the BigQuery and Spark adapters. From dbt v1.2 and onwards, users have an easier alternative to [creating an entirely new materialization](https://docs.getdbt.com/guides/create-new-materializations.md). They define and use their own "custom" incremental strategies by: 1. Defining a macro named `get_incremental_STRATEGY_sql`. Note that `STRATEGY` is a placeholder and you should replace it with the name of your custom incremental strategy. 2. Configuring `incremental_strategy: STRATEGY` within an incremental model. dbt won't validate user-defined strategies, it will just look for the macro by that name, and raise an error if it can't find one. For example, a user-defined strategy named `insert_only` can be defined and used with the following files: macros/my\_custom\_strategies.sql ```sql {% macro get_incremental_insert_only_sql(arg_dict) %} {% do return(some_custom_macro_with_sql(arg_dict["target_relation"], arg_dict["temp_relation"], arg_dict["unique_key"], arg_dict["dest_columns"], arg_dict["incremental_predicates"])) %} {% endmacro %} {% macro some_custom_macro_with_sql(target_relation, temp_relation, unique_key, dest_columns, incremental_predicates) %} {%- set dest_cols_csv = get_quoted_csv(dest_columns | map(attribute="name")) -%} insert into {{ target_relation }} ({{ dest_cols_csv }}) ( select {{ dest_cols_csv }} from {{ temp_relation }} ) {% endmacro %} ``` models/my\_model.sql ```sql {{ config( materialized="incremental", incremental_strategy="insert_only", ... ) }} ... ``` If you use a custom microbatch macro, use the [`require_batched_execution_for_custom_microbatch_strategy` behavior flag](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_batched_execution_for_custom_microbatch_strategy.md) in your `dbt_project.yml` to control batched execution. Set it to `true` to opt in before the flag matures. After the flag matures (default: `true`), set it to `false` to revert to single-invocation behavior. ##### Custom strategies from a package To use the `merge_null_safe` custom incremental strategy from the `example` package: * [Install the package](https://docs.getdbt.com/docs/build/packages.md#how-do-i-add-a-package-to-my-project) * Add the following macro to your project: macros/my\_custom\_strategies.sql ```sql {% macro get_incremental_merge_null_safe_sql(arg_dict) %} {% do return(example.get_incremental_merge_null_safe_sql(arg_dict)) %} {% endmacro %} ``` --- ### About MetricFlow This guide introduces MetricFlow's fundamental ideas for people new to this feature. MetricFlow, which powers the Semantic Layer, helps you define and manage the logic for your company's metrics. It's an opinionated set of abstractions and helps data consumers retrieve metric datasets from a data platform quickly and efficiently. MetricFlow handles SQL query construction and defines the specification for dbt semantic models and metrics. It allows you to define metrics in your dbt project and query them with [MetricFlow commands](https://docs.getdbt.com/docs/build/metricflow-commands.md) whether in dbt or dbt v1. #### Prerequisites Before you start, consider the following guidelines: (Applies to dbt v1.12 and later) * Define metrics in YAML and query them using the [latest metric specifications](https://docs.getdbt.com/docs/build/semantic-models.md). * Available on the [dbt v2](https://docs.getdbt.com/docs/local/install-dbt.md) or [dbt Latest](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) in the dbt platform. * Use MetricFlow with Snowflake, BigQuery, Databricks, Postgres (dbt v1 only), or Redshift. * Discover insights and query your metrics using the [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) and its diverse range of [available integrations](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md). #### MetricFlow MetricFlow is a SQL query generation tool designed to streamline metric creation across different data dimensions for diverse business needs. * It operates through YAML files, where a semantic graph links language to data. This graph comprises [semantic models](https://docs.getdbt.com/docs/build/semantic-models.md) (data entry points) and [metrics](https://docs.getdbt.com/docs/build/metrics-overview.md) (functions for creating quantitative indicators). * MetricFlow is developed and maintained by dbt Labs and works with the [Apache Ossie](https://github.com/apache/ossie) format. Starting in dbt v1.12, you can also define semantic models using [Ossie documents](https://docs.getdbt.com/docs/build/ossie-semantic-models.md) as an alternative to dbt's native YAML configuration. * MetricFlow is compatible with dbt version 1.6 and higher. * MetricFlow is distributed under the [Apache 2.0 license](https://github.com/dbt-labs/metricflow/blob/main/LICENSE). Data practitioners and enthusiasts are highly encouraged to contribute. Read more about [MetricFlow's license history](https://github.com/dbt-labs/metricflow?tab=readme-ov-file#license-history). * As a part of the Semantic Layer, MetricFlow empowers organizations to define metrics using YAML abstractions. * To query metric dimensions, dimension values, and validate configurations, use [MetricFlow commands](https://docs.getdbt.com/docs/build/metricflow-commands.md). note MetricFlow doesn't support dbt [builtin functions or packages](https://docs.getdbt.com/reference/dbt-jinja-functions/builtins.md) at this time, however, support is planned for the future. MetricFlow abides by these principles: * **Flexibility with completeness**: Define metric logic using flexible abstractions on any data model. * **DRY (Don't Repeat Yourself)**: Minimize redundancy by enabling metric definitions whenever possible. * **Simplicity with gradual complexity:** Approach MetricFlow using familiar data modeling concepts. * **Performance and efficiency**: Optimize performance while supporting centralized data engineering and distributed logic ownership. ##### Semantic graph We're introducing a new concept: a "semantic graph". It's the relationship between semantic models and YAML configurations that creates a data landscape for building metrics. You can think of it like a map, where tables are like locations, and the connections between them (edges) are like roads. Although it's under the hood, the semantic graph is a subset of the DAG, and you can see the semantic models as nodes on the DAG. The semantic graph helps us decide which information is available to use for consumption and which is not. The connections between tables in the semantic graph are more about relationships between the information. This is different from the DAG, where the connections show dependencies between tasks. When MetricFlow generates a metric, it uses its SQL engine to figure out the best path between tables using the framework defined in YAML files for semantic models and metrics. When these models and metrics are correctly defined, they can be used downstream with Semantic Layer's integrations. ##### Semantic models Semantic models are the starting points of your data and correspond to models in your dbt project. You can create multiple semantic models from each model. Semantic models have metadata, like a data table, that define important information such as the table name and primary keys for the graph to be navigated correctly. For a semantic model, there are three main pieces of metadata: * [Entities](https://docs.getdbt.com/docs/build/entities.md): The join keys of your semantic model (think of these as the traversal paths, or edges between semantic models). * [Dimensions](https://docs.getdbt.com/docs/build/dimensions.md): These are the ways you want to group or slice/dice your metrics. (Applies to dbt v1.12 and later) * [Simple metrics](https://docs.getdbt.com/docs/build/simple.md): Metrics that directly reference a single column expression within a semantic model, without any additional columns involved. ##### Metrics (Applies to dbt v1.12 and later) Metrics, which is a key concept, are functions that combine simple metrics, constraints, or other mathematical functions to define new quantitative indicators. MetricFlow uses various aggregation types, such as average, sum, and count distinct, to create metrics. Dimensions add context to metrics and without them, a metric is simply a number for all time. You can define metrics in the same YAML files as your semantic models, or create a new file. MetricFlow supports different metric types: (Applies to dbt v1.12 and later) * [Conversion](https://docs.getdbt.com/docs/build/conversion.md): Tracks when a base event and a subsequent conversion event occurs for an entity within a set time period. * [Cumulative](https://docs.getdbt.com/docs/build/cumulative.md): Aggregates a simple metric over a given window. * [Derived](https://docs.getdbt.com/docs/build/derived.md): Defines a metric as an expression of other metrics, which allows you to do calculations on top of metrics. * [Ratio](https://docs.getdbt.com/docs/build/ratio.md): Defines a metric as the ratio of two simple metrics, such as revenue per customer. * [Simple](https://docs.getdbt.com/docs/build/simple.md): Defines a metric that directly references a single column expression within a semantic model. #### Use case In the upcoming sections, we'll show how data practitioners currently calculate metrics and compare it to how MetricFlow makes defining metrics easier and more flexible. The following example data is based on the Jaffle Shop repo. You can view the complete [dbt project](https://github.com/dbt-labs/jaffle-sl-template). The tables we're using in our example model are: * `orders` is a production data platform export that has been cleaned up and organized for analytical consumption * `customers` is a partially denormalized table in this case with a column derived from the orders table through some upstream process To make this more concrete, consider the metric `order_total`, which is defined using the SQL expression: `select sum(order_total) as order_total from orders` This expression calculates the total revenue for all orders by summing the `order_total` column in the orders table. In a business setting, the metric `order_total` is often calculated according to different categories, such as: * Time, for example `date_trunc(ordered_at, 'day')` * Order Type, using `is_food_order` dimension from the `orders` table ##### Calculate metrics Next, we'll compare how data practitioners currently calculate metrics with multiple queries versus how MetricFlow simplifies and streamlines the process. ##### Calculate with multiple queries The following example displays how data practitioners typically would calculate the `order_total` metric aggregated. It's also likely that analysts are asked for more details on a metric, like how much revenue came from new customers. Using the following query creates a situation where multiple analysts working on the same data, each using their own query method — this can lead to confusion, inconsistencies, and a headache for data management. ```sql select date_trunc('day',orders.ordered_at) as day, case when customers.first_ordered_at is not null then true else false end as is_new_customer, sum(orders.order_total) as order_total from orders left join customers on orders.customer_id = customers.customer_id group by 1, 2 ``` ##### Calculate with MetricFlow In the following three example tabs, use MetricFlow to define a semantic model that uses `order_total` as a metric and a sample schema to create consistent and accurate results — eliminating confusion, code duplication, and streamlining your workflow. ##### Revenue example (Applies to dbt v1.12 and later) In this example, a simple metric named `order_total` is defined on the `orders` model and semantic model. The metric sums the `order_total` column. The time dimension `metric_time` provides daily granularity and can be rolled up to weekly or monthly periods. Additionally, the `customers` semantic model defines a derived categorical dimension `is_new_customer`, which returns `true` when `first_ordered_at` is `not null` and `false` otherwise. ```yaml models: - name: orders # The name of the model semantic_model: enabled: true name: orders_semantic_model agg_time_dimension: metric_time # Default aggregation time dimension columns: # Primary entity - order_id - name: order_id description: "Primary key for orders table" entity: type: primary name: order_id label: "Order ID" # Foreign entity - customer - name: customer_id description: "Foreign key linking to customers" entity: type: foreign name: customer label: "Customer" # Time dimension - metric_time - name: ordered_at granularity: day dimension: type: time label: "Order Date" description: "Date when the order was placed" metrics: # Simple metric for order total revenue - name: order_total description: "Total revenue from orders" label: "Order Total Revenue" type: simple agg: sum expr: order_total - name: customers # The customers model with semantic layer constructs defined semantic_model: enabled: true name: customers_semantic_model agg_time_dimension: first_ordered_at columns: # Primary entity - customer - name: customer_id description: "Primary key for customers table" entity: type: primary name: customer label: "Customer" # Time dimension - first_ordered_at - name: first_ordered_at description: "Date of customer's first order" granularity: day dimension: type: time name: first_ordered_at label: "First Order Date" ``` ##### More dimensions example Similarly, you can add additional dimensions like `is_food_order` to your semantic models to incorporate even more dimensions to slice and dice your revenue `order_total`. (Applies to dbt v1.12 and later) ```yaml models: - name: orders # The name of the semantic model semantic_model: enabled: true name: orders_semantic_model agg_time_dimension: metric_time # Default aggregation time dimension columns: # Primary entity - order_id - name: order_id description: "Primary key for orders table" entity: type: primary name: order_id label: "Order ID" # Foreign entity - customer - name: customer_id description: "Foreign key linking to customers" entity: type: foreign name: customer label: "Customer" # Time dimension - metric_time - name: ordered_at description: "Date when the order was placed" granularity: day dimension: type: time name: metric_time label: "Order Date" # Categorical dimension - is_food_order - name: is_food_order description: "Indicates if this is a food order" dimension: type: categorical name: is_food_order label: "Is Food Order" metrics: # Simple metric for order total revenue - name: order_total description: "Total revenue from orders" label: "Order Total Revenue" type: simple agg: sum expr: order_total ``` ##### Advanced example Imagine an even more complex metric is needed, such as the amount of money earned each day from food orders from returning customers. Without MetricFlow, the data practitioner's original SQL might look like this: ```sql select date_trunc('day',orders.ordered_at) as day, sum(case when is_food_order = true then order_total else null end) as food_order, sum(orders.order_total) as sum_order_total, food_order/sum_order_total from orders left join customers on orders.customer_id = customers.customer_id where case when customers.first_ordered_at is not null then true else false end = true group by 1 ``` MetricFlow simplifies the SQL process through metric YAML configurations as shown below. You can also commit them to your git repository to ensure everyone on the data and business teams can see and approve them as the true and only source of information. (Applies to dbt v1.12 and later) ```yaml models: - name: orders semantic_model: enabled: true name: orders_semantic_model agg_time_dimension: ordered_at # Default aggregation time dimension columns: # Primary entity - order_id - name: order_id description: "Primary key for orders table" entity: type: primary name: order_id label: "Order ID" # Foreign entity - customer - name: customer_id description: "Foreign key linking to customers" entity: type: foreign name: customer label: "Customer" # Time dimension - ordered_at - name: ordered_at description: "Date when the order was placed" granularity: day dimension: type: time label: "Order Date" # Categorical dimension - is_food_order - name: is_food_order description: "Indicates if this is a food order" dimension: type: categorical name: is_food_order label: "Is Food Order" metrics: # Simple metric for total order revenue - name: order_total description: "Total revenue from orders" label: "Order Total Revenue" type: simple agg: sum expr: order_total # Simple metric for food order revenue - name: food_revenue description: "Revenue from food orders only" label: "Food order revenue" type: simple agg: sum expr: "case when is_food_order = true then order_total else 0 end" # Simple metric for count of distinct customers in orders - name: total_customers description: "Count of unique customers with orders" label: "Total customers" type: simple agg: count_distinct expr: customer_id - name: customers # The name of the second semantic model semantic_model: enabled: true name: customers_semantic_model agg_time_dimension: first_ordered_at columns: # Primary entity - customer - name: customer_id description: "Primary key for customers table" entity: type: primary name: customer label: "Customer" # Time dimension - first_ordered_at - name: first_ordered_at description: "Date of customer's first order" granularity: day dimension: type: time name: first_ordered_at label: "First Order Date" metrics: - name: food_revenue_per_customer description: "Revenue from food orders from returning customers" label: "Food % of order total" type: ratio numerator: name: food_revenue denominator: name: total_customers ``` #### FAQs  Do my datasets need to be normalized? Not at all! While a cleaned and well-modeled dataset can be extraordinarily powerful and is the ideal input, you can use any dataset from raw to fully denormalized datasets. It's recommended that you apply quality data consistency, such as filtering bad data, normalizing common objects, and data modeling of keys and tables, in upstream applications. The Semantic Layer is more efficient at doing data denormalization instead of normalization. If you have not invested in data consistency, that is okay. The Semantic Layer can take SQL queries or expressions to define consistent datasets.  Why is normalized data the ideal input? MetricFlow is built to do denormalization efficiently. There are better tools to take raw datasets and accomplish the various tasks required to build data consistency and organized data models. On the other end, by putting in denormalized data you are potentially creating redundancy which is technically challenging to manage, and you are reducing the potential granularity that MetricFlow can use to aggregate metrics.  How does the dbt Semantic Layer handle joins? The dbt Semantic Layer, powered by MetricFlow, builds joins based on the types of keys and parameters that are passed to entities. To better understand how joins are constructed, see the documentation on [join types](https://docs.getdbt.com/docs/build/join-logic.md#types-of-joins). Rather than capturing arbitrary join logic, MetricFlow captures the types of each identifier and then helps users navigate to appropriate joins. This allows us to avoid the construction of fan out and chasm joins as well as generate legible SQL.  Are entities and join keys the same thing? If it helps you to think of entities as join keys, that is very reasonable. Entities in MetricFlow have applications beyond joining two tables, such as acting as a dimension.  Can a table without a primary or unique entities have dimensions? Yes, but because a dimension is considered an attribute of the primary or unique entity of the table, they are only usable by the metrics that are defined in that table. They cannot be joined to metrics from other tables. This is common in event logs. #### Related docs * [Joins](https://docs.getdbt.com/docs/build/join-logic.md) * [Validations](https://docs.getdbt.com/docs/build/validation.md) --- ### About microbatch incremental models Use microbatch incremental models to process large time-series datasets efficiently. info Available for [dbt **v1 Latest**](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) and dbt v1.9 or higher. If you use a custom microbatch macro, set a [distinct behavior flag](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_batched_execution_for_custom_microbatch_strategy.md) in your `dbt_project.yml` to enable batched execution. If you don't have a custom microbatch macro, you don't need to set this flag as dbt will handle microbatching automatically for any model using the [microbatch strategy](#how-microbatch-compares-to-other-incremental-strategies). Read and participate in the discussion: [Core#10672](https://github.com/dbt-labs/dbt/discussions/10672). Refer to [Supported incremental strategies by adapter](https://docs.getdbt.com/docs/build/incremental-strategy.md#supported-incremental-strategies-by-adapter) for a list of supported adapters. #### What is "microbatch" in dbt? Incremental models in dbt are a [materialization](https://docs.getdbt.com/docs/build/materializations.md) designed to efficiently update your data warehouse tables by only transforming and loading *new or changed data* since the last run. Instead of reprocessing an entire dataset every time, incremental models process a smaller number of rows, and then append, update, or replace those rows in the existing table. This can significantly reduce the time and resources required for your data transformations. Microbatch is an incremental strategy designed for large time-series datasets: * It relies solely on a time column ([`event_time`](https://docs.getdbt.com/reference/resource-configs/event-time.md)) to define time-based ranges for filtering. * Set the `event_time` column for your microbatch model and its direct parents (upstream models). Note that this differs from `partition_by`, which groups rows into partitions. Required For incremental microbatch models, if your upstream models don't have `event_time` configured, dbt *cannot* automatically filter them during batch processing and will perform full table scans on every batch run. To avoid this, configure `event_time` on every upstream model that should be filtered. Learn how to exclude a model from auto-filtering by [opting out of auto-filtering](https://docs.getdbt.com/docs/build/incremental-microbatch.md#opting-out-of-auto-filtering). * It complements, rather than replaces, existing incremental strategies by focusing on efficiency and simplicity in batch processing. * Unlike traditional incremental strategies, microbatch enables you to [reprocess failed batches](https://docs.getdbt.com/docs/build/incremental-microbatch.md#retry), auto-detect [parallel batch execution](https://docs.getdbt.com/docs/build/parallel-batch-execution.md), and eliminate the need to implement complex conditional logic for [backfilling](#backfills). * Note that microbatch might not be the best [strategy](https://docs.getdbt.com/docs/build/incremental-strategy.md) for all use cases. Consider other strategies for use cases such as not having a reliable `event_time` column or if you want more control over the incremental logic. Read more in [How `microbatch` compares to other incremental strategies](#how-microbatch-compares-to-other-incremental-strategies). #### How microbatch works When dbt runs a microbatch model — whether for the first time, during incremental runs, or in specified backfills — it will split the processing into multiple queries (or "batches"), based on the `event_time` and `batch_size` you configure. Each "batch" corresponds to a single bounded time period (by default, a single day of data). Where other incremental strategies operate only on "old" and "new" data, microbatch models treat every batch as an atomic unit that can be built or replaced on its own. Each batch is independent and idempotent. This is a powerful abstraction that makes it possible for dbt to run batches [separately](#backfills), concurrently, and [retry](#retry) them independently. ##### Adapter-specific behavior dbt's microbatch strategy uses the most efficient mechanism available for "full batch" replacement on each adapter. This can vary depending on the adapter: * `dbt-postgres`: Uses the `merge` strategy, which performs "update" or "insert" operations. * `dbt-redshift`: Uses the `delete+insert` strategy, which "inserts" or "replaces." * `dbt-snowflake`: Uses the `delete+insert` strategy, which "inserts" or "replaces." * `dbt-bigquery`: Uses the `insert_overwrite` strategy, which "inserts" or "replaces." * `dbt-spark`: Uses the `insert_overwrite` strategy, which "inserts" or "replaces." * `dbt-databricks`: Uses the `replace_where` strategy, which "inserts" or "replaces." Check out the [supported incremental strategies by adapter](https://docs.getdbt.com/docs/build/incremental-strategy.md#supported-incremental-strategies-by-adapter) for more info. #### Example A `sessions` model aggregates and enriches data that comes from two other models: * `page_views` is a large, time-series table. It contains many rows, new records almost always arrive after existing ones, and existing records rarely update. It uses the `page_view_start` column as its `event_time`. * `customers` is a relatively small dimensional table. Customer attributes update often, and not in a time-based manner — that is, older customers are just as likely to change column values as newer customers. The customers model doesn't configure an `event_time` column. As a result: * Each batch of `sessions` will filter `page_views` to the equivalent time-bounded batch. * The `customers` table isn't filtered, resulting in a full scan for every batch. tip In addition to configuring `event_time` for the target table, you should also specify it for any upstream models that you want to filter, even if they have different time columns. models/staging/page\_views.yml ```yaml models: - name: page_views config: event_time: page_view_start ``` We run the `sessions` model for October 1, 2024, and then again for October 2. It produces the following queries: ##### Model definition The [`event_time`](https://docs.getdbt.com/reference/resource-configs/event-time.md) for the `sessions` model is set to `session_start`, which marks the beginning of a user’s session on the website. This setting allows dbt to combine multiple page views (each tracked by their own `page_view_start` timestamps) into a single session. This way, `session_start` differentiates the timing of individual page views from the broader timeframe of the entire user session. models/sessions.sql ```sql {{ config( materialized='incremental', incremental_strategy='microbatch', event_time='session_start', begin='2020-01-01', batch_size='day' ) }} with page_views as ( -- this ref will be auto-filtered select * from {{ ref('page_views') }} ), customers as ( -- this ref won't select * from {{ ref('customers') }} ) select page_views.id as session_id, page_views.page_view_start as session_start, customers.* from page_views left join customers on page_views.customer_id = customers.id ``` ##### Compiled (Oct 1, 2024) target/compiled/sessions.sql ```sql with page_views as ( select * from ( -- filtered on configured event_time select * from "analytics"."page_views" where page_view_start >= '2024-10-01 00:00:00' -- Oct 1 and page_view_start < '2024-10-02 00:00:00' ) ), customers as ( select * from "analytics"."customers" ), ... ``` ##### Compiled (Oct 2, 2024) target/compiled/sessions.sql ```sql with page_views as ( select * from ( -- filtered on configured event_time select * from "analytics"."page_views" where page_view_start >= '2024-10-02 00:00:00' -- Oct 2 and page_view_start < '2024-10-03 00:00:00' ) ), customers as ( select * from "analytics"."customers" ), ... ``` dbt will instruct the data platform to take the result of each batch query and [insert, update, or replace](#adapter-specific-behavior) the contents of the `analytics.sessions` table for the same day of data. To perform this operation, dbt will use the most efficient atomic mechanism for "full batch" replacement that is available on each data platform. For details, see [How microbatch works](#how-microbatch-works). It does not matter whether the table already contains data for that day. Given the same input data, the resulting table is the same no matter how many times a batch is reprocessed. [![Each batch of sessions filters page\_views to the matching time-bound batch, but doesn't filter sessions, performing a full scan for each batch.](/img/docs/building-a-dbt-project/microbatch/microbatch_filters.png?v=2 "Each batch of sessions filters page_views to the matching time-bound batch, but doesn't filter sessions, performing a full scan for each batch.")](#)Each batch of sessions filters page\_views to the matching time-bound batch, but doesn't filter sessions, performing a full scan for each batch. #### Relevant configs Several configurations are relevant to microbatch models, and some are required: | Config | Description | Default | Type | Required | | --------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------- | -------- | | [`event_time`](https://docs.getdbt.com/reference/resource-configs/event-time.md) | The column indicating "at what time did the row occur." Required for your microbatch model and any direct parents that should be filtered. | N/A | Column | Required | | [`begin`](https://docs.getdbt.com/reference/resource-configs/begin.md) | The "beginning of time" for the microbatch model. This is the starting point for any initial or full-refresh builds. For example, a daily-grain microbatch model run on `2024-10-01` with `begin = '2023-10-01` will process 366 batches (it's a leap year!) plus the batch for "today." | N/A | Date | Required | | [`batch_size`](https://docs.getdbt.com/reference/resource-configs/batch-size.md) | The granularity of your batches. Supported values are `hour`, `day`, `month`, and `year` | N/A | String | Required | | [`lookback`](https://docs.getdbt.com/reference/resource-configs/lookback.md) | Process X batches prior to the latest bookmark to capture late-arriving records. | `1` | Integer | Optional | | [`concurrent_batches`](https://docs.getdbt.com/reference/resource-properties/concurrent_batches.md) | Overrides dbt's auto detect for running batches concurrently (at the same time). Read more about [configuring concurrent batches](https://docs.getdbt.com/docs/build/parallel-batch-execution.md#configure-concurrent_batches). Setting to<br />\* `true` runs batches concurrently (in parallel).<br />\* `false` runs batches sequentially (one after the other). | `None` | Boolean | Optional | [![The event\_time column configures the real-world time of this record](/img/docs/building-a-dbt-project/microbatch/event_time.png?v=2 "The event_time column configures the real-world time of this record")](#)The event\_time column configures the real-world time of this record ##### Required configs for specific adapters Some adapters require additional configurations for the microbatch strategy. This is because each adapter implements the microbatch strategy differently. The following table lists the required configurations for the specific adapters, in addition to the standard microbatch configs: | Adapter | `unique_key` config | `partition_by` config | | ------------------------------------------------------------------------------------------------------------------------------- | ------------------- | --------------------- | | [`dbt-postgres`](https://docs.getdbt.com/reference/resource-configs/postgres-configs.md#incremental-materialization-strategies) | ✅ Required | N/A | | [`dbt-spark`](https://docs.getdbt.com/reference/resource-configs/spark-configs.md#incremental-models) | N/A | ✅ Required | | [`dbt-bigquery`](https://docs.getdbt.com/reference/resource-configs/bigquery-configs.md#merge-behavior-incremental-models) | N/A | ✅ Required | For example, if you're using `dbt-postgres`, configure `unique_key` as follows: models/sessions.sql ```sql {{ config( materialized='incremental', incremental_strategy='microbatch', unique_key='sales_id', ## required for dbt-postgres event_time='transaction_date', begin='2023-01-01', batch_size='day' ) }} select sales_id, transaction_date, customer_id, product_id, total_amount from {{ source('sales', 'transactions') }} ``` In this example, `unique_key` is required because `dbt-postgres` microbatch uses the `merge` strategy, which needs a `unique_key` to identify which rows dbt should merge in the data warehouse. Without a `unique_key`, dbt can't match rows between the incoming batch and the existing table. ##### Full refresh As a best practice, we recommend [configuring `full_refresh: false`](https://docs.getdbt.com/reference/resource-configs/full_refresh.md) on microbatch models so that they ignore invocations with the `--full-refresh` flag. Note that running `dbt run --full-refresh` on a microbatch model by itself won't reset or reload data unless you have a `begin` datetime config for the model. If you need to reprocess historical data, we recommend using a targeted backfill with `--event-time-start` and `--event-time-end`. You must configure both for the full refresh to successfully run. ```bash dbt run --full-refresh --event-time-start "2024-01-01" --event-time-end "2024-02-01" ``` #### Usage **You must write your model query to process (read and return) exactly one "batch" of data**. This is a simplifying assumption and a powerful one: * You don’t need to think about `is_incremental` filtering * You don't need to pick among DML strategies (upserting/merging/replacing) * You can preview your model, and see the exact records for a given batch that will appear when that batch is processed and written to the table When you run a microbatch model, dbt will evaluate which batches need to be loaded, break them up into a SQL query per batch, and load each one independently. dbt will automatically filter upstream inputs (`source` or `ref`) that define `event_time`, based on the `lookback` and `batch_size` configs for this model. Note that dbt doesn't know the minimum `event_time` in your data — it only uses the configs you provide (like `begin`, `lookback`) to decide which batches to run. If you want to process data from the actual start of your dataset, you *must* explicitly define it using the `begin` config or the `--event-time-start` flag. During standard incremental runs, dbt will process batches according to the current timestamp and the configured `lookback`, with one query per batch. [![Configure a lookback to reprocess additional batches during standard incremental runs](/img/docs/building-a-dbt-project/microbatch/microbatch_lookback.png?v=2 "Configure a lookback to reprocess additional batches during standard incremental runs")](#)Configure a lookback to reprocess additional batches during standard incremental runs ###### Opting out of auto-filtering If there's an upstream model that configures `event_time`, but you *don't* want the reference to it to be filtered, you can specify `ref('upstream_model').render()` to opt out of auto-filtering. This isn't generally recommended — most models that configure `event_time` are fairly large, and if you don't filter the reference, each batch performs a full scan of this input table. #### Backfills Whether to fix erroneous source data or retroactively apply a change in business logic, you may need to reprocess a large amount of historical data. Backfilling a microbatch model is as simple as selecting it to run or build, and specifying a "start" and "end" for `event_time`. Note that `--event-time-start` and `--event-time-end` are mutually necessary, meaning that if you specify one, you must specify the other. As always, dbt will process the batches between the start and end as independent queries. ```bash dbt run --event-time-start "2024-09-01" --event-time-end "2024-09-04" ``` [![Configure a lookback to reprocess additional batches during standard incremental runs](/img/docs/building-a-dbt-project/microbatch/microbatch_backfill.png?v=2 "Configure a lookback to reprocess additional batches during standard incremental runs")](#)Configure a lookback to reprocess additional batches during standard incremental runs #### Retry If one or more of your batches fail, you can use `dbt retry` to reprocess *only* the failed batches. ![Partial retry](https://github.com/user-attachments/assets/f94c4797-dcc7-4875-9623-639f70c97b8f) #### Timezones For now, dbt assumes that all values supplied are in UTC: * `event_time` * `begin` * `--event-time-start` * `--event-time-end` While we may consider adding support for custom time zones in the future, we also believe that defining these values in UTC makes everyone's lives easier. #### How microbatch compares to other incremental strategies As data warehouses roll out new operations for concurrently replacing/upserting data partitions, we may find that the new operation for the data warehouse is more efficient than what the adapter uses for microbatch. In such instances, we reserve the right the update the default operation for microbatch, so long as it works as intended/documented for models that fit the microbatch paradigm. Most incremental models rely on the end user (you) to explicitly tell dbt what "new" means, in the context of each model, by writing a filter in an `{% if is_incremental() %}` conditional block. You are responsible for crafting this SQL in a way that queries [`{{ this }}`](https://docs.getdbt.com/reference/dbt-jinja-functions/this.md) to check when the most recent record was last loaded, with an optional look-back window for late-arriving records. Other incremental strategies will control *how* the data is being added into the table — whether append-only `insert`, `delete` + `insert`, `merge`, `insert overwrite`, etc — but they all have this in common. As an example: ```sql {{ config( materialized='incremental', incremental_strategy='delete+insert', unique_key='date_day' ) }} select * from {{ ref('stg_events') }} {% if is_incremental() %} -- this filter will only be applied on an incremental run -- add a lookback window of 3 days to account for late-arriving records where date_day >= (select {{ dbt.dateadd("day", -3, "max(date_day)") }} from {{ this }}) {% endif %} ``` For this incremental model: * "New" records are those with a `date_day` greater than the maximum `date_day` that has previously been loaded * The lookback window is 3 days * When there are new records for a given `date_day`, the existing data for `date_day` is deleted and the new data is inserted Let’s take our same example from before, and instead use the new `microbatch` incremental strategy: models/staging/stg\_events.sql ```sql {{ config( materialized='incremental', incremental_strategy='microbatch', event_time='event_occurred_at', batch_size='day', lookback=3, begin='2020-01-01', full_refresh=false ) }} select * from {{ ref('stg_events') }} -- this ref will be auto-filtered ``` Where you’ve also set an `event_time` for the model’s direct parents - in this case, `stg_events`: models/staging/stg\_events.yml ```yaml models: - name: stg_events config: event_time: my_time_field ``` And that’s it! When you run the model, each batch templates a separate query. For example, if you were running the model on October 1, dbt would template separate queries for each day between September 28 and October 1, inclusive — four batches in total. The query for `2024-10-01` would look like: target/compiled/staging/stg\_events.sql ```sql select * from ( select * from "analytics"."stg_events" where my_time_field >= '2024-10-01 00:00:00' and my_time_field < '2024-10-02 00:00:00' ) ``` Based on your data platform, dbt will choose the most efficient atomic mechanism to insert, update, or replace these four batches (`2024-09-28`, `2024-09-29`, `2024-09-30`, and `2024-10-01`) in the existing table. --- ### About model governance dbt supports model governance to help you control who can access models, what data they contain, how they change over time, and reference them across projects. dbt supports model governance in dbt v1 and the dbt platform, with some differences in the features available across environments/plans. * Use model governance to define model structure and visibility in dbt v1 and the dbt platform. * dbt builds on this with features like [cross-project ref](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md) that enable collaboration at scale across multiple projects, powered by its metadata service and [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md). Available in dbt Enterprise or Enterprise+ plans. All of the following features are available in dbt v1 and the dbt platform, *except* project dependencies, which is only available to [dbt Enterprise-tier plans](https://www.getdbt.com/pricing). * [**Model access**](https://docs.getdbt.com/docs/mesh/govern/model-access.md) — Mark models as "public" or "private" to distinguish between mature data products and implementation details — and to control who can `ref` each. * [**Model contracts**](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md) —Guarantee the shape of a model (column names, data types, constraints) before it builds, to prevent surprises for downstream data consumers. * [**Model versions**](https://docs.getdbt.com/docs/mesh/govern/model-versions.md) — When a breaking change is unavoidable, provide a smoother upgrade pathway and deprecation window for downstream data consumers. * [**Model namespaces**](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models) — Organize models into [groups](https://docs.getdbt.com/docs/build/groups.md) and [packages](https://docs.getdbt.com/docs/build/packages.md) to delineate ownership boundaries. Models in different packages can share the same name, and the `ref` function can take the project/package namespace as its first argument. * [**Project dependencies**](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md) — Resolve references to public models in other projects ("cross-project ref") using an always-on stateful metadata service, instead of importing all models from those projects as packages. Each project serves data products (public model references) while managing its own implementation details, enabling an [enterprise data mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md). [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") Supporting data freshness SLAs Use [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) and the [`lag_tolerance`](https://docs.getdbt.com/reference/resource-configs/lag-tolerance.md) config to govern data freshness at the model level, helping your team align with freshness Service Level Agreements (SLAs) without unnecessary rebuilds. ###### Considerations There are some considerations to keep in mind when using model governance features: * Model governance features like model access, contracts, and versions strengthen trust and stability in your dbt project. Because they add structure, they can make rollbacks harder (for example, removing model access) and increase maintenance if adopted too early. Before adding governance features, consider whether your dbt project is ready to benefit from them. Introducing governance while models are still changing can complicate future changes. * Governance features are model-specific. They don't apply to other resource types, including snapshots, seeds, or sources. This is because these objects can change structure over time (for example, snapshots capture evolving historical data) and aren't suited to guarantees like contracts, access, or versioning. --- ### About profiles.yml Local development If you're using dbt from the command line, you need a `profiles.yml` file that contains the connection details for your data platform. dbt platform accounts dbt platform projects don't require a profiles.yml file unless you're developing from your local machine instead of the cloud-based UI. #### About profiles.yml The `profiles.yml` file stores database connection credentials and configuration for dbt projects, including: * **Connection details** — Account identifiers, hosts, ports, and authentication credentials. * **Target definitions** — Define different environments (dev, staging, prod) within a single profile. * **Default target** — Set which environment to use by default. * **Execution parameters** — Thread count, timeouts, and retry settings. * **Credential separation** — Keep sensitive information out of version control. The `profile` field in [`dbt_project.yml`](https://docs.getdbt.com/reference/dbt_project.yml.md) references a profile name defined in `profiles.yml`. #### Location of profiles.yml Only one `profiles.yml` file is required and it can manage multiple projects and connections. ##### dbt v2 dbt v2 searches for the parent directory of `profiles.yml` in the following order and uses the first location it finds: 1. `--profiles-dir` flag — Override for CI/CD or testing. 2. Project root directory — Project-specific credentials. 3. `~/.dbt/` directory (Recommended location) — Shared across all projects. ##### dbt v1 dbt v1 searches for the parent directory of `profiles.yml` in the following order and uses the first location it finds: (Applies to dbt v1.11 and later) 1. `--profiles-dir` flag 2. `DBT_ENGINE_PROFILES_DIR` environment variable 3. `DBT_PROFILES_DIR` environment variable (legacy variable but supported for backward compatibility) 4. Current working directory 5. `~/.dbt/` directory (Recommended location) **Note:** dbt v1 prefers `DBT_ENGINE_PROFILES_DIR` for the profiles directory, which aligns with the `DBT_ENGINE_*` env var naming in v1.11. Use `DBT_ENGINE_PROFILES_DIR` going forward; `DBT_PROFILES_DIR` remains supported for compatibility. Note: dbt v1 supports using the (Applies to dbt v1.11 and later) `DBT_ENGINE_PROFILES_DIR` environment variable or a `profiles.yml` file in the current working directory. These options aren't currently supported in dbt v2. `~/.dbt/profiles.yml` is the recommended location for the following reasons: * **Security** — Keeps credentials out of project directories and version control. * **Reusability** — A single file for all dbt projects on the machine. * **Separation** — Connection details don't travel with project code. ###### When should I use project root? Place your `profiles.yml` file in the project root directory for: * Self-contained demo or tutorial projects. * Docker containers with baked-in credentials. * CI/CD pipelines with environment-specific configs. #### Create and configure the `profiles.yml` file The easiest way to create and configure a `profiles.yml` file is to execute `dbt init` after you've installed dbt on your machine. This takes you through the process of configuring an adapter and places the file into the recommended `~/.dbt/` location. If your project has an existing `profiles.yml` file, running `dbt init` will prompt you to amend or overwrite it. If you select the existing adapter for configuration, dbt will automatically populate the existing values. You can also manually create the file and add it to the proper location. To configure an adapter manually, copy and paste the fields from the adapter setup instructions for [dbt v1](https://docs.getdbt.com/docs/local/connect-data-platform/about-dbt-connections.md) or [dbt v2](https://docs.getdbt.com/docs/local/profiles.yml.md) along with the appropriate values for each. ##### Example configuration To set up your profile, copy the correct sample profile for your warehouse into your `profiles.yml` file and update the details as follows: * Profile name: Replace the name of the profile with a sensible name – it’s often a good idea to use the name of your organization. Make sure that this is the same name as the `profile` indicated in your `dbt_project.yml` file. * `target`: This is the default target your dbt project will use. It must be one of the targets you define in your profile. Commonly it is set to `dev`. * Populating your `outputs`: * `type`: The type of data warehouse you are connecting to * Warehouse credentials: Get these from your database administrator if you don’t already have them. Remember that user credentials are very sensitive information that should not be shared. May include fields like `account`, `username`, and `password`. * `schema`: The default schema that dbt will build objects in. * `threads`: The number of threads the dbt project will run on. The following example highlighs the format of the `profiles.yml` file. Note that many of the configs are adapter-specific and their syntax varies. \~/.dbt/profiles.yml ```yml my_project_profile: # Profile name (matches dbt_project.yml) target: dev # Default target to use outputs: dev: # Development environment type: adapter_type # Required: snowflake, bigquery, databricks, redshift, postgres, etc # Connection identifiers (placeholder examples, see adapter-specific pages for supported configs) account: abc123 database: docs_team schema: dev_schema # Authentication (adapter-specific) auth_method: username_password username: username password_credentials: password # Execution settings (common across adapters) threads: 4 # Number of parallel threads # Multiple profiles (for multiple projects) my_second_project_profile: target: dev outputs: dev: type: snowflake # Example adapter account: account user: user password: password database: database schema: schema warehouse: warehouse threads: 4 ``` ##### Environment variables Use environment variables to keep sensitive credentials out of your `profiles.yml` file. Check out the [env\_var](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md) reference for more information. Example: \~/.dbt/profiles.yml ```yml my_profile: target: dev outputs: dev: type: ADAPTER_NAME account: "{{ env_var("ADAPTER_ACCOUNT") }}" user: "{{ env_var("ADAPTER_USER") }}" password: "{{ env_var("ADAPTER_PASSWORD") }}" database: "{{ env_var("ADAPTER_DATABASE") }}" schema: "{{ env_var("ADAPTER_SCHEMA") }}" warehouse: "{{ env_var("ADAPTER_WAREHOUSE") }}" role: "{{ env_var("ADAPTER_ROLE") }}" threads: 4 ``` When using dbt locally, you can also store environment variables in a `.env` file in your project root instead of setting them directly in your shell. dbt, the dbt VS Code extension, and dbt v1.12+ automatically load the `.env` file from your current working directory. Environment variables set in your shell take precedence over values in the `.env` file. For more information, refer to [About env\_var function](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md#using-the-env-file). To keep credentials out of version control, add `.env` to your `.gitignore` file — new projects on v1.12 and higher created with `dbt init` include this by default. #### User config You can set default values of global configs for all projects that you run using your local machine. Refer to [About global configs](https://docs.getdbt.com/reference/global-configs/about-global-configs.md) for details. #### Understanding targets in profiles dbt supports multiple targets within one profile to encourage the use of separate development and production environments as discussed in [dbt environments](https://docs.getdbt.com/docs/local/dbt-environments.md). A typical profile for an analyst using dbt locally will have a target named `dev`, and have this set as the default. You may also have a `prod` target within your profile, which creates the objects in your production schema. However, since it's often desirable to perform production runs on a schedule, we recommend deploying your dbt project to a separate machine other than your local machine. Most dbt users only have a `dev` target in their profile on their local machine. If you do have multiple targets in your profile, and want to use a target other than the default, you can do this using the `--target` flag when running a dbt command. For example, to run against your `prod` target instead of the default `dev` target: ```bash dbt run --target prod ``` You can use the `--target` flag with any dbt command, such as: ```bash dbt build --target prod dbt test --target dev dbt compile --target qa ``` ##### Overriding profiles and targets When running dbt commands, you can specify which profile and target to use from the CLI using the `--profile` and `--target` [flags](https://docs.getdbt.com/reference/global-configs/about-global-configs.md#available-flags). These flags override what’s defined in your `dbt_project.yml` as long as the specified profile and target are already defined in your `profiles.yml` file. To run your dbt project with a different profile or target than the default, you can do so using the followingCLI flags: * `--profile` flag — Overrides the profile set in `dbt_project.yml` by pointing to another profile defined in `profiles.yml`. * `--target` flag — Specifies the target within that profile to use (as defined in `profiles.yml`). These flags help when you're working with multiple profiles and targets and want to override defaults without changing your files. ```bash dbt run --profile my-profile-name --target dev ``` In this example, the `dbt run` command will use the `my-profile-name` profile and the `dev` target. #### Understanding warehouse credentials We recommend that each dbt user has their own set of database credentials, including a separate user for production runs of dbt – this helps debug rogue queries, simplifies ownerships of schemas, and improves security. To ensure the user credentials you use in your target allow dbt to run, you will need to ensure the user has appropriate privileges. While the exact privileges needed varies between data warehouses, at a minimum your user must be able to: * Read source data * Create schemas¹ * Read system tables Running dbt without create schema privileges If your user is unable to be granted the privilege to create schemas, your dbt runs should instead target an existing schema that your user has permission to create relations within. #### Understanding target schemas The target schema represents the default schema that dbt will build objects into, and is often used as the differentiator between separate environments within a warehouse. Schemas in BigQuery dbt uses the term "schema" in a target across all supported warehouses for consistency. Note that in the case of BigQuery, a schema is actually a dataset. The schema used for production should be named in a way that makes it clear that it is ready for end-users to use for analysis – we often name this `analytics`. In development, a pattern we’ve found to work well is to name the schema in your `dev` target `dbt_<username>`. Suffixing your name to the schema enables multiple users to develop in dbt, since each user will have their own separate schema for development, so that users will not build over the top of each other, and ensuring that object ownership and permissions are consistent across an entire schema. Note that there’s no need to create your target schema beforehand – dbt will check if the schema already exists when it runs, and create it if it doesn’t. While the target schema represents the default schema that dbt will use, it may make sense to split your models into separate schemas, which can be done by using [custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md). #### Understanding threads When dbt runs, it creates a directed acyclic graph (DAG) of links between models. The number of threads represents the maximum number of paths through the graph dbt may work on at once – increasing the number of threads can minimize the run time of your project. The default value for threads in user profiles is 4 threads. For more information, check out [using threads](https://docs.getdbt.com/docs/running-a-dbt-project/using-threads.md). #### Related docs * [Install dbt](https://docs.getdbt.com/docs/local/install-dbt.md) * [Connection profiles](https://docs.getdbt.com/docs/local/connection-profiles.md) --- ### About state-aware orchestration Private preview ### About state-aware orchestration [Private preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt platform | Enterprise, Enterprise+ Every time a job runs, state-aware orchestration automatically determines which models to build by detecting changes in code or data. State-aware orchestration is now dbt State [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) works with all engines and environments: dbt v1, dbt platform, and dbt v2 If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). important dbt v2 is currently available for installation in: * [Local command line interface (CLI) tools](https://docs.getdbt.com/docs/local/install-dbt.md?version=2) [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") * [VS Code and Cursor with the dbt extension](https://docs.getdbt.com/docs/install-dbt-extension.md) [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") * [dbt platform environments](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md#dbt-v2) Join the conversation in our Community Slack channel [`#dbt-fusion-engine`](https://getdbt.slack.com/archives/C088YCAB6GH). State-aware orchestration saves you compute costs and reduces runtime because when a job runs, it checks for new records and only builds the models that will change. [![dbt v2 powered state-aware orchestration](/img/docs/deploy/sao.gif?v=2 "dbt v2 powered state-aware orchestration")](#)dbt v2 powered state-aware orchestration We built dbt's state-aware orchestration on these four core principles: * **Real-time shared state:** All jobs write to a real-time shared model-level state, allowing dbt to rebuild only changed models regardless of which jobs the model is built in. * **Model-level queueing:** Jobs queue up at the model-level so you can avoid any 'collisions' and prevent rebuilding models that were just updated by another job. * **State-aware and state agnostic support:** You can build jobs dynamically (state-aware) or explicitly (state-agnostic). Both approaches update shared state so everything is kept in sync. * **Sensible defaults:** State-aware orchestration works out-of-the-box (natively), with an optional configuration setting for more advanced controls. For more information, refer to [state-aware advanced configurations](https://docs.getdbt.com/docs/deploy/state-aware-setup.md#advanced-configurations). note State-aware orchestration does not depend on [static analysis](https://docs.getdbt.com/docs/build/about-static-analysis.md#principles-of-static-analysis) and works even when `static_analysis` is disabled. #### Optimizing builds with state-aware orchestration State-aware orchestration uses shared state tracking to determine which models need to be built by detecting changes in code or data every time a job runs. It also supports custom refresh intervals and custom source freshness configurations, so dbt only rebuilds models when they're actually needed. For example, you can configure your project so that dbt skips rebuilding the `dim_wizards` model (and its parents) if they’ve already been refreshed within the last 4 hours, even if the job itself runs more frequently. Without configuring anything, dbt's state-aware orchestration automatically knows to build your models either when the code has changed or if there’s any new data in a source (or upstream model in the case of [dbt Mesh](https://docs.getdbt.com/docs/mesh/about-mesh.md)). **Note:** When a model fails a [data test](https://docs.getdbt.com/docs/build/data-tests.md), state-aware orchestration rebuilds it on subsequent runs instead of reusing it from prior state. This ensures dbt reevaluates models with unresolved data quality issues. ##### Handling concurrent jobs Not supported in dbt State Concurrent job handling works differently in dbt State. For a full list of behavioral differences, refer to [Known differences from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md#known-differences-from-state-aware-orchestration). If two separate jobs both depend on the same downstream model (for example, `model_ab`) and both detect upstream changes (`updates_on = any`), `model_ab` could run twice — once for each job. However, if `model_ab` was already built and nothing has changed since that build, neither job will rebuild it. Instead, both jobs will reuse the existing version instead of rebuilding. Under state-aware orchestration, all jobs read and write from the same shared state and build a model only when either the code or data state has changed. This means that each job individually evaulates whether a model needs rebuilding based on the model’s compiled code and upstream data state. What happens when jobs overlap: * If both jobs reach the same model at exactly the same time, one job waits until the other finishes. This is to prevent collisions in the data warehouse when two jobs try to build the same model at the same time. * After the first job finishes building the model, the second job still checks whether a rebuild is needed. If there are new data or code changes to incorporate, the second job builds the model again. If there are no changes and building the model would produce the same result, the second job reuses the model. To prevent a job from being built too frequently even when the code or data state has changed, you can reduce build frequency by using the `build_after` config. For information on how to use `build_after`, refer to [Model freshness](https://docs.getdbt.com/reference/resource-configs/freshness.md) and [Advanced configurations](https://docs.getdbt.com/docs/deploy/state-aware-setup.md#advanced-configurations). ##### Handling deleted tables State-aware orchestration detects and rebuilds models when their tables are deleted in the warehouse, even if there are no code or data changes. When a table is deleted in the warehouse: * dbt raises a warning that the expected table is missing. * The affected model is queued for rebuild during the current run, even if there are no code or data changes. This behavior ensures consistency between the dbt state and the actual warehouse state. It also reduces the need to manually clear cache or disable state-aware orchestration when models are modified outside of dbt. #### Efficient testing in state-aware orchestration [Private beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Private beta feature State-aware orchestration features in the dbt platform are only available in dbt v2. Contact your account manager to enable dbt v2 in your account. Data quality can get degraded in two ways: * New code changes definitions or introduces edge cases. * New data, like duplicates or unexpected values, invalidates downstream metrics. Running dbt’s out-of-the-box [data tests](https://docs.getdbt.com/docs/build/data-tests.md) (`unique`, `not_null`, `accepted_values`, `relationships`) on every build helps catch data errors before they impact business decisions. Catching these errors often requires having multiple tests on every model and running tests even when not necessary. If nothing relevant has changed, repeated test executions don’t improve coverage and only increase cost. With dbt v2, dbt gains an understanding of the SQL code based on the logical plan for the compiled code. dbt then can determine when a test must run again, or when a prior upstream test result can be reused. Efficient testing in state-aware orchestration reduces warehouse costs by avoiding redundant data tests and combining multiple tests into one run. This feature includes two optimizations: * **Test reuse** — Tests are reused in cases where no logic in the code or no new data could have changed the test's outcome. * **Test aggregation** — When there are multiple tests on a model, dbt combines tests to run as a single query against the warehouse, rather than running separate queries for each test. Currently, Efficient testing is only available in deploy jobs, not in continuous integration (CI) or merge jobs. ##### Supported data tests The following tests can be reused when Efficient testing is enabled: * [`unique`](https://docs.getdbt.com/reference/resource-properties/data-tests.md#unique) * [`not_null`](https://docs.getdbt.com/reference/resource-properties/data-tests.md#not_null) * [`accepted_values`](https://docs.getdbt.com/reference/resource-properties/data-tests.md#accepted_values) ##### Enabling Efficient testing Before enabling Efficient testing, make sure you have configured [`static_analysis`](https://docs.getdbt.com/docs/build/about-static-analysis.md#configuring-static_analysis). To enable Efficient testing: 1. From the main menu, go to **Orchestration** > **Jobs**. 2. Select your deploy job. Go to your job settings and click **Edit**. 3. Under **Enable dbt v2 cost optimization features**, expand **More options**. 4. Select **Efficient testing**. This feature is disabled by default. 5. Click **Save**. ##### Example In the following query, you’re joining an `orders` and a `customers` table: ```sql with orders as ( select * from {{ ref('orders') }} ), customers as ( select * from {{ ref('customers') }} ), joined as ( select customers.customer_id as customer_id, orders.order_id as order_id from customers left join orders on orders.customer_id = customers.customer_id ) select * from joined ``` * `not_null` test: A `left join` can introduce null values for customers without orders. Even if upstream tests verified `not_null(order_id)` in orders, the join can create null values downstream. dbt must always run a `not_null` test on `order_id` in this joined result. * `unique` test: If `orders.order_id` and `customers.customer_id` are unique upstream, uniqueness of `order_id` is preserved and the upstream result can be reused. ##### Limitations The following section lists some considerations when using Efficient testing in state-aware-orchestration: * **Aggregated tests do not support custom configs**. Tests that include the following [custom config options](https://docs.getdbt.com/reference/data-test-configs.md) will run individually rather than as part of the aggregated batch: ```yaml config: fail_calc: <string> limit: <integer> severity: error | warn error_if: <string> warn_if: <string> store_failures: true | false where: <string> ``` * **Efficient testing is available only in deploy jobs**. CI and merge jobs currently do not have the option to enable this feature. #### Related FAQs What happened to state-aware orchestration? On June 1, 2026, dbt Labs and Fivetran announced **[dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md)**[Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") as a new and improved version of state-aware orchestration. A key feature is [`lag_tolerance`](https://docs.getdbt.com/reference/resource-configs/lag-tolerance.md), which controls how much time must pass since the last upstream data change before a node is eligible for a rebuild. dbt State improves upon state-aware orchestration in a few key ways: * **Works everywhere** — dbt State works with dbt v1, dbt v2, and dbt platform, as well as external orchestrators, across both development and deployment environments. * **Smarter data freshness tracking** — dbt State tracks data freshness across the DAG and automatically propagates it through models materialized as views. Unlike state-aware orchestration's `build_after` config which compares against the model's last successful execution, dbt State's `lag_tolerance` compares against the freshness of the underlying data. * **Advanced change detection** — dbt State can detect and ignore file modifications that don't change actual transformation logic, such as adding a comment or cleaning up whitespace. If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. For details on billing after the trial ends, refer to [dbt State usage and pricing](https://docs.getdbt.com/docs/platform/billing.md#dbt-state-usage). While dbt State is in preview, there is no required migration timeline — dbt Labs will communicate a timeline when dbt State reaches general availability. To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). ### How is state-aware orchestration different from using selectors in dbt? dbt platform | Enterprise, Enterprise+ In dbt v1, running with the selectors `state:modified+` and `source_status:fresher+` builds models that either: * Have changed since the prior run (`state:modified+`) * Have upstream sources that are fresher than in the prior run (`source_status:fresher+`) Instead of relying only on these selectors and prior-run artifacts, state-aware orchestration decides whether to rebuild a model based on: * Compiled SQL diffs that ignore non-meaningful changes like whitespace and comments * Upstream data changes at runtime and model-level freshness settings * Shared state across jobs While dbt v1 uses selectors like `state:modified+` and `source_status:fresher+` to decide what to build *only for a single run in a single job*, state-aware orchestration with dbt v2 maintains a *shared, real-time model state across every job in the environment* and uses that state to determine whether a model’s code or upstream data have actually changed before rebuilding. This ensures dbt only rebuilds models when something has changed, no matter which job runs them. #### Related docs * [State-aware orchestration configuration](https://docs.getdbt.com/docs/deploy/state-aware-setup.md) * [Artifacts](https://docs.getdbt.com/docs/deploy/artifacts.md) * [Continuous integration (CI) jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md) * [`freshness`](https://docs.getdbt.com/reference/resource-configs/freshness.md) * [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) * [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) * [dbt State configs](https://docs.getdbt.com/reference/resource-configs/dbt-state-configs.md) * [Migrate to dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md) --- ### About static analysis dbt v2 [fully comprehends your project's SQL](https://docs.getdbt.com/blog/the-levels-of-sql-comprehension), enabling advanced capabilities like dialect-aware validation and precise column-level lineage. It can do this because its compilation step is more comprehensive than that of the dbt v1.x engine. When dbt v1.x referred to *compilation*, it only meant *rendering* — converting Jinja-templated strings into a SQL query to send to a database. dbt v2 can also render Jinja, but then it completes a second phase: *static analysis*, producing and validating a logical plan for every rendered query in the project. This step is the cornerstone of dbt v2's new capabilities. #### Principles of static analysis The software engineering concept of [static analysis](https://en.wikipedia.org/wiki/Static_program_analysis) describes checks that can be done on code before it runs (static == not running). The most rigorous static analysis means you can trust that if the analysis succeeds, the code will run in production without compilation errors. Less strict static analysis also surfaces helpful information to developers as they work. There's no free lunch—what you gain in responsiveness you lose in correctness guarantees. dbt v2 uses the [`static_analysis`](https://docs.getdbt.com/reference/resource-configs/static-analysis.md) config to help you control how it performs static analysis for your models. dbt v2 is unique in that it can statically analyze not just a single model in isolation, but every query from one end of your DAG to the other. Even your database can only validate the query in front of it! Concepts like [information flow theory](https://roundup.getdbt.com/i/156064124/beyond-cll-information-flow-theory-and-metadata-propagation) — although not incorporated into the dbt platform [yet](https://www.getdbt.com/blog/where-we-re-headed-with-the-dbt-fusion-engine) — rely on stable inputs and the ability to trace columns DAG-wide. ##### Baseline mode: A smooth transition from dbt v1 dbt v2 defaults to `static_analysis: baseline` mode, inspired by similar type-checking and linting tools like [TypeScript's migration approach](https://www.typescriptlang.org/docs/handbook/migrating-from-javascript.html), [basedpyright's baseline feature](https://docs.basedpyright.com/latest/benefits-over-pyright/baseline/), and [Pydantic's strict/lax modes](https://docs.pydantic.dev/latest/why/#strict-lax). The philosophy behind the above-mentioned tools and dbt v2's baseline mode is: * **Smooth transition**: Provide a familiar first-time experience for users coming from dbt v1. * **Incremental opt-in**: Offer a clear pathway to adopt more dbt v2 features over time. * **Pragmatic validation**: Catch most SQL errors without requiring a complete project overhaul. Use this style of gradual typing to start with lightweight validation, then incrementally adopt strict guarantees as your project is ready. ###### What baseline mode changes Baseline mode introduces several fundamental behavior changes compared to the previous binary (off/on) approach: * **No downloading of remote schemas** — Baseline mode does not fetch schemas from the warehouse. * **Unit tests work without strict mode** — Previously, unit tests required static analysis to be fully on. In baseline mode, they work out of the box. * **No unsafe introspection warnings** — We no longer warn about unsafe introspection, though we'd still love to help you assess it in the future. The following table shows how baseline mode expands what's available without requiring strict mode. ##### LSP feature comparison Baseline mode unlocks a meaningful set of features without requiring strict mode. We're also investing in moving more features into baseline over time. VS Code extension features by static analysis configuration: ✅ = Available | ❌ = Not available | Feature | off | baseline | strict | | ---------------------------------------------- | --- | -------- | ------ | | Go-to-definition/reference (except columns) | ✅ | ✅ | ✅ | | Table lineage | ✅ | ✅ | ✅ | | YAML validation | ✅ | ✅ | ✅ | | Render + preview SQL | ✅ | ✅ | ✅ | | Unit tests | ✅ | ✅ | ✅ | | Detect syntax errors | ❌ | ✅ | ✅ | | Preview CTE results | ❌ | ✅ | ✅ | | Go-to-definition/reference (columns) | ❌ | ❌ | ✅ | | Automatic refactor column names | ❌ | ❌ | ✅ | | Rich column lineage | ❌ | ❌ | ✅ | | Detect data type and function signature errors | ❌ | ❌ | ✅ | * dbt VS Code extension features in this table are available to all users for 14 days. * After the 14-day period, sign in or register for a dbt platform account from the dbt VS Code extension to keep using advanced capabilities. * Unregistered users can continue using core editing and build workflows without signing in. * Existing registered dbt VS Code extension users keep access to registration-required features automatically. Supported Snowflake functions To check out which Snowflake functions are supported in dbt v2 in `strict` mode, refer to [Snowflake function support](https://docs.getdbt.com/reference/resource-configs/snowflake-function-support.md) CodeLens visibility The VS Code extension and Studio IDE provide CodeLens even when static analysis is off, giving you visibility into which models have static analysis disabled and why. Ultimately, we want everyone developing in strict mode for maximum guarantees. We acknowledge this isn't a change that can happen overnight — baseline exists to smooth the transition. Many planned features (like local compute) require strict mode. We're also exploring inferring column types on your behalf, which would enable more functionality in baseline mode without requiring you to manually provide type information. ##### Introspection handling in baseline mode In `baseline` mode, all static analysis findings are warnings, not errors — your project can continue running even when the compiler flags invalid or problematic SQL. This section is a good example of why that design exists. Previously, with `strict` mode, the system assumed local schemas of your compiled models would be available. In `baseline` mode, we can no longer assume the full local schema is available and complete, so `baseline` uses the remote database as the source of truth — similar to dbt v1. The practical result is that the dbt v2 compiler may sometimes flag incorrect queries that result from introspective queries that come back empty. If you encounter this, you can: 1. Ignore the warning 2. Build the model locally 3. (Coming soon) Use `warn_error_options` to disable the warning For example, consider this query using the `dbt_utils.unpivot` macro: ```sql select * from ( {{ dbt_utils.unpivot( relation=ref('example_model'), cast_to='integer', exclude=['order_id', 'customer_id'], field_name='product_type', value_name='quantity' ) }} ) ``` If the introspection query fails or returns no results, this renders to: ```sql select * from ( ) ``` This is invalid SQL. In `baseline` mode, dbt v2 displays a warning so your project can continue running while still alerting you to the issue: ```bash dbt0101: no viable alternative at input '( )' --> models/example_model.sql:17:1 ``` ###### Migration scenarios Migrating to dbt v2 can involve more than moving YAML around. Some scenarios that can make migration more involved include: 1. **Limited access to sources**: You don't have access to all the sources and models of a large dbt project. 2. **Intricate Jinja workflows**: Your project uses post-hooks and introspection extensively. 3. **Package compatibility**: Your project depends on packages that aren't yet dbt v2-compatible. 4. **Unsupported SQL features**: Your models or sources use advanced data types (`STRUCT`, `ARRAY`, `GEOGRAPHY`) or built-in functions (`AI.PREDICT`, `JSON_FLATTEN`, `st_pointfromgeohash`) not yet supported by dbt v2. Setting `static_analysis` to `baseline` mode lets you start using dbt v2 immediately while you address these scenarios incrementally. As you resolve compatibility issues, you can opt specific models or your entire project into `strict` mode for maximum validation guarantees. #### Recapping the differences between engines dbt v1: * Renders and runs models one at a time. * Never runs static analysis. dbt v2 (baseline mode — default): * Statically analyzes all models, catching most SQL errors while providing a familiar migration experience. dbt v2 (strict mode): * Renders and statically analyzes all models before execution begins. * Guarantees nothing runs until the entire project is proven valid. * Parses `CREATE FUNCTION` in [`sql_header`](https://docs.getdbt.com/reference/resource-configs/sql_header.md) and in [`on-run-start`](https://docs.getdbt.com/reference/project-configs/on-run-start-on-run-end.md) hooks, then registers those UDFs so strict compilation can resolve calls to them. `baseline` and `off` don't register UDFs this way. See [User-defined functions (UDFs) in `strict` mode](https://docs.getdbt.com/reference/resource-configs/static-analysis.md#user-defined-functions-udfs-in-strict-mode). #### Configuring `static_analysis` You can modify the way static analysis is applied for specific models in your project. The static analysis configuration cascades from most strict to least strict. Going downstream in your lineage, a model can keep the same mode or relax it — it can't be stricter than its parent. Setting `static_analysis: strict` on a model does not automatically set `strict` for downstream models; they keep the project default unless you set them explicitly. For rules and examples, refer to [How modes cascade in your lineage](#how-modes-cascade-in-your-lineage) and [strict mode inheritance](#strict-mode-inheritance). Some models are also downgraded automatically, regardless of what you configure. Refer to [Custom materializations and static analysis](#custom-materializations). The [`static_analysis`](https://docs.getdbt.com/reference/resource-configs/static-analysis.md) config options are: * `baseline` (default): Statically analyze SQL. This is the recommended starting point for users transitioning from dbt v1, providing a smooth migration experience while still catching most SQL errors. * `strict` (previously `on`): Statically analyze all SQL before execution begins. Use this for maximum validation guarantees — nothing runs until the entire project is proven valid. * `off`: Skip SQL analysis on this model and its descendants. (Applies to dbt v2.0 and later) Any run that uses `strict` mode requires authentication using [`dbt login`](https://docs.getdbt.com/reference/commands/login.md?version=2.0), whether `strict` is set with the `--static-analysis strict` CLI flag or in `dbt_project.yml`. Unauthenticated runs fall back to `baseline`. Deprecated values The `on` and `unsafe` values are deprecated and will be removed in May 2026. Use `strict` instead. When you disable static analysis, features of the VS Code extension which depend on SQL comprehension will be unavailable. The best place to configure `static_analysis` is as a config on an individual model or group of models. As a debugging aid, you can also use the [`--static-analysis strict`](https://docs.getdbt.com/reference/global-configs/static-analysis-flag.md) or `--static-analysis off` CLI flags to override all model-level configuration. ##### Incrementally adopting strict mode Once you're comfortable with dbt v2 in baseline mode, you can incrementally opt models or directories into `strict` mode: dbt\_project.yml ```yml name: jaffle_shop models: jaffle_shop: # Start with strict analysis on your cleanest models staging: +static_analysis: strict # Keep baseline for models that need more work marts: +static_analysis: baseline ``` ###### How modes cascade in your lineage Two rules determine which mode a model can use: * **Eligibility rule:** A model is eligible for static analysis only if all of its upstream dependencies are eligible. * **Strictness rule:** A model can't be stricter than its parents. The strictness hierarchy is `strict` → `baseline` → `off`. Going downstream, a model can keep its parent's mode or relax it, but it can't tighten it: | Upstream model | Downstream can be | | -------------- | ---------------------------------- | | `strict` | `strict`, `baseline`, or `off` | | `baseline` | `baseline` or `off` (not `strict`) | | `off` | `off` only | The strictness rule exists because `baseline` doesn't produce the full analyzed schema that `strict` needs from its upstream models. Without that information, a downstream model can't run strict-level type checking. For the complete reference, refer to [How static analysis modes cascade](https://docs.getdbt.com/reference/resource-configs/static-analysis.md#how-static-analysis-modes-cascade). ###### Strict mode inheritance Unlike `baseline` or `off`, `strict` mode doesn't propagate to downstream models. If you configure a model as `strict`, its downstream models won't inherit `strict` mode unless you set them explicitly. To make all models `strict`, you must set `+static_analysis: strict` on root models first, or use the project-wide config in the next section at the project level. For example, in A → B → C with a default of `baseline`, configuring A (a root node) as `strict` makes only A `strict` — B and C remain baseline unless configured. To make the full chain `strict`, set `+static_analysis: strict` on each relevant model or group, or use a project-wide setting. This approach lets you gain the benefits of strict validation where possible while keeping the flexibility of baseline analysis for models that aren't yet compatible. Refer to [CLI options](https://docs.getdbt.com/reference/global-configs/command-line-options.md) and [Configurations and properties](https://docs.getdbt.com/reference/configs-and-properties.md) to learn more about configs. ##### Custom materializations If a model uses a [custom materialization](https://docs.getdbt.com/guides/create-new-materializations.md), v2 turns static analysis `off` for that model and for every model downstream of it. It does this automatically, without an error or a warning, no matter what you set `static_analysis` to. Because custom materialization is code you wrote, and it can change the finished table in ways v2 can't predict (for example, adding, renaming, or retyping columns). Rather than check your SQL against a schema that might be wrong, dbt skips analysis. It's the same reason dbt skips [introspective queries](#introspection-handling-in-baseline-mode), whose results also aren't known until the model runs. Two kinds of custom materializations trigger the downgrade to `off`: * **A name you invented:** such as `materialized='my_custom_load'`. Find these in your model configs. * **A built-in name:** such as your own macro named `materialization table, default`. These are harder to spot, because models that say `materialized='table'` look standard but run your code instead of dbt's. What this means in practice: * Models using a custom materialization don't fail because of static analysis. * Setting `static_analysis: strict` (or `baseline`) on those models has no effect as the downgrade to `off` overrides other settings. * Because `off` cascades downstream, every model downstream of a model using a custom materialization is also ineligible for static analysis. In a project where most models use a custom materialization, this can disable static analysis, and the [features that depend on it](#lsp-feature-comparison), for the majority of the DAG. If your project depends heavily on custom materializations and you want static analysis coverage, these are your options today: * Convert those models to built-in materializations where practical * Isolate custom materializations so fewer downstream models are affected note We're reevaluating this automatic downgrade. The intent is for `baseline` analysis to keep working for models with custom materializations, and for you to account for schema-modifying materializations yourself when using `strict`. This page will be updated when that behavior changes. ##### Identify a model's mode The mode you configure for a model isn't always the mode in effect. This is because a model's effective mode depends on its parents, and on [custom materializations](#custom-materializations). You can see when a model has static analysis off in the [dbt VS Code extension](https://docs.getdbt.com/docs/about-dbt-extension.md) and the Studio IDE both of which show a CodeLens above your models, indicating which models have static analysis disabled and why. Keep in mind that `dbt ls --output json --output-keys config.static_analysis` reports the mode you *configured* for each model, not the mode v2 resolves after applying the cascading rules and automatic downgrades. ##### Example configurations ###### Configure strict for the entire project Many teams want to enable `strict` mode for the whole project and all packages. You can do this by setting `+static_analysis: strict` under each resource type in `dbt_project.yml` for your project name (and for any package names if you want those to be strict too): dbt\_project.yml ```yaml models: my_project: +static_analysis: strict seeds: my_project: +static_analysis: strict snapshots: my_project: +static_analysis: strict tests: my_project: +static_analysis: strict unit_tests: my_project: +static_analysis: strict sources: my_project: +static_analysis: strict analyses: my_project: +static_analysis: strict ``` Use your project name in place of `my_project` — that's the same value as the `name:` key at the top of `dbt_project.yml` (for example, `jaffle_shop`). To apply `strict` to a package as well, add another entry under each resource type using the package name as the key; for example, under `models:` add `your_package_name:` with `+static_analysis: strict` beneath it. ###### Disable static analysis for all models in a package: dbt\_project.yml ```yml name: jaffle_shop models: jaffle_shop: marts: +materialized: table a_package_with_introspective_queries: +static_analysis: off ``` ###### Disable static analysis in YAML: models/my\_udf\_using\_model.yml ```yml models: - name: model_with_static_analysis_off config: static_analysis: off ``` ###### Disable static analysis for a model using a custom UDF: models/my\_udf\_using\_model.sql ```sql {{ config(static_analysis='off') }} select user_id, my_cool_udf(ip_address) as cleaned_ip from {{ ref('my_model') }} ``` ##### Can I use strict mode in development and baseline in deployment? Yes. This pattern is valid and recommended: use `strict` while you develop for stronger validation, and keep `baseline` in deployment for faster runs that are less likely to stop on analysis findings. For more information, including CLI examples and an optional environment variable pattern, refer to [Optimize static analysis for development and deployment](https://docs.getdbt.com/best-practices/optimize-static-analysis-for-development-and-deployment.md). ##### When should I turn static analysis `off`? With baseline mode enabled by default, static analysis is less likely to block your runs. You should only disable it if dbt v2 cannot parse SQL that is valid for your database of choice. This is a very rare occurrence. If you encounter this situation, please [open an issue](https://github.com/dbt-labs/dbt-fusion/issues) with an example of the failing SQL so we can update our parsers. #### More information about dbt v2 * [About the dbt extension](https://docs.getdbt.com/docs/about-dbt-extension.md) * [Supported features matrix](https://docs.getdbt.com/docs/dbt/supported-features.md) * [Install dbt](https://docs.getdbt.com/docs/local/install-dbt.md) * [Quickstart for dbt v2](https://docs.getdbt.com/guides/dbt.md?step=1) * [Upgrade guide](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md) * [dbt v2 license agreement](https://www.getdbt.com/dbt-fusion-engine-license-agreement) --- ### About the --empty flag ### About the `--empty` flag note The `--empty` flag is not currently available for Python models. If the flag is used with a Python model, it will be ignored. During dbt development, you might want to validate that your models are semantically correct without the time-consuming cost of building the entire model in the data warehouse. The [`run`](https://docs.getdbt.com/reference/commands/run.md), [`build`](https://docs.getdbt.com/reference/commands/build.md), [`snapshot`](https://docs.getdbt.com/reference/commands/snapshot.md), and [`compile`](https://docs.getdbt.com/reference/commands/compile.md) commands support the `--empty` flag. Starting in dbt v1.12, [`seed`](https://docs.getdbt.com/reference/commands/seed.md) also supports the `--empty` flag for building schema-only dry runs. The `--empty` flag limits the refs and sources to zero rows. dbt will still execute the model SQL against the target data warehouse but will avoid expensive reads of input data. This validates dependencies and ensures your models will build properly. ##### Examples Run all models in a project while building only the schemas in your development environment: ```text dbt run --empty ``` Run a specific model: ```text dbt run --select path/to/your_model --empty ``` dbt will build and execute the SQL, resulting in an empty schema in the data warehouse. --- ### About the --sample flag ### About the `--sample` flag note The `--sample` flag is not currently available for Python models. If the flag is used with a Python model, it will be ignored. Seeds will be created normally, but are sampled when referenced by downstream nodes. Large data sets can drastically increase build times and reduce how quickly dbt developers can build and test new code. The dbt `--sample` flag can help to reduce build times and warehouse spend by running dbt in sample mode. Sample mode enables you to address cases where you don't need to build the entire model during the development or CI cycle but include enough data to validate the outputs. Sample mode takes the [`--empty` flag's](https://docs.getdbt.com/docs/build/empty-flag.md) validation of semantic results a step further by including a sampling of data from the model(s) in your development schema. It won't solve every scenario; for example, there are cases where not all joins will be populated. However, it presents a viable solution for faster building, testing, and validating many strategies. The `--sample` flag will become more robust over time, but it only supports time-based sampling for now. #### Using the `--sample` flag The `--sample` flag is available for the [`run`](https://docs.getdbt.com/reference/commands/run.md) and [`build`](https://docs.getdbt.com/reference/commands/build.md) commands. When used, sample mode generates filtered refs and sources. Since it's using time-based sampling, if you have refs like `{{ ref('some_model') }}` being sampled, you need to set [`event_time`](https://docs.getdbt.com/reference/resource-configs/event-time.md) for `some_model` to the field that will be used as the timestamp. There are two time-based sample specifications supported for sample mode: * **Relative time specs:** Filters sampled data from the time the command is run back to a specified integer and granularity. Supported granularities are: * Hours * Days * Months * Years * **Static time specs:** Filters your data between a defined start and end period using date and/or timestamp. ##### Examples Let's say you want to run your `stg_customers` model and build the table in your development schema with a relative time spec sample size of three days. Your command in the IDE would look something like this: ```text dbt run --select path/to/stg_customers --sample="3 days" ``` If you have an even larger model, for example, `stg_orders` you can set sample mode to hours: ```text dbt run --select path/to/stg_customers --sample="6 hours" ``` Next, let's say you want to validate data for your entire business from a sample size further in the past - your busiest week in July, from the first until closing time on the eighth. You can run the following: ```text dbt run --sample="{'start': '2024-07-01', 'end': '2024-07-08 18:00:00'}" ``` To prevent a `ref` from being sampled, append `.render()` to it: ```sql with source as ( select * from {{ ref('stg_customers').render() }} ), ... ``` dbt will then execute the model SQL against the target data warehouse and build the tables with data from the sample sizes. --- ### About the dbt VS Code extension Preview ### About the dbt VS Code extension [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Local development The dbt VS Code extension brings a hyper-fast, intelligent, and cost-efficient dbt development experience to VS Code. This is the only way to enjoy all the power of dbt v2 while developing with a self-hosted installation. * *Save time and resources* with near-instant parsing, live error detection, powerful IntelliSense capabilities, and more. * *Stay in flow* with a seamless, end-to-end dbt development experience designed from scratch for local dbt development. The dbt VS Code extension is available in the [VS Code Marketplace](https://marketplace.visualstudio.com/items?itemName=dbtLabsInc.dbt). *Note, this is a public preview release. Behavior may change ahead of the broader generally available (GA) release.* The dbt VS Code extension works with v2, the default free-to-use product you get when you install dbt. #### Version compatibility dbt v1 and v2 both support the dbt VS Code extension. You get editor features out of the box as long as your project complies with the v2 language spec: * Autocomplete for `ref`, sources, macros, and doc blocks * Go to definition and find references for models, sources, macros, and docs * Jinja syntax error detection * Model renames that update `ref` calls, and source or table reference renames * View compiled SQL, and preview a model, selected SQL, or an individual CTE * Run `dbt run`, `build`, `test`, and selectors * Table descriptions, metadata, and metadata on hover * Production column types, descriptions, and tests * Visual and textual table lineage in the extension pane If your project isn't compliant with the v2 language spec yet, the extension has [dbt-autofix](https://github.com/dbt-labs/dbt-autofix) built in. Opt in to fix deprecations across your project in a single pull request. Each issue in the **Problems** pane also has an autofix action, available from the lightbulb or right-click menu. To unlock the full language server, including column-level lineage, column go-to definition and find references, nonexistent-column and type detection, warehouse-specific syntax checks, column renames that propagate downstream, and `SELECT *` expansion, enable [`static_analysis: strict`](https://docs.getdbt.com/reference/resource-configs/static-analysis.md?version=2) and resolve your parse errors. Refer to [About dbt LSP](https://docs.getdbt.com/docs/about-dbt-lsp.md) for the full feature list. Try out the dbt v2 quickstart guide Check out the [v2 quickstart guide](https://docs.getdbt.com/guides/dbt.md?step=1) to try the dbt VS Code extension in action. #### Navigating the dbt extension Once the dbt VS Code extension has been installed, several visual enhancements will be added to your IDE to help you navigate the features and functionality. To read more about the features and functionality, see the [dbt extension features](https://docs.getdbt.com/docs/dbt-extension-features.md). Check out the following video to see the features and functionality of the dbt VS Code extension: [dbt v2 + VS Code extension walkthrough](https://app.storylane.io/share/a1rkqx0mbd7a) ##### Sign in and feature access After you install the dbt VS Code extension, all [features](https://docs.getdbt.com/docs/dbt-extension-features.md) are available for 14 days with no registration. After that, most features continue to work without registration. To keep using advanced features, [register](https://docs.getdbt.com/docs/sign-in-dbt-extension.md) for a free dbt platform account. Previously only registered users had access to the dbt VS Code extension. Without registration, the vast majority of features continue to work. Only advanced features prompt you to register or sign in to an existing account after the trial period ends. Strict static analysis requires registration If your project uses `static_analysis: strict` in `dbt_project.yml` or you pass `--static-analysis strict` at runtime, registration for a dbt platform account is required regardless of the trial status. Refer to [VS Code extension features](https://docs.getdbt.com/docs/dbt-extension-features.md#feature-availability) for the full list of features and their availability. When you register or sign in for advanced features, authentication is handled by [`dbt login`](https://docs.getdbt.com/reference/commands/login.md?version=2.0). Your login state is shared across the CLI, dbt VS Code extension, and dbt State (if you log in using dbt platform). You can use the [get started wizard](https://docs.getdbt.com/docs/install-dbt-extension.md#getting-started) or run `dbt login` from your terminal, then restart or reload VS Code. The extension detects your login automatically. ##### The dbt extension menu The dbt logo on the sidebar (or the **dbt Extension** text on the bottom tray) launches the main menu for the extension. This menu contains helpful information and actions you can take: * **Get started button:** Launches the [v2 upgrade](https://docs.getdbt.com/docs/upgrade-to-dbt-extension.md) workflow. * **Extension info:** Information about the extension, v2, and your dbt project. Includes configuration options and actions. * **Help:** Quick links to support, bug submissions, and documentation. [![dbt VS Code extension welcome screen.](/img/docs/extension/sidebar-menu.png?v=2 "dbt VS Code extension welcome screen.")](#)dbt VS Code extension welcome screen. ##### Caching The dbt extension caches important schema information from your data warehouse to improve speed and performance. This will automatically update over time, but if recent changes have been made that aren't reflected in your project, you can manually update the schema information: 1. Click the **dbt logo** on the sidebar to open the menu. 2. Expand the **Extension info** section and location the **Actions** subsection. 3. Click **Clear Cache** to update. #### Using the extension Once installed, the dbt extension automatically activates when you open any `.sql` or `.yml` file inside of a dbt project directory. #### Configuration After installation, you may want to configure the extension to better fit your development workflow: 1. Open the VS Code settings by pressing `Ctrl+,` (Windows/Linux) or `Cmd+,` (Mac). 2. Search for `dbt`. On this page, you can adjust the extension’s configuration options to fit your needs. [![dbt extension settings within the VS Code settings.](/img/docs/extension/dbt-extension-settings.png?v=2 "dbt extension settings within the VS Code settings.")](#)dbt extension settings within the VS Code settings. #### Known limitations The following are currently known limitations of the dbt extension: * **Remote development:** The dbt extension does not yet support remote development sessions over SSH. Support will be added in a future release. For more information on remote development, refer to [Supporting Remote Development and GitHub Codespaces](https://code.visualstudio.com/api/advanced-topics/remote-extensions) and [Visual Studio Code Server](https://code.visualstudio.com/docs/remote/vscode-server). * **Working with YAML files:** Today, the dbt extension has the following limitations with operating on YAML files: * Go-to-definition is not supported for nodes defined in YAML files (like snapshots). * Renaming models and columns will not update references in YAML files. * Future releases of the dbt extension will address these limitations. * **Renaming models:** When you rename a model file, the dbt extension applies edits to update all `ref()` calls that reference the renamed model. Due to limitations of VS Code's Language Server Client, the extension can't auto-save these edited files. As a result, renaming a model file may cause compiler errors in your project. To fix these errors, either manually save each file that the dbt extension edited, or click **File** --> **Save All** to save all edited files. * **Using Cursor's Agent mode:** When using the dbt extension in Cursor, lineage visualization works best in Editor mode and doesn't render in Agent mode. If you're working in Agent mode and need to view lineage, switch to Editor mode to access the full lineage tab functionality. ##### Extension conflicts The extension may occasionally conflict with other VS Code extensions that provide similar services (such as code validation). You may need to disable these third-party extensions while working with the dbt extension. **YAML by Red Hat:** The YAML extension by Red Hat may erroneously flag some keys (such as `static_analysis`) in dbt YAML files as invalid in the IDE. [![Static analysis erroneously tagged as invalid](/img/docs/extension/false-yaml-error.png?v=2 "Static analysis erroneously tagged as invalid")](#)Static analysis erroneously tagged as invalid To solve this issue, do one of the following: * (Recommended) Disable the Red Hat YAML extension while working with the dbt extension. * Add the following configuration to your VS Code `settings.json` file: ```json "yaml.schemas": { "Core/dbtschema.json": "data/dbt/models/**/schema.yml", "": "data/dbt/dbt_project.yml" }, ``` This could disable *all* use of the schema store, resulting in unintended consequences. #### dbt Wizard The dbt VS Code extension and [dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) are designed to work together. The extension gives you fast parsing, inline error detection, and IntelliSense. [dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) adds an AI layer on top — one that works with a live understanding of your project through dbt's [native metadata engine](https://docs.getdbt.com/docs/dbt-ai/about-dbt-ai.md), a structured index of your [lineage](https://docs.getdbt.com/docs/explore/explore-projects.md), model health, test coverage, and semantic definitions. * **Build and refactor from natural language**: Describe what you want, review the diff, approve or redirect before anything is saved. * **Validate changes before they land**: The agent compiles and runs against your warehouse in a tight loop — not just editor linting. * **Work with your full project map**: Traverse the [DAG](https://docs.getdbt.com/docs/explore/explore-projects.md), understand downstream impact, and keep tests and YAML in sync as models change. For data practitioners, combining the extension's dbt-powered editor experience with dbt Wizard's project-aware agent means less manual YAML, fewer context switches, and faster iteration on complex modeling work. See [dbt Wizard quickstart](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) to get started. #### Support dbt platform customers can contact dbt Labs support at <support@getdbt.com>. You can also get in touch with us by reaching out to your Account Manager directly. For organizations that are not customers of the dbt platform, the best place for questions and discussion is the [dbt Community Slack](https://www.getdbt.com/community/join-the-community). We welcome feedback as we work to continuously improve the extension, and would love to hear from you! For more information regarding support and acceptable use of the dbt VS Code extension, refer to our [Acceptable Use Policy](https://www.getdbt.com/dbt-assets/vscode-plugin-aup). Developing locally as a dbt platform user? Refer to the [Hybrid development with dbt platform and dbt v2](https://docs.getdbt.com/guides/dbt-platform-local-workflow.md) guide for how to keep credentials, environment variables, and v2 versions in sync between your local extension and dbt platform. #### More information about dbt v2 * [About the dbt extension](https://docs.getdbt.com/docs/about-dbt-extension.md) * [Supported features matrix](https://docs.getdbt.com/docs/dbt/supported-features.md) * [Install dbt](https://docs.getdbt.com/docs/local/install-dbt.md) * [Quickstart for dbt v2](https://docs.getdbt.com/guides/dbt.md?step=1) * [Upgrade guide](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md) * [dbt v2 license agreement](https://www.getdbt.com/dbt-fusion-engine-license-agreement) --- ### Access Catalog from dbt platform features dbt platform Access Catalog from jobs, Insights, and other areas of dbt to move quickly between run context, lineage, and resource metadata in your project. This page describes how to open Catalog from orchestration and exploration workflows in dbt. The primary way to open Catalog is **Catalog** in the navigation; you can also open it from jobs and Insights as described in the sections below. ##### Lineage tab in jobs The **Lineage tab** in dbt jobs displays the lineage associated with the [job run](https://docs.getdbt.com/docs/deploy/jobs.md). You can open Catalog directly from this tab to understand the dependencies and relationships of resources in your project. ###### Access Catalog from the lineage tab * From a job, select the **Lineage tab**. * Double-click a node in the lineage graph to open a new tab and view its metadata in Catalog. [![Access dbt Catalog from the lineage tab by double-clicking on the lineage node.](/img/docs/collaborate/dbt-explorer/explorer-from-lineage.gif?v=2 "Access dbt Catalog from the lineage tab by double-clicking on the lineage node.")](#)Access dbt Catalog from the lineage tab by double-clicking on the lineage node. ##### Model timing tab in jobs [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") The [model timing tab](https://docs.getdbt.com/docs/deploy/run-visibility.md#model-timing) in dbt jobs displays the composition, order, and time taken by each model in a job run. You can open Catalog from the **model timing tab** to investigate resources, diagnose performance bottlenecks, understand dependencies and relationships of slow-running models, and make changes to improve their performance. ###### Access Catalog from the model timing tab * From a job, select the **model timing tab**. * Hover over a resource and select **View in Catalog** to open its metadata in Catalog. [![Access dbt Catalog from the model timing tab by hovering over the resource and clicking 'View in Explorer'.](/img/docs/collaborate/dbt-explorer/explorer-from-model-timing.png?v=2 "Access dbt Catalog from the model timing tab by hovering over the resource and clicking 'View in Explorer'.")](#)Access dbt Catalog from the model timing tab by hovering over the resource and clicking 'View in Explorer'. ##### dbt Insights [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") Open Catalog from [Insights](https://docs.getdbt.com/docs/explore/access-dbt-insights.md) to view project lineage and resources such as tables, columns, metrics, dimensions, and more. To open Catalog from Insights, select the **Catalog** icon in the Query console sidebar menu, then search for the resource you want. [![dbt Insights integrated with dbt Catalog](/img/docs/dbt-insights/insights-explorer.png?v=2 "dbt Insights integrated with dbt Catalog")](#)dbt Insights integrated with dbt Catalog --- ### Access the dbt Insights interface dbt platform | Enterprise, Enterprise+ Learn how to access Insights, run queries, and view results. Insights provides a rich console experience with editor navigation. You can expect Insights to: * Enable you to write SQL queries, with the option to open multiple tabs * Have SQL + dbt autocomplete suggestions and syntax highlighting * Save SQL queries * View the results of the query and its details using the **Data** or **Details** tabs * Create a visualization of your query results using the **Chart** tab * View the history of queries and their statuses (like Success, Error, Pending) using the **Query history** tab * Use dbt Copilot in Insights to generate or edit SQL queries using natural language prompts * Integrate with [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md), [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md), and [Canvas](https://docs.getdbt.com/docs/platform/canvas.md) to provide a seamless experience for data exploration, AI-assisted writing, and collaboration #### Access the dbt Insights interface Before accessing Insights, ensure that the [prerequisites](https://docs.getdbt.com/docs/explore/dbt-insights.md#prerequisites) are met. 1. To access Insights, select the **Insights** option in the navigation sidebar. 2. If your [user credentials](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md#get-started-with-the-studio-ide) aren’t set up, Insights will prompt you to set them up. The ability to query data is subject to warehouse provider permissions according to your user credentials. 3. Once your credentials are set up, you can write, run, and edit SQL queries in the Insights editor for existing models in your project. #### Run queries To run queries in Insights, you can use: * Standard SQL * Jinja ([`ref`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md), [`source`](https://docs.getdbt.com/reference/dbt-jinja-functions/source.md) functions, and other Jinja functions) * Links from SQL code `ref` to the corresponding Explorer page * CTEs and subqueries * Basic aggregations and joins * Semantic Layer queries using Semantic Layer Jinja functions #### Example Let's use an example to illustrate how to run queries in Insights: * A [Jaffle Shop](https://github.com/dbt-labs/jaffle-shop) location wants to count unique orders and unique customers to understand whether they can expand their awesome Jaffle shop business to other parts of the world. * To express this logic in SQL, you (an analyst assigned to this project) want to understand yearly trends to help guide expansion decisions. Write the following SQL query to calculate the number of unique customers, cities, and total order revenue: <br /> <br /> ```sql with orders as ( select * from {{ ref('orders') }} ), customers as ( select * from {{ ref('customers') }} ) select date_trunc('year', ordered_at) as order_year, count(distinct orders.customer_id) as unique_customers, count(distinct orders.location_id) as unique_cities, to_char(sum(orders.order_total), '999,999,999.00') as total_order_revenue from orders join customers on orders.customer_id = customers.customer_id group by 1 order by 1 ``` ##### Use dbt Copilot To make things easier, [use dbt Copilot in Insights](https://docs.getdbt.com/docs/explore/navigate-dbt-insights.md) to save time and explore other ways to analyze the data. dbt Copilot can help you quickly update the query or generate a new one based on your prompt. dbt Wizard is the recommended agent for dbt work dbt Wizard is the recommended AI agent for governed data development in dbt. It handles the full development lifecycle — investigation, building, validation, and shipping — grounded in your dbt project's lineage, tests, contracts, and metric definitions. Refer to [dbt AI FAQs](https://docs.getdbt.com/docs/dbt-ai/dbt-ai-faqs.md#is-dbt-wizard-the-same-as-dbt-copilot), [Billing](https://docs.getdbt.com/docs/dbt-ai/wizard-billing-faqs.md), and [dbt's Terms of Use](https://www.getdbt.com/terms-of-use) for more information. 1. Click the **dbt Copilot** icon in the Query console sidebar. 2. In the dropdown menu above the dbt Copilot prompt box, select **Generate SQL**. 3. Enter your prompt in natural language and ask for a yearly breakdown of unique customers and total revenue. 4. Click **↑** to submit your prompt. 5. dbt Copilot responds with: * A summary of the query * An explanation of the logic * The SQL it generated * Options to **Add** or **Replace** the existing query with the generated SQL 6. Review the output and click **Replace** to use the dbt Copilot-generated SQL in your editor. 7. Click **Run** to preview the results. [![dbt Insights with dbt Copilot](/img/docs/dbt-insights/insights-copilot.png?v=2 "dbt Insights with dbt Copilot")](#)dbt Insights with dbt Copilot From here, you can: * Continue building or modifying the query using dbt Copilot. * Explore the [results](#view-results) in the **Data** tab. * [View metadata and query details](#view-details) in the **Details** tab. * [Visualize results](#chart-results) in the **Chart** tab. * Check the [**Query history**](#query-history) for status and past runs. * Use [**Catalog**](#use-dbt-explorer) to explore model lineage and context. * If you want to save the query, you can click **Save Insight** in the [query console menu](https://docs.getdbt.com/docs/explore/navigate-dbt-insights.md#query-console-menu) to save it for future reference. Want to turn a query into a model? You can access the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) or [Canvas](https://docs.getdbt.com/docs/platform/canvas.md) from the [Query console menu](https://docs.getdbt.com/docs/explore/navigate-dbt-insights.md#query-console-menu) to promote your SQL into a reusable dbt model — all within dbt! ##### View results Using the same example, you can perform some exploratory data analysis by running the query and: * Viewing results in **Data** tab — View the paginated results of the query. * Sorting results — Click on the column header to sort the results by that column. * Exporting to CSV — On the top right of the table, click the download button to export the dataset. [![dbt Insights Export to CSV](/img/docs/dbt-insights/insights-export-csv.png?v=2 "dbt Insights Export to CSV")](#)dbt Insights Export to CSV ##### View details View the details of the query by clicking on the **Details** tab: * **Query metadata** — dbt Copilot-generated title and description, the supplied SQL, and corresponding compiled SQL. * **Connection details** — Relevant data platform connection information. * **Query details** — Query duration, status, column count, row count. [![dbt Insights Details tab](/img/docs/dbt-insights/insights-details.png?v=2 "dbt Insights Details tab")](#)dbt Insights Details tab ##### Chart results Visualize the chart results of the query by clicking on the **Chart** tab to: * Select the chart type using the chart icon. * Choose from **line chart, bar chart, or scatterplot**. * Select the axis and columns to visualize using the **Chart settings** icon. [![dbt Insights Chart tab](/img/docs/dbt-insights/insights-chart.png?v=2 "dbt Insights Chart tab")](#)dbt Insights Chart tab ##### Query history View the history of queries and their statuses (All, Success, Error, or Pending) using the **Query history** icon: * Select a query to re-run to view the results. * Search for past queries and filter by status. * Hover over the query to view the SQL code or copy it. The query history is stored indefinitely. [![dbt Insights Query history icon](/img/docs/dbt-insights/insights-query-history.png?v=2 "dbt Insights Query history icon")](#)dbt Insights Query history icon ##### Use dbt Catalog Access [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) directly in Insights to view project resources such as models, columns, metrics, and dimensions, and more — all integrated in the Insights interface. This integrated view allows you and your users to maintain your query workflow, while getting more context on models, semantic models, metrics, macros, and more. The integrated Catalog view comes with: * Same search capabilities as Catalog * Allows users to narrow down displayed objects by type * Hyperlink from SQL code `ref` to the corresponding Catalog page * View assets in more detail by opening with the full Catalog experience or open them in dbt Copilot. To access Catalog, click on the **Catalog** icon in the [Query console sidebar menu](https://docs.getdbt.com/docs/explore/navigate-dbt-insights.md#query-console-sidebar-menu). [![dbt Insights integrated with dbt Catalog](/img/docs/dbt-insights/insights-explorer.png?v=2 "dbt Insights integrated with dbt Catalog")](#)dbt Insights integrated with dbt Catalog ##### Set Jinja environment Set the compilation environment to control how Jinja functions are rendered. This feature: * Supports "typed" environments marked as `Production`, `Staging`, and/or `Development`. * Enables you to run Semantic Layer. queries against staging environments (development environments not supported). * Still uses the individual user credentials, so users must have appropriate access to query `PROD` and `STG`. * Changing the environment changes context for the Catalog view in Insights, as well as the environment context during the handoff to Catalog and Canvas. For example, switching to `Staging` in Insights and selecting **View in Catalog** will open the `Staging` view in Catalog. [![Set the environment for your Jinja context](/img/docs/dbt-insights/insights-jinja-environment.png?v=2 "Set the environment for your Jinja context")](#)Set the environment for your Jinja context #### Save your Insights Insights offers a robust save feature for quickly finding the queries you use most. There's also an option to share saved Insights with other dbt users (and have them share with you). Click the **bookmark icon** in a query to add it to your list! * Click the **bookmark icon** on the right menu to manage your saved Insights. You can view your personal and shared queries [![Manage your saved Insights](/img/docs/dbt-insights/saved-insights.png?v=2 "Manage your saved Insights")](#)Manage your saved Insights * View saved Insight details including description and creation date in the **Overview** tab. * View the Insight history in the **Version history** tab. Click a version to compare it the current and view changes. #### Considerations * Insights uses your user credentials to query. You have the ability to query against any object in your data warehouse that is accessible using your user credentials configured in **Account settings**. * Every Jinja function uses [`defer --favor-state`](https://docs.getdbt.com/reference/node-selection/defer.md) to resolve Jinja. #### FAQs * What’s the difference between Insights and Catalog? * That’s a great question! Catalog helps you understand your dbt project's structure, resources, lineage, and metrics, offering context for your data. * Insights builds on that context, allowing you to write, run, and iterate on SQL queries directly in dbt. It’s designed for ad-hoc or exploratory analysis and empowers business users and analysts to explore data, ask questions, and collaborate seamlessly. * Catalog provides the context, while Insights enables action. --- ### Add data tests to your DAG Tip Use [dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-overview.md) to generate data tests in the dbt platform or locally in the CLI. #### Related reference docs * [Test command](https://docs.getdbt.com/reference/commands/test.md) * [Data test properties](https://docs.getdbt.com/reference/resource-properties/data-tests.md) * [Data test configurations](https://docs.getdbt.com/reference/data-test-configs.md) * [Test selection examples](https://docs.getdbt.com/reference/node-selection/test-selection-examples.md) important Tests are now called data tests to disambiguate from [unit tests](https://docs.getdbt.com/docs/build/unit-tests.md). The YAML key `tests:` is still supported as an alias for `data_tests:`. Refer to [New `data_tests:` syntax](#new-data_tests-syntax) for more information. #### Overview Data tests are assertions you make about your models and other resources in your dbt project (for example, sources, seeds, and snapshots). When you run `dbt test`, dbt will tell you if each test in your project passes or fails. You can use data tests to improve the integrity of the SQL in each model by making assertions about the results generated. Out of the box, you can test whether a specified column in a model only contains non-null values, unique values, or values that have a corresponding value in another model (for example, a `customer_id` for an `order` corresponds to an `id` in the `customers` model), and values from a specified list. You can extend data tests to suit business logic specific to your organization – any assertion that you can make about your model in the form of a select query can be turned into a data test. Data tests return a set of failing records. Generic data tests (also known as schema tests) are defined using `test` blocks. Like almost everything in dbt, data tests are SQL queries. In particular, they are `select` statements that seek to grab "failing" records, ones that disprove your assertion. If you assert that a column is unique in a model, the test query selects for duplicates; if you assert that a column is never null, the test seeks nulls. If the data test returns zero failing rows, it passes, and your assertion has been validated. There are two ways of defining data tests in dbt: * A **singular** data test, in its simplest form, is when you write a SQL query that returns failing rows, you can save that query in a `.sql` file within your [test directory](https://docs.getdbt.com/reference/project-configs/test-paths.md). It's now a data test, and it will be executed by the `dbt test` command. * A **generic** data test is a parameterized query that accepts arguments. The test query is defined in a special `test` block (like a [macro](https://docs.getdbt.com/docs/build/jinja-macros.md)). Once defined, you can reference the generic test by name throughout your `.yml` files—define it on models, columns, sources, snapshots, and seeds. dbt ships with four generic data tests built in, and we think you should use them! Defining data tests is a great way to confirm that your outputs and inputs are as expected, and helps prevent regressions when your code changes. Because you can use them over and over again, making similar assertions with minor variations, generic data tests tend to be much more common—they should make up the bulk of your dbt data testing suite. That said, both ways of defining data tests have their time and place. Creating your first data tests If you're new to dbt, we recommend that you check out our [online dbt Fundamentals course](https://learn.getdbt.com/learn/course/dbt-fundamentals/data-tests-30min/building-tests?page=1) or [quickstart guide](https://docs.getdbt.com/guides.md) to build your first dbt project with models and tests. #### Singular data tests The simplest way to define a data test is by writing the exact SQL that will return failing records. We call these "singular" data tests, because they're one-off assertions usable for a single purpose. These tests are defined in `.sql` files, typically in your `tests` directory (as defined by your `test-paths` config). **Note:** The `tests/` directory (`test-paths`) is reserved for singular and generic data tests (SQL). Unit test YAML definitions must live under your project’s `model-paths` (for example, in the `models/` directory), not in `tests/`. You can use Jinja (including `ref` and `source`) in the test definition, just like you can when creating models. Each `.sql` file contains one `select` statement, and it defines one data test: tests/assert\_total\_payment\_amount\_is\_positive.sql ```sql -- Refunds have a negative amount, so the total amount should always be >= 0. -- Therefore return records where total_amount < 0 to make the test fail. select order_id, sum(amount) as total_amount from {{ ref('fct_payments') }} group by 1 having total_amount < 0 ``` The test name is the file name: `assert_total_payment_amount_is_positive`. Note: * Omit semicolons (;) at the end of the SQL statement in your singular test files, as they can cause your data test to fail. * Singular data tests placed in the tests directory are automatically executed when running `dbt test`. Don't reference singular tests in `model_name.yml`, as they are not treated as generic tests or macros, and doing so will result in an error. To add a description to a singular data test in your project, add a `.yml` file to your `tests` directory, for example, `tests/schema.yml` with the following content: tests/schema.yml ```yaml data_tests: - name: assert_total_payment_amount_is_positive description: > Refunds have a negative amount, so the total amount should always be >= 0. Therefore return records where total amount < 0 to make the test fail. ``` Singular data tests are so easy that you may find yourself writing the same basic structure repeatedly, only changing the name of a column or model. By that point, the test isn't so singular! In that case, we recommend generic data tests. #### Generic data tests Certain data tests are generic: they can be reused over and over again. A generic data test is defined in a `test` block, which contains a parameterized query and accepts arguments. It might look like: ```sql {% test not_null(model, column_name) %} select * from {{ model }} where {{ column_name }} is null {% endtest %} ``` You'll notice that there are two arguments, `model` and `column_name`, which are then templated into the query. This is what makes the data test "generic": it can be defined on as many columns as you like, across as many models as you like, and dbt will pass the values of `model` and `column_name` accordingly. Once that generic test has been defined, it can be added as a *property* on any existing model (or source, seed, or snapshot). These properties are added in `.yml` files in the same directory as your resource. info If this is your first time working with adding properties to a resource, check out the docs on [declaring properties](https://docs.getdbt.com/reference/configs-and-properties.md). Out of the box, dbt ships with four generic data tests already defined: `unique`, `not_null`, `accepted_values`, and `relationships`. Here's a full example using those tests on an `orders` model: ```yaml models: - name: orders columns: - name: order_id data_tests: - unique - not_null - name: status data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['placed', 'shipped', 'completed', 'returned'] - name: customer_id data_tests: - relationships: arguments: to: ref('customers') field: id ``` In plain English, these data tests translate to: * `unique`: the `order_id` column in the `orders` model should be unique * `not_null`: the `order_id` column in the `orders` model should not contain null values * `accepted_values`: the `status` column in the `orders` model should be one of `'placed'`, `'shipped'`, `'completed'`, or `'returned'` * `relationships`: each `customer_id` in the `orders` model exists as an `id` in the `customers` table (also known as referential integrity) Behind the scenes, dbt constructs a `select` query for each data test, using the parameterized query from the generic test block. These queries return the rows where your assertion is *not* true; if the test returns zero rows, your assertion passes. You can find more information about these data tests, and additional configurations (including [`severity`](https://docs.getdbt.com/reference/resource-configs/severity.md) and [`tags`](https://docs.getdbt.com/reference/resource-configs/tags.md)) in the [reference section](https://docs.getdbt.com/reference/resource-properties/data-tests.md). You can also add descriptions to the Jinja macro that provides the core logic of a generic data test. Refer to the [Add description to generic data test logic](https://docs.getdbt.com/best-practices/writing-custom-generic-tests.md#add-description-to-generic-data-test-logic) for more information. ##### More generic data tests Those four tests are enough to get you started. You'll quickly find you want to use a wider variety of data tests — a good thing! You can also install generic data tests from a package, or write your own, to use (and reuse) across your dbt project. Check out the [guide on custom generic data tests](https://docs.getdbt.com/best-practices/writing-custom-generic-tests.md) for more information. info There are generic data tests defined in some open-source packages, such as [dbt-utils](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/) and [dbt-expectations](https://hub.getdbt.com/calogica/dbt_expectations/latest/) — skip ahead to the docs on [packages](https://docs.getdbt.com/docs/build/packages.md) to learn more! ##### Example To add a generic (or "schema") data test to your project: 1. Add a `.yml` file to your `models` directory, for example, `models/schema.yml`, with the following content (you may need to adjust the `name:` values for an existing model) models/schema.yml ```yaml models: - name: orders columns: - name: order_id data_tests: - unique - not_null ``` 2. Run the [`dbt test` command](https://docs.getdbt.com/reference/commands/test.md): ```text $ dbt test Found 3 models, 2 tests, 0 snapshots, 0 analyses, 130 macros, 0 operations, 0 seed files, 0 sources 17:31:05 | Concurrency: 1 threads (target='learn') 17:31:05 | 17:31:05 | 1 of 2 START test not_null_order_order_id..................... [RUN] 17:31:06 | 1 of 2 PASS not_null_order_order_id........................... [PASS in 0.99s] 17:31:06 | 2 of 2 START test unique_order_order_id....................... [RUN] 17:31:07 | 2 of 2 PASS unique_order_order_id............................. [PASS in 0.79s] 17:31:07 | 17:31:07 | Finished running 2 tests in 7.17s. Completed successfully Done. PASS=2 WARN=0 ERROR=0 SKIP=0 TOTAL=2 ``` 3. Check out the SQL dbt is running by either: * **dbt:** checking the Details tab. * **dbt v1:** checking the `target/compiled` directory **Unique test** ##### Compiled SQL ```sql select * from ( select order_id from analytics.orders where order_id is not null group by order_id having count(*) > 1 ) validation_errors ``` ##### Templated SQL ```sql select * from ( select {{ column_name }} from {{ model }} where {{ column_name }} is not null group by {{ column_name }} having count(*) > 1 ) validation_errors ``` **Not null test** ##### Compiled SQL ```sql select * from analytics.orders where order_id is null ``` ##### Templated SQL ```sql select * from {{ model }} where {{ column_name }} is null ``` #### Running only data tests To run data tests while excluding unit tests, use the `test_type` selector — this works across all engines (dbt v1 and dbt v2): ```bash dbt test --select "test_type:data" ``` In dbt (v1.9+), you can also use `dbt test --resource-type test`. For more options, refer to [test selection examples](https://docs.getdbt.com/reference/node-selection/test-selection-examples.md). #### Storing data test failures Normally, a data test query will calculate failures as part of its execution. If you set the optional `--store-failures` flag, the [`store_failures`](https://docs.getdbt.com/reference/resource-configs/store_failures.md), or the [`store_failures_as`](https://docs.getdbt.com/reference/resource-configs/store_failures_as.md) configs, dbt will first save the results of a test query to a table in the database, and then query that table to calculate the number of failures. This workflow allows you to query and examine failing records much more quickly in development: [![Store test failures in the database for faster development-time debugging.](/img/docs/building-a-dbt-project/test-store-failures.gif?v=2 "Store test failures in the database for faster development-time debugging.")](#)Store test failures in the database for faster development-time debugging. Note that, if you choose to store data test failures: * Test result tables are created in a schema suffixed or named `dbt_test__audit`, by default. It is possible to change this value by setting a `schema` config. (For more details on schema naming, see [using custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md).) * A test's results will always **replace** previous failures for the same test. #### New `data_tests:` syntax Data tests were historically called "tests" in dbt as the only form of testing available. With the introduction of unit tests, the key was renamed from `tests:` to `data_tests:`. dbt still supports `tests:` in your YAML configuration files for backward-compatibility purposes, and you might see it used throughout our documentation. However, you can't have a `tests` and a `data_tests` key associated with the same resource (for example, a single model) at the same time. models/schema.yml ```yaml models: - name: orders columns: - name: order_id data_tests: - unique - not_null ``` dbt\_project.yml ```yaml data_tests: +store_failures: true ``` #### Tests with and without `arguments` Some generic data tests need only the test name. Others need extra inputs, which you nest under an `arguments` property (available in v1.10.5 and higher). In v2, nesting test inputs under `arguments` is required. You will get a validation error if you set test inputs as top-level properties next to the test name. ##### Without `arguments` Tests like `unique` and `not_null` don't need extra inputs. For example, `unique` asserts that a column has no duplicate values, and `not_null` asserts that a column has no nulls. dbt already knows the model and column from the YAML context, so you can list the test by name: models/schema.yml ```yaml models: - name: orders columns: - name: order_id data_tests: - unique - not_null ``` ##### With `arguments` Tests like `accepted_values` and `relationships` need extra inputs for the test macro. For example, `accepted_values` asserts that a column's values are in a supplied list, and `relationships` asserts that each value exists in another model (referential integrity). Nest those inputs under `arguments:`: models/schema.yml ```yaml models: - name: orders columns: - name: status data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. values: ['placed', 'shipped', 'completed', 'returned'] - name: customer_id data_tests: - relationships: arguments: to: ref('customers') field: id ``` Use `arguments:` for inputs to the test macro (for example, `values`, `to`, and `field`). Use [`config`](https://docs.getdbt.com/reference/data-test-configs.md) for framework options such as `severity`, `where`, and `store_failures`. If you previously set test inputs as top-level properties next to the test name, nest them under `arguments:` instead. For details, refer to [`require_generic_test_arguments_property`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_generic_test_arguments_property.md) and [MissingArgumentsPropertyInGenericTestDeprecation](https://docs.getdbt.com/reference/deprecations.md#missingargumentspropertyingenerictestdeprecation). #### Add context to failing rows Data tests return one row for each failure. The columns in your test's SQL select statement are the columns you see when you debug failures, including when you [store test failures](#storing-data-test-failures). Built-in tests often return only the failing value. To include more context per failing row, write a [singular data test](#singular-data-tests) (or a [custom generic data test](https://docs.getdbt.com/best-practices/writing-custom-generic-tests.md)) that selects the extra columns you want. Building on the `accepted_values` test above, here's a singular test that also captures `customer_id` for context. tests/assert\_unexpected\_order\_status.sql ```sql select order_id, status, customer_id from {{ ref('orders') }} where status not in ('placed', 'shipped', 'completed', 'returned') ``` When you run `dbt test --store-failures` (or set the [`store_failures`](https://docs.getdbt.com/reference/resource-configs/store_failures.md) config), dbt saves those rows so you can query them and inspect every selected column. #### FAQs What data tests are available for me to use in dbt? Out of the box, dbt ships with the following data tests: * `unique` * `not_null` * `accepted_values` * `relationships` (for example, referential integrity) You can also write your own [custom generic tests](https://docs.getdbt.com/docs/build/data-tests.md#generic-data-tests). Some additional generic tests have been open-sourced in the [dbt-utils package](https://github.com/dbt-labs/dbt-utils#generic-tests). Check out the docs on [packages](https://docs.getdbt.com/docs/build/packages.md) to learn how to make these tests available in your project. How do I test one model at a time? Running tests on one model looks very similar to running a model: use the `--select` flag (or `-s` flag), followed by the name of the model: ```shell dbt test --select customers ``` Check out the [model selection syntax documentation](https://docs.getdbt.com/reference/node-selection/syntax.md) for full syntax, and [test selection examples](https://docs.getdbt.com/reference/node-selection/test-selection-examples.md) in particular. One of my tests failed, how can I debug it? To debug a failing test, find the SQL that dbt ran by: * dbt: * Within the test output, click on the failed test, and then select "Details". * dbt v1: * Open the file path returned as part of the error message. * Navigate to the `target/compiled/schema_tests` directory for all compiled test queries. Copy the SQL into a query editor (in dbt, you can paste it into a new `Statement`), and run the query to find the records that failed. What data tests should I add to my project? We recommend that every model has a data test on a primary key, that is, a column that is `unique` and `not_null`. We also recommend that you test any assumptions on your source data. For example, if you believe that your payments can only be one of three payment methods, you should test that assumption regularly — a new payment method may introduce logic errors in your SQL. In advanced dbt projects, we recommend using [sources](https://docs.getdbt.com/docs/build/sources.md) and running these source data-integrity tests against the sources rather than models. When should I run my data tests? You should run your data tests whenever you are writing new code (to ensure you haven't broken any existing models by changing SQL), and whenever you run your transformations in production (to ensure that your assumptions about your source data are still valid). Can I store my data tests in a directory other than the \`tests\` directory in my project? By default, dbt expects your singular data test files to be located in the `tests` subdirectory of your project, and generic data test definitions to be located in `tests/generic` or `macros`. To change this, update the [test-paths](https://docs.getdbt.com/reference/project-configs/test-paths.md) configuration in your `dbt_project.yml` file, like so: dbt\_project.yml ```yml test-paths: ["my_cool_tests"] ``` Then, you can define generic data tests in `my_cool_tests/generic/`, and singular data tests everywhere else in `my_cool_tests/`. How do I run data tests on just my sources? To run data tests on all sources, use the following command: ```shell dbt test --select "source:*" ``` (You can also use the `-s` shorthand here instead of `--select`) To run data tests on one source (and all of its tables): ```shell $ dbt test --select source:jaffle_shop ``` And, to run data tests on one source table only: ```shell $ dbt test --select source:jaffle_shop.orders ``` Can I set test failure thresholds? You can use the `error_if` and `warn_if` configs to set custom failure thresholds in your tests. For more details, see [reference](https://docs.getdbt.com/reference/resource-configs/severity.md) for more information. You can also try the following solutions: * Setting the [severity](https://docs.getdbt.com/reference/resource-configs/severity.md) to `warn` or `error` * Writing a [custom generic test](https://docs.getdbt.com/best-practices/writing-custom-generic-tests.md) that accepts a threshold argument ([example](https://discourse.getdbt.com/t/creating-an-error-threshold-for-schema-tests/966)) Can I test the uniqueness of two columns? Yes, there's a few different options for testing the uniqueness of two columns. Consider an orders table that contains records from multiple countries, and the combination of ID and country code is unique: | order\_id | country\_code | | --------- | ------------- | | 1 | AU | | 2 | AU | | ... | ... | | 1 | US | | 2 | US | | ... | ... | Here are some approaches: ###### 1. Create a unique key in the model and test that models/orders.sql ```sql select country_code || '-' || order_id as surrogate_key, ... ``` models/orders.yml ```yml models: - name: orders columns: - name: surrogate_key data_tests: - unique ``` ###### 2. Test an expression models/orders.yml ```yml models: - name: orders data_tests: - unique: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. column_name: "(country_code || '-' || order_id)" ``` ###### 3. Use the `dbt_utils.unique_combination_of_columns` test This is especially useful for large datasets since it is more performant. Check out the docs on [packages](https://docs.getdbt.com/docs/build/packages.md) for more information. models/orders.yml ```yml models: - name: orders data_tests: - dbt_utils.unique_combination_of_columns: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. combination_of_columns: - country_code - order_id ``` --- ### Add Exposures to your DAG Exposures make it possible to define and describe a downstream use of your dbt project, such as in a dashboard, application, or data science pipeline. By defining exposures, you can then: * run, test, and list resources that feed into your exposure * populate a dedicated page in the auto-generated [documentation](https://docs.getdbt.com/docs/build/documentation.md) site with context relevant to data consumers Exposures can be defined in two ways: * Manual — Declared [explicitly](https://docs.getdbt.com/docs/build/exposures.md#declaring-an-exposure) in your project’s YAML files. * Automatic — dbt [creates and visualizes downstream exposures](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures.md) automatically for supported integrations, removing the need for manual YAML definitions. These downstream exposures are stored in dbt’s metadata system, appear in [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md), and behave like manual exposures. However, they don’t exist in YAML files. ##### Declaring an exposure Exposures are defined in `.yml` files nested under an `exposures:` key. The following example shows an exposure definition in a `models/<filename>.yml` file: models/\<filename>.yml ```yaml exposures: - name: weekly_jaffle_metrics label: Jaffles by the Week type: dashboard maturity: high url: https://bi.tool/dashboards/1 description: > Did someone say "exponential growth"? depends_on: - ref('fct_orders') - ref('dim_customers') - source('gsheets', 'goals') - metric('count_orders') owner: name: Callum McData email: data@jaffleshop.com ``` ##### Available properties *Required:* * **name**: a unique exposure name written in [snake case](https://en.wikipedia.org/wiki/Snake_case) * **type**: one of `dashboard`, `notebook`, `analysis`, `ml`, `application` (used to organize in docs site) * **owner**: `name` or `email` required; additional properties allowed *Expected:* * **depends\_on**: list of refable nodes, including `metric`, `ref`, and `source`. While possible, it is highly unlikely you will ever need an `exposure` to depend on a `source` directly. tip Do not confuse this `depends_on` YAML property with the [`-- depends_on`](https://docs.getdbt.com/reference/dbt-jinja-functions/statement-blocks.md#example-using-depends_on) a SQL comment directive defined at the top of a model SQL file, which explicitly adds dependencies to the DAG and ensures the referenced resources are built before the model executes. *Optional:* * **label**: May contain spaces, capital letters, or special characters. * **url**: Activates and populates the link to **View this exposure** in the upper right corner of the generated documentation site * **maturity**: Indicates the level of confidence or stability in the exposure. One of `high`, `medium`, or `low`. For example, you could use `high` maturity for a well-established dashboard, widely used and trusted within your organization. Use `low` maturity for a new or experimental analysis. *General properties (optional)* * [**description**](https://docs.getdbt.com/reference/resource-properties/description.md) * [**tags**](https://docs.getdbt.com/reference/resource-configs/tags.md) * [**meta**](https://docs.getdbt.com/reference/resource-configs/meta.md) * [**enabled**](https://docs.getdbt.com/reference/resource-configs/enabled.md) — You can set this property at the exposure level or at the project level in the [`dbt_project.yml`](https://docs.getdbt.com/reference/dbt_project.yml.md) file. ##### Referencing exposures Once an exposure is defined, you can run commands that reference it: ```text dbt run -s +exposure:weekly_jaffle_report dbt test -s +exposure:weekly_jaffle_report ``` When we generate the [Catalog site](https://docs.getdbt.com/docs/explore/explore-projects.md), you'll see the exposure appear: [![Exposures has a dedicated section, under the project name in dbt Catalog, which lists each exposure in your project.](/img/docs/building-a-dbt-project/dbt-explorer-exposures.png?v=2 "Exposures has a dedicated section, under the project name in dbt Catalog, which lists each exposure in your project.")](#)Exposures has a dedicated section, under the project name in dbt Catalog, which lists each exposure in your project. [![Exposures appear as nodes in the dbt Catalog DAG. It displays an orange 'EXP' indicator within the node. ](/img/docs/building-a-dbt-project/dag-exposures.png?v=2 "Exposures appear as nodes in the dbt Catalog DAG. It displays an orange 'EXP' indicator within the node. ")](#)Exposures appear as nodes in the dbt Catalog DAG. It displays an orange 'EXP' indicator within the node. #### Related docs * [Exposure properties](https://docs.getdbt.com/reference/exposure-properties.md) * [`exposure:` selection method](https://docs.getdbt.com/reference/node-selection/methods.md#exposure) * [Data health tiles](https://docs.getdbt.com/docs/explore/data-tile.md) --- ### Add groups to your DAG A group is a collection of nodes within a dbt DAG. Groups are named, and every group has an `owner`. They enable intentional collaboration within and across teams by restricting [access to private](https://docs.getdbt.com/reference/resource-configs/access.md) models. Group members may include models, tests, seeds, snapshots, analyses, and metrics. (Not included: sources and exposures.) Each node may belong to only one group. ##### Declaring a group Groups are defined in `.yml` files, nested under a `groups:` key. In version 1.10 and higher, you can add a `description` and a `meta` config to add more information about the group. (Applies to dbt v1.10 and later) models/marts/finance/finance.yml ```yaml groups: - name: finance description: "All finance-related models owned by the Finance team." # optional owner: # 'name' or 'email' is required; additional properties will no longer be allowed in a future release email: finance@jaffleshop.com config: meta: # optional data_owner: Finance team cost_center: finance data_classification: sensitive ``` ###### Centrally defining a group To centrally define a group in your project, there are two options: * Create one `_groups.yml` file in the root of the `models` directory. * Create one `_groups.yml` file in the root of a `groups` directory. For this option, you also need to configure [`model-paths`](https://docs.getdbt.com/reference/project-configs/model-paths.md) in the `dbt_project.yml` file: ```yml model-paths: ["models", "groups"] ``` ##### Group properties The following properties are available when defining a group: | Property | Required | Description | | ------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | Required | A unique name for the group within the project. Used to assign resources via the [`group` config](https://docs.getdbt.com/reference/resource-configs/group.md). | | `owner` | Required | Identifies who is responsible for the group. Must include either `name:` or `email:`. | | `description` | Optional | A human readable description of the group's purpose. Supports markdown and the [`doc` Jinja function](https://docs.getdbt.com/reference/dbt-jinja-functions/doc.md). Supported in v1.10 and later. | | `config.meta` | Optional | A dictionary of arbitrary key/value metadata about the group. Useful for storing information such as cost centers, data classifications, or team contact details. | models/marts/finance/finance.yml ```yaml groups: - name: finance description: "All models owned by the Finance team." owner: email: finance@jaffleshop.com config: meta: data_owner: Finance team cost_center: finance data_classification: sensitive ``` ##### Adding a model to a group Use the `group` configuration to add one or more models to a group. ##### Project-level dbt\_project.yml ```yml models: marts: finance: +group: finance ``` ##### Model-level models/schema.yml ```yml models: - name: model_name config: group: finance ``` ##### In-file models/model\_name.sql ```sql {{ config(group = 'finance') }} select ... ``` ##### Referencing a model in a group By default, all models within a group have the `protected` [access modifier](https://docs.getdbt.com/reference/resource-configs/access.md). This means they can be referenced by downstream resources in *any* group in the same project, using the [`ref`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) function. If a grouped model's `access` property is set to `private`, only resources within its group can reference it. models/schema.yml ```yml models: - name: finance_private_model config: access: private # changed to config in v1.10 group: finance # in a different group! - name: marketing_model config: group: marketing ``` models/marketing\_model.sql ```sql select * from {{ ref('finance_private_model') }} ``` ```shell $ dbt run -s marketing_model ... dbt.exceptions.DbtReferenceError: Parsing Error Node model.jaffle_shop.marketing_model attempted to reference node model.jaffle_shop.finance_private_model, which is not allowed because the referenced node is private to the finance group. ``` #### Related docs * [Model Access](https://docs.getdbt.com/docs/mesh/govern/model-access.md#groups) * [Group configuration](https://docs.getdbt.com/reference/resource-configs/group.md) * [Group selection](https://docs.getdbt.com/reference/node-selection/methods.md#group) --- ### Add Seeds to your DAG #### Related reference docs * [Seed configurations](https://docs.getdbt.com/reference/seed-configs.md) * [Seed properties](https://docs.getdbt.com/reference/seed-properties.md) * [`seed` command](https://docs.getdbt.com/reference/commands/seed.md) #### Overview Seeds are CSV files in your dbt project (typically in your `seeds` directory), that dbt can load into your data warehouse using the `dbt seed` command. Seeds can be referenced in downstream models the same way as referencing models — by using the [`ref` function](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md). Because these CSV files are located in your dbt repository, they are version controlled and code reviewable. Seeds are best suited to static data which changes infrequently. Good use-cases for seeds: * A list of mappings of country codes to country names * A list of test emails to exclude from analysis * A list of employee account IDs Poor use-cases of dbt seeds: * Loading raw data that has been exported to CSVs * Any kind of production data containing sensitive information. For example personal identifiable information (PII) and passwords. #### Example To load a seed file in your dbt project: 1. Add the file to your `seeds` directory, with a `.csv` file extension, for example, `seeds/country_codes.csv` seeds/country\_codes.csv ```text country_code,country_name US,United States CA,Canada GB,United Kingdom ... ``` 2. Run the `dbt seed` [command](https://docs.getdbt.com/reference/commands/seed.md) — a new table will be created in your warehouse in your target schema, named `country_codes` ```text $ dbt seed Found 2 models, 3 tests, 0 archives, 0 analyses, 53 macros, 0 operations, 1 seed file 14:46:15 | Concurrency: 1 threads (target='dev') 14:46:15 | 14:46:15 | 1 of 1 START seed file analytics.country_codes........................... [RUN] 14:46:15 | 1 of 1 OK loaded seed file analytics.country_codes....................... [INSERT 3 in 0.01s] 14:46:16 | 14:46:16 | Finished running 1 seed in 0.14s. Completed successfully Done. PASS=1 ERROR=0 SKIP=0 TOTAL=1 ``` 3. Refer to seeds in downstream models using the `ref` function. models/orders.sql ```sql -- This refers to the table created from seeds/country_codes.csv select * from {{ ref('country_codes') }} ``` #### Configuring seeds Seeds are configured in your `dbt_project.yml`, check out the [seed configurations](https://docs.getdbt.com/reference/seed-configs.md) docs for a full list of available configurations. #### Documenting and testing seeds You can document and test seeds in YAML by declaring properties — check out the docs on [seed properties](https://docs.getdbt.com/reference/seed-properties.md) for more information. #### FAQs Can I use seeds to load raw data? Seeds should **not** be used to load raw data (for example, large CSV exports from a production database). Since seeds are version controlled, they are best suited to files that contain business-specific logic, for example a list of country codes or user IDs of employees. Loading CSVs using dbt's seed functionality is not performant for large files. Consider using a different tool to load these CSVs into your data warehouse. Can I store my seeds in a directory other than the \`seeds\` directory in my project? By default, dbt expects your seed files to be located in the `seeds` subdirectory of your project. To change this, update the [seed-paths](https://docs.getdbt.com/reference/project-configs/seed-paths.md) configuration in your `dbt_project.yml` file, like so: dbt\_project.yml ```yml seed-paths: ["custom_seeds"] ``` The columns of my seed changed, and now I get an error when running the \`seed\` command, what should I do? If you changed the columns of your seed, you may get a `Database Error`: ##### Snowflake ```shell $ dbt seed Running with dbt=1.6.0-rc2 Found 0 models, 0 tests, 0 snapshots, 0 analyses, 130 macros, 0 operations, 1 seed file, 0 sources 12:12:27 | Concurrency: 8 threads (target='dev_snowflake') 12:12:27 | 12:12:27 | 1 of 1 START seed file dbt_claire.country_codes...................... [RUN] 12:12:30 | 1 of 1 ERROR loading seed file dbt_claire.country_codes.............. [ERROR in 2.78s] 12:12:31 | 12:12:31 | Finished running 1 seed in 10.05s. Completed with 1 error and 0 warnings: Database Error in seed country_codes (seeds/country_codes.csv) 000904 (42000): SQL compilation error: error line 1 at position 62 invalid identifier 'COUNTRY_NAME' Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1 ``` ##### Redshift ```shell $ dbt seed Running with dbt=1.6.0-rc2 Found 0 models, 0 tests, 0 snapshots, 0 analyses, 149 macros, 0 operations, 1 seed file, 0 sources 12:14:46 | Concurrency: 1 threads (target='dev_redshift') 12:14:46 | 12:14:46 | 1 of 1 START seed file dbt_claire.country_codes...................... [RUN] 12:14:46 | 1 of 1 ERROR loading seed file dbt_claire.country_codes.............. [ERROR in 0.23s] 12:14:46 | 12:14:46 | Finished running 1 seed in 1.75s. Completed with 1 error and 0 warnings: Database Error in seed country_codes (seeds/country_codes.csv) column "country_name" of relation "country_codes" does not exist Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1 ``` In this case, you should rerun the command with a `--full-refresh` flag, like so: ```text dbt seed --full-refresh ``` **Why is this the case?** When you typically run dbt seed, dbt truncates the existing table and reinserts the data. This pattern avoids a `drop cascade` command, which may cause downstream objects (that your BI users might be querying!) to get dropped. However, when column names are changed, or new columns are added, these statements will fail as the table structure has changed. The `--full-refresh` flag will force dbt to `drop cascade` the existing table before rebuilding it. How do I test and document seeds? To test and document seeds, use a [properties file](https://docs.getdbt.com/reference/configs-and-properties.md) and nest the configurations under a `seeds:` key #### Example seeds/properties.yml ```yml seeds: - name: country_codes description: A mapping of two letter country codes to country names columns: - name: country_code data_tests: - unique - not_null - name: country_name data_tests: - unique - not_null ``` How do I set a datatype for a column in my seed? dbt will infer the datatype for each column based on the data in your CSV. You can also explicitly set a datatype using the `column_types` [configuration](https://docs.getdbt.com/reference/resource-configs/column_types.md) like so: dbt\_project.yml ```yml seeds: jaffle_shop: # you must include the project name warehouse_locations: +column_types: zipcode: varchar(5) ``` How do I run models downstream of a seed? You can run models downstream of a seed using the [model selection syntax](https://docs.getdbt.com/reference/node-selection/syntax.md), and treating the seed like a model. For example, the following would run all models downstream of a seed named `country_codes`: ```shell $ dbt run --select country_codes+ ``` How do I preserve leading zeros in a seed? If you need to preserve leading zeros (for example in a zipcode or mobile number), include leading zeros in your seed file, and use the `column_types` [configuration](https://docs.getdbt.com/reference/resource-configs/column_types.md) with a varchar datatype of the correct length. How do I build one seed at a time? You can use a `--select` option with the `dbt seed` command, like so: ```shell $ dbt seed --select country_codes ``` There is also an `--exclude` option. Check out more in the [model selection syntax](https://docs.getdbt.com/reference/node-selection/syntax.md) documentation. Do hooks run with seeds? Yes! The following hooks are available: * [pre-hooks & post-hooks](https://docs.getdbt.com/reference/resource-configs/pre-hook-post-hook.md) * [on-run-start & on-run-end hooks](https://docs.getdbt.com/reference/project-configs/on-run-start-on-run-end.md) Configure these in your `dbt_project.yml` file. --- ### Add snapshots to your DAG #### Related documentation * [Snapshot configurations](https://docs.getdbt.com/reference/snapshot-configs.md) * [Snapshot properties](https://docs.getdbt.com/reference/snapshot-properties.md) * [`snapshot` command](https://docs.getdbt.com/reference/commands/snapshot.md) Learn by video! For video tutorials on Snapshots, go to dbt Learn and check out the [Snapshots course](https://learn.getdbt.com/courses/snapshots). #### What are snapshots? Analysts often need to "look back in time" at previous data states in their mutable tables. While some source data systems are built in a way that makes accessing historical data possible, this is not always the case. dbt provides a mechanism, **snapshots**, which records changes to a mutable table over time. Snapshots implement [type-2 Slowly Changing Dimensions](https://en.wikipedia.org/wiki/Slowly_changing_dimension#Type_2:_add_new_row) over mutable source tables. These Slowly Changing Dimensions (or SCDs) identify how a row in a table changes over time. Imagine you have an `orders` table where the `status` field can be overwritten as the order is processed. | id | status | updated\_at | | -- | ------- | ----------- | | 1 | pending | 2024-01-01 | Now, imagine that the order goes from "pending" to "shipped". That same record will now look like: | id | status | updated\_at | | -- | ------- | ----------- | | 1 | shipped | 2024-01-02 | This order is now in the "shipped" state, but we've lost the information about when the order was last in the "pending" state. This makes it difficult (or impossible) to analyze how long it took for an order to ship. dbt can "snapshot" these changes to help you understand how values in a row change over time. Here's an example of a snapshot table for the previous example: | id | status | updated\_at | dbt\_valid\_from | dbt\_valid\_to | | -- | ------- | ----------- | ---------------- | -------------- | | 1 | pending | 2024-01-01 | 2024-01-01 | 2024-01-02 | | 1 | shipped | 2024-01-02 | 2024-01-02 | `null` | #### Configuring snapshots (Applies to dbt v1.9 and later) Configure your snapshots in YAML files to tell dbt how to detect record changes. Define snapshots configurations in YAML files, alongside your models, for a cleaner, faster, and more consistent set up. Place snapshot YAML files in the models directory or in a snapshots directory. snapshots/orders\_snapshot.yml ```yaml snapshots: - name: string relation: relation # source('my_source', 'my_table') or ref('my_model') description: markdown_string config: database: string schema: string alias: string strategy: timestamp | check unique_key: column_name_or_expression check_cols: [column_name] | all updated_at: column_name snapshot_meta_column_names: dictionary dbt_valid_to_current: string hard_deletes: ignore | invalidate | new_record ``` The following table outlines the configurations available for snapshots: | Config | Description | Required? | Example | | ----------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------- | -------------------------------- | | [database](https://docs.getdbt.com/reference/resource-configs/database.md) | Specify a custom database for the snapshot | No | analytics | | [schema](https://docs.getdbt.com/reference/resource-configs/schema.md) | Specify a custom schema for the snapshot | No | snapshots | | [alias](https://docs.getdbt.com/reference/resource-configs/alias.md) | Specify an alias for the snapshot | No | your\_custom\_snapshot | | [strategy](https://docs.getdbt.com/reference/resource-configs/strategy.md) | The snapshot strategy to use. Valid values: `timestamp` or `check` | Yes | timestamp | | [unique\_key](https://docs.getdbt.com/reference/resource-configs/unique_key.md) | A primary key column(s) (string or array) or expression for the record | Yes | `id` or `[order_id, product_id]` | | [check\_cols](https://docs.getdbt.com/reference/resource-configs/check_cols.md) | If using the `check` strategy, then the columns to check | Only if using the `check` strategy | \["status"] | | [updated\_at](https://docs.getdbt.com/reference/resource-configs/updated_at.md) | A column in your snapshot query results that indicates when each record was last updated, used in the `timestamp` strategy. May support ISO date strings and unix epoch integers, depending on the data platform you use. | Only if using the `timestamp` strategy | updated\_at | | [dbt\_valid\_to\_current](https://docs.getdbt.com/reference/resource-configs/dbt_valid_to_current.md) | Set a custom indicator for the value of `dbt_valid_to` in current snapshot records (like a future date). By default, this value is `NULL`. When configured, dbt will use the specified value instead of `NULL` for `dbt_valid_to` for current records in the snapshot table. | No | string | | [snapshot\_meta\_column\_names](https://docs.getdbt.com/reference/resource-configs/snapshot_meta_column_names.md) | Customize the names of the snapshot meta fields | No | dictionary | | [hard\_deletes](https://docs.getdbt.com/reference/resource-configs/hard-deletes.md) | Specify how to handle deleted rows from the source. Supported options are `ignore` (default), `invalidate` (replaces the legacy `invalidate_hard_deletes=true`), and `new_record`. | No | string | * In v1.9, `target_schema` became optional, allowing snapshots to be environment-aware. By default, without `target_schema` or `target_database` defined, snapshots now use the `generate_schema_name` or `generate_database_name` macros to determine where to build. * Developers can still set a custom location with [`schema`](https://docs.getdbt.com/reference/resource-configs/schema.md) and [`database`](https://docs.getdbt.com/reference/resource-configs/database.md) configs, consistent with other resource types. * A number of other configurations are also supported (for example, `tags` and `post-hook`). For the complete list, refer to [Snapshot configurations](https://docs.getdbt.com/reference/snapshot-configs.md). * You can configure snapshots from both the `dbt_project.yml` file and a `config` block. For more information, refer to the [configuration docs](https://docs.getdbt.com/reference/snapshot-configs.md). * Starting dbt v1.12, you can inspect the SQL generated for snapshots by running [`dbt compile`](https://docs.getdbt.com/reference/commands/compile.md). You can find compiled SQL files in the `target/compiled/` directory of your dbt project. ##### Add a snapshot to your project To add a snapshot to your project follow these steps. For users on versions 1.8 and earlier, refer to [Legacy snapshot configurations](https://docs.getdbt.com/reference/resource-configs/snapshots-jinja-legacy.md). 1. Create a properties YAML file in your `snapshots` directory: `snapshots/orders_snapshot.yml` and add your configuration details. You can also configure your snapshot from your project YAML file (`dbt_project.yml`) ([docs](https://docs.getdbt.com/reference/snapshot-configs.md)). snapshots/orders\_snapshot.yml ```yaml snapshots: - name: orders_snapshot relation: source('jaffle_shop', 'orders') config: schema: snapshots database: analytics unique_key: id strategy: timestamp updated_at: updated_at dbt_valid_to_current: "to_date('9999-12-31')" # Specifies that current records should have `dbt_valid_to` set to `'9999-12-31'` instead of `NULL`. ``` 2. (Optional) Apply transformations using an ephemeral model. By default, snapshots reference a source directly (as shown in the YAML in the previous step). If you need to apply transformations (such as filtering or deduplication), define an ephemeral model first to apply those transformations, and reference it in the snapshot relation field instead of calling `source()` directly. For example, here's an ephemeral model: models/ephemeral\_orders.sql ```sql {{ config(materialized='ephemeral') }} select * from {{ source('jaffle_shop', 'orders') }} ``` This is how to reference the ephemeral model in the `relation` field: snapshots/orders\_snapshot.yml ```yaml snapshots: - name: orders_snapshot relation: ref('ephemeral_orders') ... rest of config... ``` 3. Check whether the result set of your query includes a reliable timestamp column that indicates when a record was last updated. For our example, the `updated_at` column reliably indicates record changes, so we can use the `timestamp` strategy. If your query result set does not have a reliable timestamp, you'll need to instead use the `check` strategy — more details on this below. 4. Run the `dbt snapshot` [command](https://docs.getdbt.com/reference/commands/snapshot.md) — for our example, a new table will be created at `analytics.snapshots.orders_snapshot`. The [`schema`](https://docs.getdbt.com/reference/resource-configs/schema.md) config will utilize the `generate_schema_name` macro. ```text $ dbt snapshot Running with dbt=1.9.0 15:07:36 | Concurrency: 8 threads (target='dev') 15:07:36 | 15:07:36 | 1 of 1 START snapshot snapshots.orders_snapshot...... [RUN] 15:07:36 | 1 of 1 OK snapshot snapshots.orders_snapshot..........[SELECT 3 in 1.82s] 15:07:36 | 15:07:36 | Finished running 1 snapshots in 0.68s. Completed successfully Done. PASS=2 ERROR=0 SKIP=0 TOTAL=1 ``` Compiled SQL for snapshots Starting dbt v1.12, you can inspect the SQL generated for this snapshot by running [`dbt compile`](https://docs.getdbt.com/reference/commands/compile.md) or `dbt compile --select orders_snapshot`. Open the compiled SQL in `target/compiled/` to inspect or debug the generated queries. Each snapshot is compiled into its own SQL file, even if multiple snapshots are defined in the same source file. 5. Inspect the results by selecting from the table dbt created (`analytics.snapshots.orders_snapshot`). After the first run, you should see the results of your query, plus the [snapshot meta fields](#snapshot-meta-fields) as described later on. 6. Run the `dbt snapshot` command again and inspect the results. If any records have been updated, the snapshot should reflect this. 7. Select from the `snapshot` in downstream models using the `ref` function. models/changed\_orders.sql ```sql select * from {{ ref('orders_snapshot') }} ``` 8. Snapshots are only useful if you run them frequently — schedule the `dbt snapshot` command to run regularly. ##### Configuration best practices  Use the timestamp strategy where possible The timestamp strategy is recommended because it handles column additions and deletions more efficiently than the `check` strategy. This is because it's more robust to schema changes, especially when columns are added or removed over time. The timestamp strategy relies on a single `updated_at` field, which means it avoids the need to constantly update your snapshot configuration as your source table evolves. Why timestamp is the preferred strategy: * Requires tracking only one column (`updated_at`) * Automatically handles new or removed columns in the source table * Less prone to errors when the table schema evolves over time (for example, if using the `check` strategy, you might need to update the `check_cols` configuration)  Use dbt\_valid\_to\_current for easier date range queries By default, `dbt_valid_to` is `NULL` for current records. However, if you set the [`dbt_valid_to_current` configuration](https://docs.getdbt.com/reference/resource-configs/dbt_valid_to_current.md) (available in dbt v1.9+), `dbt_valid_to` will be set to your specified value (such as `9999-12-31`) for current records. This allows for straightforward date range filtering.  Ensure your unique key is really unique The unique key is used by dbt to match rows up, so it's extremely important to make sure this key is actually unique! If you're snapshotting a source, I'd recommend adding a uniqueness test to your source ([example](https://github.com/dbt-labs/jaffle_shop/blob/8e7c853c858018180bef1756ec93e193d9958c5b/models/staging/schema.yml#L26)). (Applies to dbt v1.9 and later)  Use a schema that is separate to your models' schema Snapshots can't be rebuilt. Because of this, it's a good idea to put snapshots in a separate schema so end users know they're special. From there, you may want to set different privileges on your snapshots compared to your models, and even run them as a different user (or role, depending on your warehouse) to make it very difficult to drop a snapshot unless you really want to.  Use ephemeral model to clean or transform data before snapshotting If you need to clean or transform your data before snapshotting, create an ephemeral model or a staging model that applies the necessary transformations. Then, reference this model in your snapshot configuration. This approach keeps your snapshot definitions clean and allows you to test and run transformations separately. ##### How snapshots work When you run the [`dbt snapshot` command](https://docs.getdbt.com/reference/commands/snapshot.md): * **On the first run:** dbt will create the initial snapshot table — this will be the result set of your `select` statement, with additional columns including `dbt_valid_from` and `dbt_valid_to`. All records will have a `dbt_valid_to = null` or the value specified in [`dbt_valid_to_current`](https://docs.getdbt.com/reference/resource-configs/dbt_valid_to_current.md) (available in dbt 1.9+) if configured. * **On subsequent runs:** dbt will check which records have changed or if any new records have been created: * The `dbt_valid_to` column will be updated for any existing records that have changed. * The updated record and any new records will be inserted into the snapshot table. These records will now have `dbt_valid_to = null` or the value configured in `dbt_valid_to_current` (available in dbt v1.9+). Snapshots ignore full refresh Snapshots ignore both the `full_refresh` config and the `--full-refresh` flag. A command such as `dbt build --full-refresh` or `dbt snapshot --full-refresh` that includes a snapshot node runs the snapshot as normal — it won't drop or recreate the snapshot table, so existing snapshot history is preserved. (Applies to dbt v1.9 and later) ###### Note * These column names can be customized to your team or organizational conventions using the [snapshot\_meta\_column\_names](#snapshot-meta-fields) config. * Use the `dbt_valid_to_current` config to set a custom indicator for the value of `dbt_valid_to` in current snapshot records (like a future date such as `9999-12-31`). By default, this value is `NULL`. When set, dbt will use this specified value instead of `NULL` for `dbt_valid_to` for current records in the snapshot table. * Use the [`hard_deletes`](https://docs.getdbt.com/reference/resource-configs/hard-deletes.md) config to track hard deletes by adding a new record when row become "deleted" in source. Supported options are `ignore`, `invalidate`, and `new_record`. Snapshots can be referenced in downstream models the same way as referencing models — by using the [ref](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) function. #### Detecting row changes Snapshot "strategies" define how dbt knows if a row has changed. There are two strategies built-in to dbt: * [Timestamp](#timestamp-strategy-recommended) — Uses an `updated_at` column to determine if a row has changed. * [Check](#check-strategy) — Compares a list of columns between their current and historical values to determine if a row has changed. ##### Timestamp strategy (recommended) The `timestamp` strategy uses an `updated_at` field to determine if a row has changed. If the configured `updated_at` column for a row is more recent than the last time the snapshot ran, then dbt will invalidate the old record and record the new one. If the timestamps are unchanged, then dbt will not take any action. Why timestamp is recommended? * Requires tracking only one column (`updated_at`) * Automatically handles new or removed columns in the source table * Less prone to errors when the table schema evolves over time (for example, if using the `check` strategy, you might need to update the `check_cols` configuration) The `timestamp` strategy requires the following configurations: | Config | Description | Example | | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | | updated\_at | A column which represents when the source row was last updated. May support ISO date strings and unix epoch integers, depending on the data platform you use. | `updated_at` | **Example usage:** (Applies to dbt v1.9 and later) snapshots/orders\_snapshot.yml ```yaml snapshots: - name: orders_snapshot_timestamp relation: source('jaffle_shop', 'orders') config: schema: snapshots unique_key: id strategy: timestamp updated_at: updated_at ``` ##### Check strategy The `check` strategy is useful for tables which do not have a reliable `updated_at` column. This strategy works by comparing a list of columns between their current and historical values. If any of these columns have changed, then dbt will invalidate the old record and record the new one. If the column values are identical, then dbt will not take any action. The `check` strategy requires the following configurations: | Config | Description | Example | | ----------- | --------------------------------------------------------------------- | ------------------- | | check\_cols | A list of columns to check for changes, or `all` to check all columns | `["name", "email"]` | check\_cols = 'all' The `check` snapshot strategy can be configured to track changes to *all* columns by supplying `check_cols = 'all'`. It is better to explicitly enumerate the columns that you want to check. Consider using a surrogate key to condense many columns into a single column. ###### Example usage (Applies to dbt v1.9 and later) snapshots/orders\_snapshot.yml ```yaml snapshots: - name: orders_snapshot_check relation: source('jaffle_shop', 'orders') config: schema: snapshots unique_key: id strategy: check check_cols: - status - is_cancelled ``` ###### Example usage with `updated_at` When using the `check` strategy, dbt tracks changes by comparing values in `check_cols`. By default, dbt uses the timestamp to update `dbt_updated_at`, `dbt_valid_from` and `dbt_valid_to` fields. Optionally you can set an `updated_at` column: * If `updated_at` is configured, the `check` strategy uses this column instead, as with the timestamp strategy. * If `updated_at` value is null, dbt defaults to using the current timestamp. Check out the following example, which shows how to use the `check` strategy with `updated_at`: ```yaml snapshots: - name: orders_snapshot relation: ref('stg_orders') config: schema: snapshots unique_key: order_id strategy: check check_cols: - status - is_cancelled updated_at: updated_at ``` In this example: * If at least one of the specified `check_cols `changes, the snapshot creates a new row. If the `updated_at` column has a value (is not null), the snapshot uses it; otherwise, it defaults to the timestamp. * If `updated_at` isn’t set, then dbt automatically falls back to [using the current timestamp](#sample-results-for-the-check-strategy) to track changes. * Use this approach when your `updated_at` column isn't reliable for tracking record updates, but you still want to use it — rather than the snapshot's execution time — whenever row changes are detected. ##### Hard deletes (opt-in) (Applies to dbt v1.9 and later) In dbt v1.9 and higher, the [`hard_deletes`](https://docs.getdbt.com/reference/resource-configs/hard-deletes.md) config replaces the `invalidate_hard_deletes` config to give you more control on how to handle deleted rows from the source. The `hard_deletes` config is not a separate strategy but an additional opt-in feature that can be used with any snapshot strategy. The `hard_deletes` config has three options/fields: | Field | Description | | ------------------ | --------------------------------------------------------------------------------------------------------------------------------- | | `ignore` (default) | No action for deleted records. | | `invalidate` | Behaves the same as the existing `invalidate_hard_deletes=true`, where deleted records are invalidated by setting `dbt_valid_to`. | | `new_record` | Tracks deleted records as new rows using the `dbt_is_deleted` [meta field](#snapshot-meta-fields) when records are deleted. |  When to use the hard\_deletes and invalidate\_hard\_deletes config? **Use `invalidate_hard_deletes` (v1.8 and earlier) if:** * Gaps in the snapshot history (missing records for deleted rows) are acceptable. * You want to invalidate deleted rows by setting their `dbt_valid_to` timestamp to the current time (implicit delete). * You are working with smaller datasets where tracking deletions as a separate state is unnecessary. **Use `hard_deletes: new_record` (v1.9 and higher) if:** * You want to maintain continuous snapshot history without gaps. * You want to explicitly track deletions by adding new rows with a `dbt_is_deleted` column (explicit delete). * You are working with larger datasets where explicitly tracking deleted records improves data lineage clarity. ###### Example usage snapshots/orders\_snapshot.yml ```yaml snapshots: - name: orders_snapshot_hard_delete relation: source('jaffle_shop', 'orders') config: schema: snapshots unique_key: id strategy: timestamp updated_at: updated_at hard_deletes: new_record # options are: 'ignore', 'invalidate', or 'new_record' ``` In this example, the `hard_deletes: new_record` config will add a new row for deleted records with the `dbt_is_deleted` column set to `True`. Any restored records are added as new rows with the `dbt_is_deleted` field set to `False`. The resulting table will look like this: | id | status | updated\_at | dbt\_valid\_from | dbt\_valid\_to | dbt\_is\_deleted | | -- | -------- | ---------------- | ---------------- | ---------------- | ---------------- | | 1 | pending | 2024-01-01 10:47 | 2024-01-01 10:47 | 2024-01-01 11:05 | False | | 1 | shipped | 2024-01-01 11:05 | 2024-01-01 11:05 | 2024-01-01 11:20 | False | | 1 | deleted | 2024-01-01 11:20 | 2024-01-01 11:20 | 2024-01-01 12:00 | True | | 1 | restored | 2024-01-01 12:00 | 2024-01-01 12:00 | | False | #### Snapshot meta-fields Snapshot tables will be created as a clone of your source dataset, plus some additional meta-fields. In dbt v1.9+ (or available sooner in [the **v1 Latest** release track in the dbt platform](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md)): * These column names can be customized to your team or organizational conventions using the [`snapshot_meta_column_names`](https://docs.getdbt.com/reference/resource-configs/snapshot_meta_column_names.md) config. * Use the [`dbt_valid_to_current` config](https://docs.getdbt.com/reference/resource-configs/dbt_valid_to_current.md) to set a custom indicator for the value of `dbt_valid_to` in current snapshot records (like a future date such as `9999-12-31`). By default, this value is `NULL`. When set, dbt will use this specified value instead of `NULL` for `dbt_valid_to` for current records in the snapshot table. * Use the [`hard_deletes`](https://docs.getdbt.com/reference/resource-configs/hard-deletes.md) config to track deleted records as new rows with the `dbt_is_deleted` meta field when using the `hard_deletes='new_record'` field. | Field | Meaning | Notes | Example | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- | | `dbt_valid_from` | The timestamp when this snapshot row became valid. | Use this to order versions of a record. For a given row, this often matches `dbt_updated_at`, but they mean different things. Refer to [Strategy and meta-field timestamps](#strategy-and-meta-field-timestamps) for examples by strategy. | `snapshot_meta_column_names: {dbt_valid_from: start_date}` | | `dbt_valid_to` | The timestamp when this row became invalidated. For current records, this is `NULL` by default or the value specified in `dbt_valid_to_current`. | The most recent snapshot record will have `dbt_valid_to` set to `NULL` or the specified value. | `snapshot_meta_column_names: {dbt_valid_to: end_date}` | | `dbt_scd_id` | A unique key generated for each snapshot row. | Used internally by dbt to identify each SCD version. Refer to [How dbt\_scd\_id is calculated](https://docs.getdbt.com/reference/resource-configs/snapshot_meta_column_names.md#how-dbt_scd_id-is-calculated) to learn how dbt builds this key. | `snapshot_meta_column_names: {dbt_scd_id: scd_id}` | | `dbt_updated_at` | The source record's change timestamp when this snapshot row was inserted. | Used internally by dbt. [Strategy and meta-field timestamps](#strategy-and-meta-field-timestamps) describes which value populates this column for each strategy. | `snapshot_meta_column_names: {dbt_updated_at: modified_date}` | | `dbt_is_deleted` | A string value indicating if the record has been deleted. (`True` if deleted, `False` if not deleted). | Added when `hard_deletes='new_record'` is configured. | `snapshot_meta_column_names: {dbt_is_deleted: is_deleted}` | All of these column names can be customized using the `snapshot_meta_column_names` config. Refer to this [example](https://docs.getdbt.com/reference/resource-configs/snapshot_meta_column_names.md#example) for more details. On insert, `dbt_valid_from` and `dbt_updated_at` are set from the same value. They represent validity start and recorded change time, respectively. ##### Strategy and meta-field timestamps The timestamps used for each column depend on the strategy you use: * For the `timestamp` strategy, the configured `updated_at` column is used to populate the `dbt_valid_from`, `dbt_valid_to` and `dbt_updated_at` columns.  Sample results for the timestamp strategy Snapshot query results at `2024-01-01 11:00` | id | status | updated\_at | | -- | ------- | ---------------- | | 1 | pending | 2024-01-01 10:47 | Snapshot results (note that `11:00` is not used anywhere): | id | status | updated\_at | dbt\_valid\_from | dbt\_valid\_to | dbt\_updated\_at | | -- | ------- | ---------------- | ---------------- | -------------- | ---------------- | | 1 | pending | 2024-01-01 10:47 | 2024-01-01 10:47 | | 2024-01-01 10:47 | Query results at `2024-01-01 11:30`: | id | status | updated\_at | | -- | ------- | ---------------- | | 1 | shipped | 2024-01-01 11:05 | Snapshot results (note that `11:30` is not used anywhere): | id | status | updated\_at | dbt\_valid\_from | dbt\_valid\_to | dbt\_updated\_at | | -- | ------- | ---------------- | ---------------- | ---------------- | ---------------- | | 1 | pending | 2024-01-01 10:47 | 2024-01-01 10:47 | 2024-01-01 11:05 | 2024-01-01 10:47 | | 1 | shipped | 2024-01-01 11:05 | 2024-01-01 11:05 | | 2024-01-01 11:05 | Snapshot results with `hard_deletes='new_record'`: | id | status | updated\_at | dbt\_valid\_from | dbt\_valid\_to | dbt\_updated\_at | dbt\_is\_deleted | | -- | ------- | ---------------- | ---------------- | ---------------- | ---------------- | ---------------- | | 1 | pending | 2024-01-01 10:47 | 2024-01-01 10:47 | 2024-01-01 11:05 | 2024-01-01 10:47 | False | | 1 | shipped | 2024-01-01 11:05 | 2024-01-01 11:05 | 2024-01-01 11:20 | 2024-01-01 11:05 | False | | 1 | deleted | 2024-01-01 11:20 | 2024-01-01 11:20 | | 2024-01-01 11:20 | True | * For the `check` strategy, the current timestamp is used to populate each column. If configured, the `check` strategy uses the `updated_at` column instead, as with the timestamp strategy.  Sample results for the check strategy Snapshot query results at `2024-01-01 11:00` | id | status | | -- | ------- | | 1 | pending | Snapshot results: | id | status | dbt\_valid\_from | dbt\_valid\_to | dbt\_updated\_at | | -- | ------- | ---------------- | -------------- | ---------------- | | 1 | pending | 2024-01-01 11:00 | | 2024-01-01 11:00 | Query results at `2024-01-01 11:30`: | id | status | | -- | ------- | | 1 | shipped | Snapshot results: | id | status | dbt\_valid\_from | dbt\_valid\_to | dbt\_updated\_at | | -- | ------- | ---------------- | ---------------- | ---------------- | | 1 | pending | 2024-01-01 11:00 | 2024-01-01 11:30 | 2024-01-01 11:00 | | 1 | shipped | 2024-01-01 11:30 | | 2024-01-01 11:30 | Snapshot results with `hard_deletes='new_record'`: | id | status | dbt\_valid\_from | dbt\_valid\_to | dbt\_updated\_at | dbt\_is\_deleted | | -- | ------- | ---------------- | ---------------- | ---------------- | ---------------- | | 1 | pending | 2024-01-01 11:00 | 2024-01-01 11:30 | 2024-01-01 11:00 | False | | 1 | shipped | 2024-01-01 11:30 | 2024-01-01 11:40 | 2024-01-01 11:30 | False | | 1 | deleted | 2024-01-01 11:40 | | 2024-01-01 11:40 | True | #### FAQs How do I run one snapshot at a time? To run one snapshot, use the `--select` flag, followed by the name of the snapshot: ```shell $ dbt snapshot --select order_snapshot ``` Check out the [model selection syntax documentation](https://docs.getdbt.com/reference/node-selection/syntax.md) for more operators and examples. How often should I run the snapshot command? Snapshots are a batch-based approach to [change data capture](https://en.wikipedia.org/wiki/Change_data_capture). The `dbt snapshot` command must be run on a schedule to ensure that changes to tables are actually recorded! While individual use-cases may vary, snapshots are intended to be run between hourly and daily. If you find yourself snapshotting more frequently than that, consider if there isn't a more appropriate way to capture changes in your source data tables. What happens if I add new columns to my snapshot query? When the columns of your source query changes, dbt will attempt to reconcile this change in the destination snapshot table. dbt does this by: 1. Creating new columns from the source query in the destination table 2. Expanding the size of string types where necessary (eg. `varchar`s on Redshift) dbt *will not* delete columns in the destination snapshot table if they are removed from the source query. It will also not change the type of a column beyond expanding the size of varchar columns. That is, if a `string` column is changed to a `date` column in the snapshot source query, dbt will not attempt to change the type of the column in the destination table. Do hooks run with snapshots? Yes! The following hooks are available for snapshots: * [pre-hooks](https://docs.getdbt.com/reference/resource-configs/pre-hook-post-hook.md) * [post-hooks](https://docs.getdbt.com/reference/resource-configs/pre-hook-post-hook.md) * [on-run-start](https://docs.getdbt.com/reference/project-configs/on-run-start-on-run-end.md) * [on-run-end](https://docs.getdbt.com/reference/project-configs/on-run-start-on-run-end.md) Can I store my snapshots in a directory other than the \`snapshot\` directory in my project? By default, dbt expects your snapshot files to be located in the `snapshots` subdirectory of your project. To change this, update the [snapshot-paths](https://docs.getdbt.com/reference/project-configs/snapshot-paths.md) configuration in your `dbt_project.yml` file, like so: dbt\_project.yml ```yml snapshot-paths: ["snapshots"] ``` Note that you cannot co-locate snapshots and models in the same directory. Debug Snapshot target is not a snapshot table errors If you see the following error when you try executing the snapshot command: > Snapshot target is not a snapshot table (missing `dbt_scd_id`, `dbt_valid_from`, `dbt_valid_to`) Double check that you haven't inadvertently caused your snapshot to behave like table materializations by setting its `materialized` config to be `table`. Prior to dbt version 1.4, it was possible to have a snapshot like this: ```sql {% snapshot snappy %} {{ config(materialized = 'table', ...) }} ... {% endsnapshot %} ``` dbt is treating snapshots like tables (issuing `create or replace table ...` statements) **silently** instead of actually snapshotting data (SCD2 via `insert` / `merge` statements). When upgrading to dbt versions 1.4 and higher, dbt now raises a Parsing Error (instead of silently treating snapshots like tables) that reads: ```text A snapshot must have a materialized value of 'snapshot' ``` This tells you to change your `materialized` config to `snapshot`. But when you make that change, you might encounter an error message saying that certain fields like `dbt_scd_id` are missing. This error happens because, previously, when dbt treated snapshots as tables, it didn't include the necessary [snapshot meta-fields](https://docs.getdbt.com/docs/build/snapshots.md#snapshot-meta-fields) in your target table. Since those meta-fields don't exist, dbt correctly identifies that you're trying to create a snapshot in a table that isn't actually a snapshot. When this happens, you have to start from scratch — re-snapshotting your source data as if it was the first time by dropping your "snapshot" which isn't a real snapshot table. Then dbt snapshot will create a new snapshot and insert the snapshot meta-fields as expected. --- ### Add sources to your DAG #### Related reference docs * [Source properties](https://docs.getdbt.com/reference/source-properties.md) * [Source configurations](https://docs.getdbt.com/reference/source-configs.md) * [`{{ source() }}` Jinja function](https://docs.getdbt.com/reference/dbt-jinja-functions/source.md) * [`source freshness` command](https://docs.getdbt.com/reference/commands/source.md) #### Using sources Sources make it possible to name and describe the data loaded into your warehouse by your Extract and Load tools. By declaring these tables as sources in dbt, you can then * select from source tables in your models using the [`{{ source() }}` function,](https://docs.getdbt.com/reference/dbt-jinja-functions/source.md) helping define the lineage of your data * test your assumptions about your source data * calculate the freshness of your source data ##### Declaring a source Sources are defined in `.yml` files nested under a `sources:` key. models/\<filename>.yml ```yaml sources: - name: jaffle_shop database: raw schema: jaffle_shop tables: - name: orders - name: customers - name: stripe tables: - name: payments ``` \*By default, `schema` will be the same as `name`. Add `schema` only if you want to use a source name that differs from the existing schema. If you're not already familiar with these files, be sure to check out [the documentation on properties.yml files](https://docs.getdbt.com/reference/configs-and-properties.md) before proceeding. ##### Selecting from a source Once a source has been defined, it can be referenced from a model using the [`{{ source()}}` function](https://docs.getdbt.com/reference/dbt-jinja-functions/source.md). models/orders.sql ```sql select ... from {{ source('jaffle_shop', 'orders') }} left join {{ source('jaffle_shop', 'customers') }} using (customer_id) ``` dbt will compile this to the full table name: target/compiled/jaffle\_shop/models/my\_model.sql ```sql select ... from raw.jaffle_shop.orders left join raw.jaffle_shop.customers using (customer_id) ``` Using the `{{ source () }}` function also creates a dependency between the model and the source table. [![The source function tells dbt a model is dependent on a source ](/img/docs/building-a-dbt-project/sources-dag.png?v=2 "The source function tells dbt a model is dependent on a source ")](#)The source function tells dbt a model is dependent on a source ##### Testing and documenting sources You can also: * Add data tests to sources * Add descriptions to sources, that get rendered as part of your documentation site These should be familiar concepts if you've already added data tests and descriptions to your models (if not check out the guides on [testing](https://docs.getdbt.com/docs/build/data-tests.md) and [documentation](https://docs.getdbt.com/docs/build/documentation.md)). models/\<filename>.yml ```yaml sources: - name: jaffle_shop description: This is a replica of the Postgres database used by our app tables: - name: orders database: raw description: > One record per order. Includes cancelled and deleted orders. columns: - name: id description: Primary key of the orders table data_tests: - unique - not_null - name: status description: Note that the status can change over time - name: ... - name: ... ``` You can find more details on the available properties for sources in the [reference section](https://docs.getdbt.com/reference/source-properties.md). ##### FAQs What if my source is in a poorly named schema or table? By default, dbt will use the `name:` parameters to construct the source reference. If these names are a little less-than-perfect, use the [schema](https://docs.getdbt.com/reference/resource-properties/schema.md) and [identifier](https://docs.getdbt.com/reference/resource-properties/identifier.md) properties to define the names as per the database, and use your `name:` property for the name that makes sense! models/\<filename>.yml ```yml sources: - name: jaffle_shop database: raw schema: postgres_backend_public_schema tables: - name: orders identifier: api_orders ``` In a downstream model: ```sql select * from {{ source('jaffle_shop', 'orders') }} ``` Will get compiled to: ```sql select * from raw.postgres_backend_public_schema.api_orders ``` What if my source is in a different database to my target database? Use the [`database` property](https://docs.getdbt.com/reference/resource-properties/database.md) to define the database that the source is in. models/\<filename>.yml ```yml sources: - name: jaffle_shop database: raw schema: jaffle_shop tables: - name: orders - name: customers ``` I need to use quotes to select from my source, what should I do? This is reasonably common on Snowflake in particular. By default, dbt will not quote the database, schema, or identifier for the source tables that you've specified. To force dbt to quote one of these values, use the [`quoting` property](https://docs.getdbt.com/reference/resource-properties/quoting.md): models/\<filename>.yml ```yaml sources: - name: jaffle_shop database: raw schema: jaffle_shop quoting: database: true schema: true identifier: true tables: - name: order_items - name: orders # This overrides the `jaffle_shop` quoting config quoting: identifier: false ``` How do I run data tests on just my sources? To run data tests on all sources, use the following command: ```shell dbt test --select "source:*" ``` (You can also use the `-s` shorthand here instead of `--select`) To run data tests on one source (and all of its tables): ```shell $ dbt test --select source:jaffle_shop ``` And, to run data tests on one source table only: ```shell $ dbt test --select source:jaffle_shop.orders ``` How do I run models downstream of one source? To run models downstream of a source, use the `source:` selector: ```shell $ dbt run --select source:jaffle_shop+ ``` (You can also use the `-s` shorthand here instead of `--select`) To run models downstream of one source table: ```shell $ dbt run --select source:jaffle_shop.orders+ ``` Check out the [model selection syntax](https://docs.getdbt.com/reference/node-selection/syntax.md) for more examples! #### Source data freshness With a couple of extra configs, dbt can optionally capture the "freshness" of the data in your source tables. This is useful for understanding if your data pipelines are in a healthy state, and is a critical component of defining Service Level Agreements (SLAs) for your warehouse. ##### dbt v2 and dbt State State-aware orchestration is now dbt State [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) works with all engines and environments: dbt v1, dbt platform, and dbt v2 If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). If you're using dbt v2 with [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md), dbt automatically tracks source freshness using warehouse metadata. You don't need to configure `warn_after` or `error_after` for dbt to detect when source data changes. If you're using [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md), use [`lag_tolerance`](https://docs.getdbt.com/reference/resource-configs/lag-tolerance.md) to control how frequently models rebuild based on upstream data changes. You can also configure `loaded_at_field` or `loaded_at_query` on your source for more accurate freshness detection (for example, for streaming data or late-arriving records). However, you should still configure source freshness if you want to: * Receive SLA alerts when sources don't update within expected timeframes. * Define custom freshness logic using `loaded_at_field` or `loaded_at_query` (for example, for streaming data or partial loads). * Track freshness for source views. dbt v2 treats views as "always fresh" since it can't determine freshness from view metadata. ##### Declaring source freshness To configure source freshness information, add a `freshness` block to your source and `loaded_at_field` to your table declaration: models/\<filename>.yml ```yaml sources: - name: jaffle_shop database: raw config: freshness: # default freshness # changed to config in v1.9 warn_after: {count: 12, period: hour} error_after: {count: 24, period: hour} loaded_at_field: _etl_loaded_at # changed to config in v1.10 tables: - name: orders config: freshness: # make this a little more strict warn_after: {count: 6, period: hour} error_after: {count: 12, period: hour} - name: customers # this inherits the default freshness defined in the jaffle_shop source block at the beginning - name: product_skus config: freshness: null # do not check freshness for this table ``` In the `freshness` block, one or both of `warn_after` and `error_after` can be provided. If neither is provided, then dbt will not calculate freshness for the tables in this source. Additionally, the `loaded_at_field` is required to calculate freshness for a table (except for cases where dbt can leverage warehouse metadata to calculate freshness). If a `loaded_at_field`, or viable alternative, is not provided, then dbt will not calculate freshness for the table. These configs are applied hierarchically, so `freshness` and `loaded_at_field` values specified for a `source` will flow through to all of the `tables` defined in that source. This is useful when all of the tables in a source have the same `loaded_at_field`, as the config can just be specified once in the top-level source definition. ##### Checking source freshness To obtain freshness information for your sources, use the `dbt source freshness` command ([reference docs](https://docs.getdbt.com/reference/commands/source.md)): ```text $ dbt source freshness ``` Behind the scenes, dbt uses the freshness properties to construct a `select` query, shown below. You can find this query in the [query logs](https://docs.getdbt.com/faqs/Runs/checking-logs.md). ```sql select max(_etl_loaded_at) as max_loaded_at, convert_timezone('UTC', current_timestamp()) as calculated_at from raw.jaffle_shop.orders ``` The results of this query are used to determine whether the source is fresh or not: [![Uh oh! Not everything is as fresh as we'd like!](/img/docs/building-a-dbt-project/snapshot-freshness.png?v=2 "Uh oh! Not everything is as fresh as we'd like!")](#)Uh oh! Not everything is as fresh as we'd like! ##### Build models based on source freshness Our best practice recommendation is to use [data source freshness](https://docs.getdbt.com/docs/build/sources.md#declaring-source-freshness). This will allow settings to be transfered into a `.yml` file where source freshness is defined on [model level](https://docs.getdbt.com/reference/resource-properties/freshness.md). To build models based on source freshness in dbt: 1. Run `dbt source freshness` to check the freshness of your sources. 2. Use the `dbt build --select source_status:fresher+` command to build and test models downstream of fresher sources. Using these commands in order makes sure models update with the latest data. This eliminates wasted compute cycles on unchanged data and builds models *only* when necessary. Set [source freshness checks](https://docs.getdbt.com/docs/deploy/source-freshness.md#enabling-source-freshness-checks) to 30 minutes, then run a job which rebuilds every hour. This setup retrieves all the models and rebuilds them in one attempt if their source freshness has expired. For more information, refer to [Source freshness check frequency](https://docs.getdbt.com/docs/deploy/source-freshness.md#source-freshness-check-frequency). ##### Filter Some databases can have tables where a filter over certain columns are required, in order prevent a full scan of the table, which could be costly. In order to do a freshness check on such tables a `filter` argument can be added to the configuration, for example, `filter: _etl_loaded_at >= date_sub(current_date(), interval 1 day)`. For the example above, the resulting query would look like ```sql select max(_etl_loaded_at) as max_loaded_at, convert_timezone('UTC', current_timestamp()) as calculated_at from raw.jaffle_shop.orders where _etl_loaded_at >= date_sub(current_date(), interval 1 day) ``` ##### FAQs How do I exclude a table from a freshness snapshot? Some tables in a data source may be updated infrequently. If you've set a `freshness` property at the source level, this table is likely to fail checks. To work around this, you can set the table's freshness to null (`freshness: null`) to "unset" the freshness for a particular table: models/\<filename>.yml ```yaml sources: - name: jaffle_shop database: raw schema: jaffle_shop config: freshness: warn_after: {count: 12, period: hour} error_after: {count: 24, period: hour} loaded_at_field: _etl_loaded_at tables: - name: orders - name: product_skus config: freshness: null # do not check freshness for this table ``` How do I snapshot freshness for one source only? Use the `--select` flag to snapshot freshness for specific sources. Eg: ```shell # Snapshot freshness for all Jaffle Shop tables: $ dbt source freshness --select source:jaffle_shop # Snapshot freshness for a particular source <Term id="table" />: $ dbt source freshness --select source:jaffle_shop.orders # Snapshot freshness for multiple particular source tables: $ dbt source freshness --select source:jaffle_shop.orders source:jaffle_shop.customers ``` See the [`source freshness` command reference](https://docs.getdbt.com/reference/commands/source.md) for more information. Are the results of freshness stored anywhere? Yes! The `dbt source freshness` command will output a pass/warning/error status for each table selected in the freshness snapshot. Additionally, dbt will write the freshness results to a file in the `target/` directory called `sources.json` by default. You can also override this destination, use the `-o` flag to the `dbt source freshness` command. After enabling source freshness within a job, configure [Artifacts](https://docs.getdbt.com/docs/deploy/artifacts.md) in your **Project Details** page, which you can find by selecting your account name on the left side menu in dbt and clicking **Account settings**. You can see the current status for source freshness by clicking **View Sources** in the job page. --- ### Administer the Semantic Layer dbt platform | Starter, Enterprise, Enterprise+ With the dbt Semantic Layer, you can centrally define business metrics, reduce code duplication and inconsistency, create self-service in downstream tools, and more. This topic shows you how to set up credentials and tokens so that other tools can query the Semantic Layer. #### Prerequisites * Have a dbt Starter, Enterprise, or Enterprise+ account. Available on all [tenant configurations](https://docs.getdbt.com/docs/platform/about-platform/tenancy.md). * Ensure your production and development environments are on a [supported dbt version](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md). * Use Snowflake, BigQuery, Databricks, Redshift, Postgres, or Trino. * Create a successful run in the environment where you configure the Semantic Layer. * **Note:** Semantic Layer supports querying in Deployment environments; development querying is coming soon. * Understand [MetricFlow's](https://docs.getdbt.com/docs/build/about-metricflow.md) key concepts powering the Semantic Layer. * Note that the Semantic Layer doesn't support using [Single sign-on (SSO)](https://docs.getdbt.com/docs/platform/manage-access/sso-overview.md) for [production credentials](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md#permissions-for-service-account-tokens), though SSO is supported for development user accounts. 📹 Learn about the dbt Semantic Layer with on-demand video courses! Explore our [dbt Semantic Layer on-demand course](https://learn.getdbt.com/courses/semantic-layer) to learn how to define and query metrics in your dbt project. Additionally, dive into mini-courses for querying the dbt Semantic Layer in your favorite tools: [Tableau](https://courses.getdbt.com/courses/tableau-querying-the-semantic-layer), [Excel](https://learn.getdbt.com/courses/querying-the-semantic-layer-with-excel), [Hex](https://courses.getdbt.com/courses/hex-querying-the-semantic-layer), and [Mode](https://courses.getdbt.com/courses/mode-querying-the-semantic-layer). #### Administer the Semantic Layer You must be part of the Owner group and have the correct [license](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md) and [permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) to administer the Semantic Layer at the environment and project level. * Enterprise+ and Enterprise plan: * Developer license with Account Admin permissions, or * Owner with a Developer license, assigned Project Creator, Database Admin, or Admin permissions. * Starter plan: Owner with a Developer license. * Free trial: You are on a free trial of the Starter plan as an Owner, which means you have access to the dbt Semantic Layer. ##### 1. Select environment Select the environment where you want to enable the Semantic Layer: 1. Navigate to **Account settings** in the navigation menu. 2. Under **Settings**, click **Projects** and select the specific project you want to enable the Semantic Layer for. 3. In the **Project details** page, navigate to the **Semantic Layer** section. Select **Configure Semantic Layer**. [![Semantic Layer section in the 'Project details' page](/img/docs/dbt-platform/semantic-layer/new-sl-configure.png?v=2 "Semantic Layer section in the 'Project details' page")](#)Semantic Layer section in the 'Project details' page 4. In the **Set Up Semantic Layer Configuration** page, select the deployment environment you want for the Semantic Layer and click **Save**. This provides administrators with the flexibility to choose the environment where the Semantic Layer will be enabled. [![Select the deployment environment to run your Semantic Layer against.](/img/docs/dbt-platform/semantic-layer/sl-select-env.png?v=2 "Select the deployment environment to run your Semantic Layer against.")](#)Select the deployment environment to run your Semantic Layer against. ##### 2. Configure credentials and create tokens There are two options for setting up Semantic Layer using API tokens: * [Add a credential and create service tokens](#add-a-credential-and-create-service-tokens) * [Configure user credentials and create personal tokens](#configure-development-credentials-and-create-a-personal-token) ###### Add a credential and create service tokens The first option is to use [service tokens](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) for authentication which are tied to an underlying data platform credential that you configure. The credential configured is used to execute queries that the Semantic Layer issues against your data platform. This credential controls the physical access to underlying data accessed by the Semantic Layer, and all access policies set in the data platform for this credential will be respected. | Feature | Starter plan | Enterprise+ and Enterprise plan | | --------------------------------------------------- | ------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Service tokens | Can create multiple service tokens linked to one credential. | Can use multiple credentials and link multiple service tokens to each credential. Note that you cannot link a single service token to more than one credential. | | Credentials per project | One credential per project. | Can [add multiple](#4-add-more-credentials) credentials per project. | | Link multiple service tokens to a single credential | ✅ | ✅ | *If you're on a Starter plan and need to add more credentials, consider upgrading to our [Enterprise+ or Enterprise plan](https://www.getdbt.com/contact). All Enterprise users can refer to [Add more credentials](#4-add-more-credentials) for detailed steps on adding multiple credentials.* ###### 1. Select deployment environment * After selecting the deployment environment, you should see the **Credentials & service tokens** page. * Click the **Add Semantic Layer credential** button. ###### 2. Configure credential * In the **1. Add credentials** section, enter the credentials specific to your data platform that you want the Semantic Layer to use. * Use credentials with minimal privileges. The Semantic Layer requires read access to the schema(s) containing the dbt models used in your semantic models for downstream applications * Use [Extended Attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) and [Environment Variables](https://docs.getdbt.com/docs/build/environment-variables.md) when connecting to the Semantic Layer. If you set a value directly in the Semantic Layer Credentials, it will have a higher priority than Extended Attributes. When using environment variables, the default value for the environment will be used. For example, set the warehouse by using `{{env_var('DBT_WAREHOUSE')}}` in your Semantic Layer credentials. Similarly, if you set the account value using `{{env_var('DBT_ACCOUNT')}}` in Extended Attributes, dbt will check both the Extended Attributes and the environment variable. [![Add credentials and map them to a service token. ](/img/docs/dbt-platform/semantic-layer/sl-add-credential.png?v=2 "Add credentials and map them to a service token. ")](#)Add credentials and map them to a service token. ###### 3. Create or link service tokens * If you have permission to create service tokens, you’ll see the [**Map new service token** option](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md#map-service-tokens-to-credentials) after adding the credential. Name the token, set permissions to 'Semantic Layer Only' and 'Metadata Only', and click **Save**. * Once the token is generated, you won't be able to view this token again, so make sure to record it somewhere safe. * If you don’t have access to create service tokens, you’ll see a message prompting you to contact your admin to create one for you. Admins can create and link tokens as needed. [![If you don’t have access to create service tokens, you can create a credential and contact your admin to create one for you.](/img/docs/dbt-platform/semantic-layer/sl-credential-no-service-token.png?v=2 "If you don’t have access to create service tokens, you can create a credential and contact your admin to create one for you.")](#)If you don’t have access to create service tokens, you can create a credential and contact your admin to create one for you. info * Starter plans can create multiple service tokens that link to a single underlying credential, but each project can only have one credential. * All Enterprise plans can [add multiple credentials](#4-add-more-credentials) and map those to service tokens for tailored access. [Book a free live demo](https://www.getdbt.com/contact) to discover the full potential of dbt Enterprise and higher plans. ###### Configure development credentials and create a personal token Using [personal access tokens (PATs)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) is also a supported authentication method for the dbt Semantic Layer. This enables user-level authentication, reducing the need for sharing tokens between users. When you authenticate using PATs, queries are run using your personal user credentials. To use PATs in Semantic Layer: 1. Configure your user credentials. 1. Click your account name at the bottom left-hand menu and go to **Account settings** > **Credentials**. 2. Select your project. 3. Click **Edit**. 4. Go to **User credentials** and enter your details. 5. Click **Save**. 2. [Create a personal access token](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md). Make sure to copy the token. You can use the generated PAT as the authentication method for Semantic Layer [APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md) and [integrations](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md). ##### 3. View connection detail 1. Go back to the **Project details** page for connection details to connect to downstream tools. 2. Copy and share the Environment ID, service or personal token, Host, as well as the service or personal token name to the relevant teams for BI connection setup. If your tool uses the GraphQL API, save the GraphQL API host information instead of the JDBC URL. For info on how to connect to other integrations, refer to [Available integrations](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md). [![After configuring, you'll be provided with the connection details to connect to you downstream tools.](/img/docs/dbt-platform/semantic-layer/sl-configure-example.png?v=2 "After configuring, you'll be provided with the connection details to connect to you downstream tools.")](#)After configuring, you'll be provided with the connection details to connect to you downstream tools. ##### 4. Add more credentials [Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") All dbt Enterprise plans can optionally add multiple credentials and map them to service tokens, offering more granular control and tailored access for different teams, which can then be shared to relevant teams for BI connection setup. These credentials control the physical access to underlying data accessed by the Semantic Layer. We recommend configuring credentials and service tokens to reflect your teams and their roles. For example, create tokens or credentials that align with your team's needs, such as providing access to finance-related schemas to the Finance team.  Considerations for linking credentials * Admins can link multiple service tokens to a single credential within a project, but each service token can only be linked to one credential per project. * When you send a request through the APIs, the service token of the linked credential will follow access policies of the underlying view and tables used to build your semantic layer requests. * Use [Extended Attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) and [Environment Variables](https://docs.getdbt.com/docs/build/environment-variables.md) when connecting to the Semantic Layer. If you set a value directly in the Semantic Layer Credentials, it will have a higher priority than Extended Attributes. When using environment variables, the default value for the environment will be used. For example, set the warehouse by using `{{env_var('DBT_WAREHOUSE')}}` in your Semantic Layer credentials. Similarly, if you set the account value using `{{env_var('DBT_ACCOUNT')}}` in Extended Attributes, dbt will check both the Extended Attributes and the environment variable. ###### 1. Add more credentials * After configuring your environment, on the **Credentials & service tokens** page, click the **Add Semantic Layer credential** button to create multiple credentials and map them to a service token. <br /> * In the **1. Add credentials** section, fill in the data platform's credential fields. We recommend using “read-only” credentials. [![Add credentials and map them to a service token. ](/img/docs/dbt-platform/semantic-layer/sl-add-credential.png?v=2 "Add credentials and map them to a service token. ")](#)Add credentials and map them to a service token. ###### 2. Map service tokens to credentials * In the **2. Map new service token** section, [map a service token to the credential](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md#map-service-tokens-to-credentials) you configured in the previous step. dbt automatically selects the service token permission set you need (Semantic Layer Only and Metadata Only). * To add another service token during configuration, click **Add Service Token**. * You can link more service tokens to the same credential later on in the **Semantic Layer Configuration Details** page. To add another service token to an existing Semantic Layer configuration, click **Add service token** under the **Linked service tokens** section. * Click **Save** to link the service token to the credential. Remember to copy and save the service token securely, as it won't be viewable again after generation. [![Use the configuration page to manage multiple credentials or link or unlink service tokens for more granular control.](/img/docs/dbt-platform/semantic-layer/sl-credentials-service-token.png?v=2 "Use the configuration page to manage multiple credentials or link or unlink service tokens for more granular control.")](#)Use the configuration page to manage multiple credentials or link or unlink service tokens for more granular control. ###### 3. Delete credentials * To delete a credential, go back to the **Credentials & service tokens** page. * Under **Linked Service Tokens**, click **Edit** and, select **Delete Credential** to remove a credential. When you delete a credential, any service tokens mapped to that credential in the project will no longer work and will break for any end users. ##### Delete configuration You can delete the entire Semantic Layer configuration for a project. Note that deleting the Semantic Layer configuration will remove all credentials and unlink all service tokens to the project. It will also cause all queries to the Semantic Layer to fail. Follow these steps to delete the Semantic Layer configuration for a project: 1. Navigate to the **Project details** page. 2. In the **Semantic Layer** section, select **Delete Semantic Layer**. 3. Confirm the deletion by clicking **Yes, delete semantic layer** in the confirmation pop up. To re-enable the dbt Semantic Layer setup in the future, you will need to recreate your setup configurations by following the [previous steps](#set-up-dbt-semantic-layer). If your semantic models and metrics are still in your project, no changes are needed. If you've removed them, you'll need to set up the YAML configs again. [![Delete the Semantic Layer configuration for a project.](/img/docs/dbt-platform/semantic-layer/sl-delete-config.png?v=2 "Delete the Semantic Layer configuration for a project.")](#)Delete the Semantic Layer configuration for a project. #### Additional configuration The following are the additional flexible configurations for Semantic Layer credentials. ##### Map service tokens to credentials * After configuring your environment, you can map additional service tokens to the same credential if you have the required [permissions](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#permission-sets). * Go to the **Credentials & service tokens** page and click the **+Add Service Token** button in the **Linked Service Tokens** section. * Type the service token name and select the permission set you need (Semantic Layer Only and Metadata Only). * Click **Save** to link the service token to the credential. * Remember to copy and save the service token securely, as it won't be viewable again after generation. [![Map additional service tokens to a credential.](/img/docs/dbt-platform/semantic-layer/sl-add-service-token.gif?v=2 "Map additional service tokens to a credential.")](#)Map additional service tokens to a credential. ##### Unlink service tokens * Unlink a service token from the credential by clicking **Unlink** under the **Linked service tokens** section. If you try to query the Semantic Layer with an unlinked credential, you'll experience an error in your BI tool because no valid token is mapped. ##### Manage from service token page **View credential from service token** * View your Semantic Layer credential directly by navigating to the **API tokens** and then **Service tokens** page. * Select the service token to view the credential it's linked to. This is useful if you want to know which service tokens are mapped to credentials in your project. ###### Create a new service token * From the **Service tokens** page, create a new service token and map it to the credential(s) (assuming the semantic layer permission exists). This is useful if you want to create a new service token and directly map it to a credential in your project. * Make sure to select the correct permission set for the service token (Semantic Layer Only and Metadata Only). [![Create a new service token and map credentials directly on the separate 'Service tokens page'.](/img/docs/dbt-platform/semantic-layer/sl-create-service-token-page.png?v=2 "Create a new service token and map credentials directly on the separate 'Service tokens page'.")](#)Create a new service token and map credentials directly on the separate 'Service tokens page'. #### Next steps * If you haven't created metrics yet, follow the [Quickstart for the Semantic Layer](https://docs.getdbt.com/guides/sl-qs.md). * Now that you've set up your credentials and tokens, start querying your metrics with the [available integrations](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md). * [Optimize querying performance](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-cache.md) using declarative caching. * [Validate semantic nodes in CI](https://docs.getdbt.com/docs/deploy/ci-jobs.md#semantic-validations-in-ci) to ensure code changes made to dbt models don't break these metrics. * If you haven't already, learn how to [build you metrics and semantic models](https://docs.getdbt.com/docs/build/build-metrics-intro.md) in your development tool of choice. * Learn about commonly asked [Semantic Layer FAQs](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-faqs.md). #### FAQs  How does caching interact with access controls? Cached data is stored separately from the underlying models. If metrics are pulled from the cache, we don’t have the security context applied to those tables at query time. In the future, we plan to clone credentials, identify the minimum access level needed, and apply those permissions to cached tables. --- ### Advanced CI dbt platform | Enterprise, Enterprise+ [Continuous integration workflows](https://docs.getdbt.com/docs/deploy/continuous-integration.md) help increase the governance and improve the quality of the data. Additionally for these CI jobs, you can use Advanced CI features, such as [compare changes](#compare-changes), that provide details about the changes between what's currently in your production environment and the pull request's latest commit, giving you observability into how data changes are affected by your code changes. By analyzing the data changes that code changes produce, you can ensure you're always shipping trustworthy data products as you're developing. How to enable this feature You can opt into Advanced CI in dbt. Please refer to [Account access to Advanced CI features](https://docs.getdbt.com/docs/platform/account-settings.md#account-access-to-advanced-ci-features) to learn how enable it in your dbt account. #### Prerequisites * You have a dbt Enterprise or Enterprise+ account. * You have [Advanced CI features](https://docs.getdbt.com/docs/platform/account-settings.md#account-access-to-advanced-features) enabled. * You use a supported data platform: BigQuery, Databricks, Postgres, Redshift, or Snowflake. Support for additional data platforms coming soon. #### Compare changes feature For [CI jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md) that have the [**dbt compare** option enabled](https://docs.getdbt.com/docs/deploy/ci-jobs.md#set-up-ci-jobs), dbt compares the changes between the last applied state of the production environment (defaulting to deferral for lower compute costs) and the latest changes from the pull request, whenever a pull request is opened or new commits are pushed. You can also compare changes in development. For more details, see [Compare changes in development](https://docs.getdbt.com/docs/deploy/advanced-ci.md#compare-changes-in-development). dbt reports the comparison differences in: * **dbt** — Shows the changes (if any) to the data's primary keys, rows, and columns in the [Compare tab](https://docs.getdbt.com/docs/deploy/run-visibility.md#compare-tab) from the [Job run details](https://docs.getdbt.com/docs/deploy/run-visibility.md#job-run-details) page. * **The pull request from your Git provider** — Shows a summary of the changes as a Git comment. [![Example of the Compare tab](/img/docs/dbt-platform/example-ci-compare-changes-tab.png?v=2 "Example of the Compare tab")](#)Example of the Compare tab [![Example of Modified rows tab which shows you details of the modified rows.](/img/docs/dbt-platform/advanced-ci-modified-rows.png?v=2 "Example of Modified rows tab which shows you details of the modified rows.")](#)Example of Modified rows tab which shows you details of the modified rows. ###### Compare changes in development You can compare changes locally in development to preview data changes caused by your local edits (for example, added/removed rows or joins) directly in your editor, and without waiting on CI. Compare changes in development is available through the dbt VS Code extension, which is powered by dbt v2. For more details on how to use this feature, see [Compare changes in local development](https://docs.getdbt.com/docs/dbt/vs-compare-changes.md).  Differences between compare changes in development and Advanced CI compare changes | Aspect | In development (compare changes) | In deployment (Advanced CI) | | ------------------- | ----------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | | **Affects** | Development for one modified model at a time | Deployment for all modified models in a project | | **Trigger** | On-demand in editor | PR open/update and CI job | | **Scope** | Your working copy and local target | Branch head versus prod state in CI | | **Output** location | Compare panel in VS Code/Cursor. Does not create a PR comment in Git provider | Deployment job compare tab and PR summary comment in Git provider | | **Data caching** | Editor-side | dbt platform [caches](https://docs.getdbt.com/docs/deploy/advanced-ci.md#about-the-cached-data) limited samples | | **Governance** | Local development credentials | Production credentials | ##### Optimizing comparisons When an [`event_time`](https://docs.getdbt.com/reference/resource-configs/event-time.md) column is specified on your model, compare changes can optimize comparisons by using only the overlapping timeframe (meaning the timeframe exists in both the CI and production environment), helping you avoid incorrect row-count changes and return results faster. This is useful in scenarios like: * **Subset of data in CI** — When CI builds only a [subset of data](https://docs.getdbt.com/best-practices/best-practice-workflows.md#limit-the-data-processed-when-in-development) (like the most recent 7 days), compare changes would interpret the excluded data as "deleted rows." Configuring `event_time` allows you to avoid this issue by limiting comparisons to the overlapping timeframe, preventing false alerts about data deletions that are just filtered out in CI. * **Fresher data in CI than in production** — When your CI job includes fresher data than production (because it has run more recently), compare changes would flag the additional rows as "new" data, even though they’re just fresher data in CI. With `event_time` configured, the comparison only includes the shared timeframe and correctly reflects actual changes in the data. [![event\_time ensures the same time-slice of data is accurately compared between your CI and production environments.](/img/docs/deploy/apples_to_apples.png?v=2 "event_time ensures the same time-slice of data is accurately compared between your CI and production environments.")](#)event\_time ensures the same time-slice of data is accurately compared between your CI and production environments. #### About the cached data After [comparing changes](#compare-changes), dbt stores a cache of no more than 100 records for each modified model for preview purposes. By caching this data, you can view the examples of changed data without rerunning the comparison against the data warehouse every time (optimizing for lower compute costs). To display the changes, dbt uses a cached version of a sample of the data records. These data records are queried from the database using the connection configuration (such as user, role, service account, and so on) that's set in the CI job's environment. You control what data to use. This may include synthetic data if pre-production or development data is heavily regulated or sensitive. * The selected data is cached on dbt Labs' systems for up to 30 days. No data is retained on dbt Labs' systems beyond this period. * The cache is encrypted and stored in an Amazon S3 or Azure blob storage in your account’s region. * dbt Labs will not access cached data from Advanced CI for its benefit and the data is only used to provide services as directed by you. * Third-party subcontractors, other than storage subcontractors, will not have access to the cached data. If you access a CI job run that's more than 30 days old, you will not be able to see the comparison results. Instead, a message will appear indicating that the data has expired. [![Example of message about expired data in the Compare tab](/img/docs/deploy/compare-expired.png?v=2 "Example of message about expired data in the Compare tab")](#)Example of message about expired data in the Compare tab #### Connection permissions The compare changes feature uses the same credentials as the CI job, as defined in the CI job’s environment. The dbt administrator must ensure that client CI credentials are appropriately restricted since all customer's account users will be able to view the comparison results and the cached data. If using dynamic data masking in the data warehouse, the cached data will no longer be dynamically masked in the Advanced CI output, depending on the permissions of the users who view it. dbt Labs recommends limiting user access to unmasked data or considering using synthetic data for the Advanced CI testing functionality. [![Example of credentials in the user settings](/img/docs/deploy/compare-credentials.png?v=2 "Example of credentials in the user settings")](#)Example of credentials in the user settings #### Troubleshooting  Compare changes CI models need to be on same database host/connection Compare Changes only works if both CI and production models live on the same database host/connection. Compare Changes runs SQL queries in the current CI job’s environment to compare the CI model (like `ci.dbt_cloud_123.foo`) to the production model (`prod.analytics.foo`). If the CI job defers to a production job that's on a different database connection or host, then the compare changes feature will not work as expected. This is because the CI environment can't access or query production objects on another host. In the following example, the CI job can’t access the production model to compare them because they’re on different database hosts: * The dbt CI job in environment `ci.dbt_cloud_123.foo` that connects to host `abc123.rds.amazonaws.com` * The dbt production job in environment `prod.analytics.foo` that connects to host `def456.rds.amazonaws.com` --- ### Advanced data modeling This section covers advanced topics for the Semantic Layer and MetricFlow, such as data modeling workflows, and more. [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/build/fill-nulls-advanced.md) ###### [Fill null values for metrics](https://docs.getdbt.com/docs/build/fill-nulls-advanced.md) [Use fill\_nulls\_with to set null metric values to zero, ensuring numeric values for every data row.](https://docs.getdbt.com/docs/build/fill-nulls-advanced.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/build/ref-metrics-in-filters.md) ###### [Metrics as dimensions with metric filters](https://docs.getdbt.com/docs/build/ref-metrics-in-filters.md) [Add metrics as dimensions to your metric filters to create more complex metrics and gain more insights.](https://docs.getdbt.com/docs/build/ref-metrics-in-filters.md) --- ### Analyses #### Overview dbt's notion of `models` makes it easy for data teams to version control and collaborate on data transformations. Sometimes though, a certain SQL statement doesn't quite fit into the mold of a dbt model. These more "analytical" SQL files can be versioned inside of your dbt project using the `analysis` functionality of dbt. Any `.sql` files found in the `analyses/` directory of a dbt project will be compiled, but not executed. This means that analysts can use dbt functionality like `{{ ref(...) }}` to select from models in an environment-agnostic way. In practice, an analysis file might look like this (via the [open source Quickbooks models](https://github.com/dbt-labs/quickbooks)): analyses/running\_total\_by\_account.sql ```sql -- analyses/running_total_by_account.sql with journal_entries as ( select * from {{ ref('quickbooks_adjusted_journal_entries') }} ), accounts as ( select * from {{ ref('quickbooks_accounts_transformed') }} ) select txn_date, account_id, adjusted_amount, description, account_name, sum(adjusted_amount) over (partition by account_id order by id rows unbounded preceding) from journal_entries order by account_id, id ``` You can configure individual analyses in their property file. For example: analyses/\<filename>.yml ```yaml analyses: - name: running_total_by_account description: Calculates the running total of adjusted amounts per account over time. config: enabled: true docs: show: true node_color: purple tags: ['finance', 'accounting'] columns: - name: account_id description: The unique identifier for the account. - name: adjusted_amount description: The adjusted transaction amount for the account. ``` For more information, refer to [Analysis properties](https://docs.getdbt.com/reference/analysis-properties.md). (Applies to dbt v1.12 and later) #### Project-level configuration You can also configure analyses at the project level in `dbt_project.yml`. To configure analyses at the project level, set the [`require_corrected_analysis_fqns`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_corrected_analysis_fqns.md) flag to `true` and add an `analyses` block in your `dbt_project.yml`. The project-level configuration applies to existing analyses in the `analyses/` folder — for example, setting `+enabled: false` disables them all. dbt\_project.yml ```yaml flags: require_corrected_analysis_fqns: true analyses: +enabled: true | false ``` #### Compiling analyses To compile this analysis into runnable sql, run: ```text dbt compile ``` Then, look for the compiled SQL file in `target/compiled/{project name}/analyses/running_total_by_account.sql`. This SQL can then be pasted into a data visualization tool, for instance. Note that no `running_total_by_account` relation will be materialized in the database as this is an `analysis`, not a `model`. --- ### Apache Iceberg support Apache Iceberg is an open table format that brings greater portability and interoperability to the data ecosystem. By standardizing how data is stored and accessed, Iceberg enables teams to use the same data across multiple engines and platforms, without replication. There are multiple layers of Iceberg support, in data platform and in dbt: * **Iceberg table format** — An open-source table format. Iceberg tables are a combination of data files in object storage (such as Parquet files in an S3 bucket), as well as metadata files (recording the table's schema, versioning, and more) that also live in object storage. * **Iceberg data catalog** — An open-source specification for a metadata system that tracks the schema, partition, and versions of multiple Iceberg tables. * **Iceberg REST protocol** (also referred to as the Iceberg REST API) — Defines standard endpoints for interacting with Iceberg-compatible catalogs. In theory, the Iceberg standard enables one query engine to read from or write to Iceberg tables in an external catalog, managed by another engine or platform. In practice, different Iceberg catalogs work differently, and different query engines support the Iceberg spec to varying degrees. To the extent possible, dbt tries to abstract away the complexity of table formats, and the divergence among vendor-specific Iceberg implementations, so teams can focus on delivering reliable, well-modeled data. To learn more, select one of the following tiles: [![](/img/icons/dbt-icon.svg)](https://docs.getdbt.com/docs/build/iceberg/about-catalogs.md) ###### [Iceberg catalogs](https://docs.getdbt.com/docs/build/iceberg/about-catalogs.md) [About Iceberg catalogs](https://docs.getdbt.com/docs/build/iceberg/about-catalogs.md) [![](/img/icons/dbt-icon.svg)](https://docs.getdbt.com/docs/build/iceberg/catalogs-yml.md) ###### [Using table\_format + catalogs.yml](https://docs.getdbt.com/docs/build/iceberg/catalogs-yml.md) [dbt support for Iceberg + catalogs](https://docs.getdbt.com/docs/build/iceberg/catalogs-yml.md) [![](/img/icons/snowflake.svg)](https://docs.getdbt.com/docs/build/iceberg/adapters/snowflake-iceberg-support.md) ###### [Snowflake + Iceberg](https://docs.getdbt.com/docs/build/iceberg/adapters/snowflake-iceberg-support.md) [Snowflake Iceberg configurations](https://docs.getdbt.com/docs/build/iceberg/adapters/snowflake-iceberg-support.md) [![](/img/icons/bigquery.svg)](https://docs.getdbt.com/docs/build/iceberg/adapters/bigquery-iceberg-support.md) ###### [BigQuery + Iceberg](https://docs.getdbt.com/docs/build/iceberg/adapters/bigquery-iceberg-support.md) [BigQuery Iceberg configurations](https://docs.getdbt.com/docs/build/iceberg/adapters/bigquery-iceberg-support.md) [![](/img/icons/databricks.svg)](https://docs.getdbt.com/docs/build/iceberg/adapters/databricks-iceberg-support.md) ###### [Databricks + Iceberg](https://docs.getdbt.com/docs/build/iceberg/adapters/databricks-iceberg-support.md) [Databricks Iceberg configurations](https://docs.getdbt.com/docs/build/iceberg/adapters/databricks-iceberg-support.md) [![](/img/icons/duckdb-seeklogo.svg)](https://docs.getdbt.com/docs/build/iceberg/adapters/duckdb-iceberg-support.md) ###### [DuckDB + Iceberg](https://docs.getdbt.com/docs/build/iceberg/adapters/duckdb-iceberg-support.md) [DuckDB Iceberg configurations](https://docs.getdbt.com/docs/build/iceberg/adapters/duckdb-iceberg-support.md) --- ### Apache Ossie semantic layer documents 💡Did you know... Available from dbt v1.12 or with the [dbt "v1 Latest" release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). dbt v1.12 and higher supports the [Apache Ossie](https://github.com/apache/ossie) standard for defining semantic models and metrics. You can place Ossie-format `.json` files in an `osi/` directory at the root of your project, or configure [`osi-paths`](https://docs.getdbt.com/reference/project-configs/osi-paths.md) in `dbt_project.yml` to use one or more custom directories relative to your project root. dbt parses them into the manifest alongside any native dbt semantic models. Ossie-sourced definitions and native dbt semantic models can coexist in the same project. #### Prerequisites * You must be on dbt v1.12 or higher. * Ossie documents must use version `0.1.0` or `0.1.1`. Any other version string raises a parse error. #### Defining semantic models using Ossie documents To define semantic models with Ossie documents: 1. Create an `osi/` directory at the root of your dbt project, at the same level as `dbt_project.yml`. To use one or more custom directories instead, configure [`osi-paths`](https://docs.getdbt.com/reference/project-configs/osi-paths.md) in `dbt_project.yml` with paths relative to your project root. 2. Add one or more Ossie `.json` files to the directory. You can organize files into subdirectories; dbt scans the entire directory tree. The `source` field must be the fully qualified warehouse location of a dbt model in this project, in the form `database.schema.alias` (for example, `my_database.my_schema.fct_orders`). dbt matches each dataset on database, schema, and model alias. Each dataset `source` must resolve to a dbt model. For restrictions on dataset sources, refer to [Limitations](#limitations). The following is an example Ossie document that defines a semantic model on a dbt model called `fct_orders`: ```json { "version": "0.1.1", "semantic_model": [ { "name": "orders", "datasets": [ { "name": "orders", "source": "my_database.my_schema.fct_orders" } ] } ] } ``` This example defines a semantic model only. To add metrics, include a `metrics` array on the semantic model per the [Ossie specification](https://github.com/apache/ossie). 3. Run any command that triggers compilation, such as `dbt compile` or `dbt run`. dbt automatically discovers and parses Ossie files. The resulting semantic models (and metrics, when defined in your Ossie documents) appear in [dbt artifacts](https://docs.getdbt.com/reference/artifacts/dbt-artifacts.md) in your `target/` directory, including [`manifest.json`](https://docs.getdbt.com/reference/artifacts/manifest-json.md), [`semantic_manifest.json`](https://docs.getdbt.com/reference/artifacts/sl-manifest.md), and [`osi_document.json`](https://docs.getdbt.com/reference/artifacts/sl-manifest.md#osi-document). #### Limitations * dbt scans only the root project's Ossie directories (configured through [`osi-paths`](https://docs.getdbt.com/reference/project-configs/osi-paths.md), default `osi/`). Ossie files in installed dependency packages are ignored. * Each Ossie dataset source must resolve to a dbt model. Ossie documents that reference sources, seeds, snapshots, or external tables are not supported. * If the Ossie converter encounters unsupported metric types or other constructs, those elements are dropped and dbt emits a warning (event code `I078`), but parsing continues. Warnings appear in the CLI and in `logs/dbt.log`; for more information, refer to [Events and logs](https://docs.getdbt.com/reference/events-logging.md). --- ### Artifacts dbt platform When running dbt jobs, dbt generates and saves *artifacts*. You can use these artifacts, like `manifest.json`, `catalog.json`, and `sources.json` to power different aspects of the dbt platform, namely: [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md), [dbt Docs](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md#dbt-docs), and [source freshness reporting](https://docs.getdbt.com/docs/build/sources.md#source-data-freshness). #### Create dbt Artifacts [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md#generate-metadata) uses the metadata provided by the [Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md) to display the details about [the state of your project](https://docs.getdbt.com/docs/dbt-apis/project-state.md). It uses metadata from your staging and production [deployment environments](https://docs.getdbt.com/docs/deploy/deploy-environments.md). Catalog automatically retrieves the metadata updates after each job run in the production or staging deployment environment so it always has the latest results for your project — meaning it's always automatically updated after each job run. To view a resource, its metadata, and what commands are needed, refer to [generate metadata](https://docs.getdbt.com/docs/explore/explore-projects.md#generate-metadata) for more details.  For dbt Docs The following steps are for legacy dbt Docs only. For the current documentation experience, see [dbt Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md). While running any job can produce artifacts, you should only associate one production job with a given project to produce the project's artifacts. You can designate this connection on the **Project details** page. To access this page: 1. From the dbt platform, click on your account name in the left side menu and select **Account settings**. 2. Select your project, and click **Edit** in the lower right. 3. Under **Artifacts**, select the jobs you want to produce documentation and source freshness artifacts for. [![Configuring Artifacts](/img/docs/dbt-platform/using-dbt-platform/project-level-artifact-updated.png?v=2 "Configuring Artifacts")](#)Configuring Artifacts If you don't see your job listed, you might need to edit the job and select **Run source freshness** and **Generate docs on run**. [![Editing the job to generate artifacts](/img/docs/dbt-platform/using-dbt-platform/edit-job-generate-artifacts.png?v=2 "Editing the job to generate artifacts")](#)Editing the job to generate artifacts When you add a production job to a project, dbt updates the content and provides links to the production documentation and source freshness artifacts it generated for that project. You can see these links by clicking **Deploy** in the upper left, selecting **Jobs**, and then selecting the production job. From the job page, you can select a specific run to see how artifacts were updated for that run only. ##### Documentation Navigate to [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) through the **Explore** link to view your project's resources and lineage to gain a better understanding of its latest production state. To view a resource, its metadata, and what commands are needed, refer to [generate metadata](https://docs.getdbt.com/docs/explore/explore-projects.md#generate-metadata) for more details. Both the job's commands and the docs generate step (triggered by the **Generate docs on run** checkbox) must succeed during the job invocation to update the documentation.  For dbt Docs When set up, dbt updates the Documentation link in the header tab so it links to documentation for this job. This link always directs you to the latest version of the documentation for your project. ##### Source Freshness To view the latest source freshness result, refer to [generate metadata](https://docs.getdbt.com/docs/explore/explore-projects.md#generate-metadata) for more detail. Then navigate to Catalog through the **Explore** link.  For dbt Docs Configuring a job for the Source Freshness artifact setting also updates the data source link under **Orchestration** > **Data sources**. The link points to the latest Source Freshness report for the selected job. --- ### Available dbt versions Whether you're using the CLI or working within the dbt platform, your environments are aligned with a versioned release of dbt. [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/dbt-versions.md) ###### [About dbt versions](https://docs.getdbt.com/docs/dbt-versions.md) [Learn about semantic versioning for dbt engines, and how long those versions are supported.](https://docs.getdbt.com/docs/dbt-versions.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/dbt-versions/dbt-version-compatibility.md) ###### [dbt v2 version compatibility](https://docs.getdbt.com/docs/dbt-versions/dbt-version-compatibility.md) [Learn about version compatibility between v2, LSP, and VS Code extension.](https://docs.getdbt.com/docs/dbt-versions/dbt-version-compatibility.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) ###### [About release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) [Learn about how versions of dbt align with the release tracks available on the dbt platform.](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) ###### [Upgrade versions in dbt platform](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) [Instructions for upgrading your dbt platform projects to the latest version of dbt, including dbt v2.](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles.md) ###### [Product lifecycles](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles.md) [Learn about the dbt product lifecycles from beta through end of life.](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/dbt-versions/experimental-features.md) ###### [Preview new dbt platform features](https://docs.getdbt.com/docs/dbt-versions/experimental-features.md) [Learn how to enable self-service beta and preview features for your dbt platform account.](https://docs.getdbt.com/docs/dbt-versions/experimental-features.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade.md) ###### [dbt version upgrade guides](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade.md) [All the information you need to prepare your projects for the next version of dbt, including dbt v2. Includes guidance on new features, behavior changes, deprecations, and much more.](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade.md) <br /> --- ### Available integrations dbt platform | Starter, Enterprise, Enterprise+ There are a number of data applications that seamlessly integrate with the Semantic Layer, powered by MetricFlow, from business intelligence tools to notebooks, spreadsheets, data catalogs, and more. These integrations allow you to query and unlock valuable insights from your data ecosystem. Use the [Semantic Layer APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md) to simplify metric queries, optimize your development workflow, and reduce coding. This approach also ensures data governance and consistency for data consumers. The following tools integrate with the dbt Semantic Layer: [![](/img/icons/pbi.svg)](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/power-bi.md) ###### [Power BI](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/power-bi.md) [Use reports to query the dbt Semantic Layer with Power BI and produce dashboards with trusted data.](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/power-bi.md) [![](/img/icons/tableau-software.svg)](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/tableau.md) ###### [Tableau](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/tableau.md) [Learn how to connect to Tableau for querying metrics and collaborating with your team.](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/tableau.md) [![](/img/icons/google-sheets-logo-icon.svg)](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/gsheets.md) ###### [Google Sheets](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/gsheets.md) [Discover how to connect to Google Sheets for querying metrics and collaborating with your team.](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/gsheets.md) [![](/img/icons/excel.svg)](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/excel.md) ###### [Microsoft Excel](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/excel.md) [Connect to Microsoft Excel to query metrics and collaborate with your team. Available for Excel Desktop or Excel Online.](https://docs.getdbt.com/docs/platform-integrations/semantic-layer/excel.md) [![](/img/icons/omni.svg)](https://docs.omni.co/integrations/dbt/semantic-layer) ###### [Omni](https://docs.omni.co/integrations/dbt/semantic-layer) [Connect Omni to the dbt Semantic Layer to query trusted metrics directly within your Omni data model.](https://docs.omni.co/integrations/dbt/semantic-layer) [](https://docs.omni.co/integrations/dbt/semantic-layer) [![](/img/icons/dot-ai.svg)](https://docs.getdot.ai/integrations/semantic-layers/dbt-semantic-layer) ###### [Dot](https://docs.getdot.ai/integrations/semantic-layers/dbt-semantic-layer) [Enable everyone to analyze data with AI in Slack or Teams.](https://docs.getdot.ai/integrations/semantic-layers/dbt-semantic-layer) [](https://docs.getdot.ai/integrations/semantic-layers/dbt-semantic-layer) [![](/img/icons/hex.svg)](https://learn.hex.tech/docs/connect-to-data/data-connections/dbt-integration#dbt-semantic-layer-integration) ###### [Hex](https://learn.hex.tech/docs/connect-to-data/data-connections/dbt-integration#dbt-semantic-layer-integration) [Check out how to connect, analyze metrics, collaborate, and discover more data possibilities.](https://learn.hex.tech/docs/connect-to-data/data-connections/dbt-integration#dbt-semantic-layer-integration) [](https://learn.hex.tech/docs/connect-to-data/data-connections/dbt-integration#dbt-semantic-layer-integration) [![](/img/icons/klipfolio.svg)](https://support.klipfolio.com/hc/en-us/articles/18164546900759-PowerMetrics-Adding-dbt-Semantic-Layer-metrics) ###### [Klipfolio PowerMetrics](https://support.klipfolio.com/hc/en-us/articles/18164546900759-PowerMetrics-Adding-dbt-Semantic-Layer-metrics) [Learn how to connect to a streamlined metrics catalog and deliver metric-centric analytics to business users.](https://support.klipfolio.com/hc/en-us/articles/18164546900759-PowerMetrics-Adding-dbt-Semantic-Layer-metrics) [](https://support.klipfolio.com/hc/en-us/articles/18164546900759-PowerMetrics-Adding-dbt-Semantic-Layer-metrics) [![](/img/icons/mode.svg)](https://mode.com/help/articles/supported-databases#dbt-semantic-layer) ###### [Mode](https://mode.com/help/articles/supported-databases#dbt-semantic-layer) [Discover how to connect, access, and get trustworthy metrics and insights.](https://mode.com/help/articles/supported-databases#dbt-semantic-layer) [](https://mode.com/help/articles/supported-databases#dbt-semantic-layer) [![](/img/icons/push.svg)](https://docs.push.ai/data-sources/semantic-layers/dbt) ###### [Push.ai](https://docs.push.ai/data-sources/semantic-layers/dbt) [Explore how to connect and use metrics to power reports and insights that drive change.](https://docs.push.ai/data-sources/semantic-layers/dbt) [](https://docs.push.ai/data-sources/semantic-layers/dbt) [![](/img/icons/sigma.svg)](https://help.sigmacomputing.com/docs/configure-a-dbt-semantic-layer-integration) ###### [Sigma (Preview)](https://help.sigmacomputing.com/docs/configure-a-dbt-semantic-layer-integration) [Connect Sigma to the dbt Semantic Layer to allow you to leverage your predefined dbt metrics in Sigma workbooks.](https://help.sigmacomputing.com/docs/configure-a-dbt-semantic-layer-integration) [](https://help.sigmacomputing.com/docs/configure-a-dbt-semantic-layer-integration) [![](/img/icons/steep.svg)](https://help.steep.app/integrations/dbt-cloud) ###### [Steep](https://help.steep.app/integrations/dbt-cloud) [Connect Steep to the dbt Semantic Layer for centralized, scalable analytics.](https://help.steep.app/integrations/dbt-cloud) [](https://help.steep.app/integrations/dbt-cloud) <br /> Before you connect to these tools, you'll need to first [set up the dbt Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md) and [generate a service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) to create **Semantic Layer Only** and **Metadata Only** permissions. ##### Custom integration * All BI tools can use [exports](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md) with the Semantic Layer, even if they don’t have a native integration. * [Consume metrics](https://docs.getdbt.com/docs/use-dbt-semantic-layer/consume-metrics.md) and develop custom integrations using different languages and tools, supported through [JDBC](https://docs.getdbt.com/docs/dbt-apis/sl-jdbc.md), ADBC, and [GraphQL](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md) APIs, and [Python SDK library](https://docs.getdbt.com/docs/dbt-apis/sl-python.md). For more info, check out [our examples on GitHub](https://github.com/dbt-labs/example-semantic-layer-clients/). * Connect to any tool that supports SQL queries. These tools must meet one of the two criteria: * Offers a generic JDBC driver option (such as DataGrip) or * Is compatible Arrow Flight SQL JDBC driver version 12.0.0 or higher. #### Related docs * [Quickstart for the Semantic Layer](https://docs.getdbt.com/guides/sl-qs.md) * [dbt Semantic Layer APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview) to learn how to integrate and query your metrics in downstream tools. * [Semantic Layer API query syntax](https://docs.getdbt.com/docs/dbt-apis/sl-jdbc.md#querying-the-api-for-metric-metadata) * [Hex Semantic Layer cells](https://learn.hex.tech/docs/explore-data/cells/data-cells/dbt-metrics-cells) to set up SQL cells in Hex. * [Resolve 'Failed APN'](https://docs.getdbt.com/faqs/Troubleshooting/sl-alpn-error.md) error when connecting to the Semantic Layer. * [Semantic Layer on-demand course](https://learn.getdbt.com/courses/semantic-layer) * [Semantic Layer FAQs](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-faqs.md) --- ### Available tools The dbt MCP server has access to many parts of the dbt experience related to development, deployment, and discovery. This page lists the tools available in the dbt MCP server and how to configure them. #### Supported tools by MCP server type The following sections list every tool by category, along with which server type (local or remote) supports it. | Tools | Local | Remote | | ------------------------------------------------- | ----- | ------ | | [dbt commands](#dbt-commands) | ✅ | ❌ | | [Semantic Layer](#semantic-layer) | ✅ | ✅ | | [SQL](#sql) | ✅ | ✅ | | [Metadata Discovery](#discovery) | ✅ | ✅ | | [Administrative API](#admin-api) | ✅ | ✅ | | [Codegen Tools](#dbt-codegen) | ✅ | ❌ | | [dbt v2 Tools](#dbt-lsp) | ✅ | ✅ | | [Product Docs Tools](#product-docs) | ✅ | ✅ | | [MCP Server Metadata Tools](#mcp-server-metadata) | ✅ | ✅ | Note that access to the Discovery API and the Semantic Layer API is limited depending on your [plan type](https://www.getdbt.com/pricing). ##### Tools that require AI features `text_to_sql` is the only tool that depends on [AI features](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md) being enabled for your account. If an admin turns AI features off, `text_to_sql` doesn't appear in the tool list for the remote MCP server. Every other tool keeps working. Turning off AI features doesn't disable remote MCP. #### dbt MCP server tool list The following tool list is available for your MCP server and is auto-fetched from the [dbt MCP server README on GitHub](https://github.com/dbt-labs/dbt-mcp#tools) when the docs are built, so it stays in sync with each release. ##### SQL Tools for executing and generating SQL on dbt Platform infrastructure. * `execute_sql`: Executes SQL on dbt Platform infrastructure with Semantic Layer support. * `text_to_sql`: Generates SQL from natural language using project context. ##### Semantic Layer To learn more about the dbt Semantic Layer, click [here](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl). * `get_dimension_values`: Gets distinct values for a dimension; option to scope to specific metrics. * `get_dimensions`: Gets dimensions for specified metrics. * `get_entities`: Gets entities for specified metrics. * `get_metrics_compiled_sql`: Returns compiled SQL for metrics without executing the query. * `list_metrics`: Retrieves all defined metrics. * `list_saved_queries`: Retrieves all saved queries. * `query_metrics`: Executes metric queries with filtering and grouping options. ##### Discovery To learn more about the dbt Discovery API, click [here](https://docs.getdbt.com/docs/dbt-cloud-apis/discovery-api). * `get_all_macros`: Retrieves macros; option to filter by package or return package names only. * `get_all_models`: Retrieves name and description of all models. * `get_all_sources`: Gets all sources with freshness status; option to filter by source name. * `get_exposure_details`: *(deprecated — use `get_node_details` instead)* * `get_exposures`: Gets all exposures (downstream dashboards, apps, or analyses). * `get_lineage`: Gets full lineage graph (ancestors and descendants) with type and depth filtering. * `get_macro_details`: *(deprecated — use `get_node_details` instead)* * `get_mart_models`: Retrieves all mart models. * `get_model_children`: *(deprecated — use `get_lineage` instead)* * `get_model_details`: *(deprecated — use `get_node_details` instead)* * `get_model_health`: Gets health signals: run status, test results, and upstream source freshness. * `get_model_parents`: *(deprecated — use `get_lineage` instead)* * `get_model_performance`: Gets execution history for a model; option to include test results. * `get_node_details`: Gets full details for any dbt resource type (model, source, exposure, test, seed, snapshot, macro, semantic\_model). * `get_related_models`: Finds similar models using semantic search. * `get_seed_details`: *(deprecated — use `get_node_details` instead)* * `get_semantic_model_details`: *(deprecated — use `get_node_details` instead)* * `get_snapshot_details`: *(deprecated — use `get_node_details` instead)* * `get_source_details`: *(deprecated — use `get_node_details` instead)* * `get_test_details`: *(deprecated — use `get_node_details` instead)* * `search`: \[Alpha] Searches for resources across the dbt project (not generally available). ##### dbt CLI Allowing your client to utilize dbt commands through the MCP tooling could modify your data models, sources, and warehouse objects. Proceed only if you trust the client and understand the potential impact. * `build`: Executes models, tests, snapshots, and seeds in DAG order. * `clone`: Clones selected nodes from the specified state to the target schema(s). * `compile`: Generates executable SQL from models/tests/analyses; useful for validating Jinja logic. * `docs`: Generates documentation for the dbt project. * `get_lineage_dev`: Retrieves lineage from local manifest.json with type and depth filtering. * `get_node_details_dev`: Retrieves node details from local manifest.json (models, seeds, snapshots, sources). * `list`: Lists resources in the dbt project by type with selector support. * `parse`: Parses and validates project files for syntax correctness. * `run`: Executes models to materialize them in the database. * `show`: Executes SQL against the database and returns results. * `test`: Runs tests to validate data and model integrity. ##### Admin API To learn more about the dbt Administrative API, click [here](https://docs.getdbt.com/docs/dbt-cloud-apis/admin-cloud-api). * `cancel_job_run`: Cancels a running job. * `get_job_details`: Gets job configuration including triggers, schedule, and dbt commands. * `get_job_run_artifacts`: Downloads a specific artifact from a job run; large artifacts return a guidance message to retry with a jq\_filter; supports jq filtering for targeted extraction. * `get_job_run_details`: Gets run details including status, timing, steps, and artifacts. * `get_job_run_error`: Gets error and/or warning details for a job run; option to include or show warnings only. * `list_job_run_artifacts`: Lists available artifacts from a job run. * `list_jobs`: Lists jobs in a dbt Platform account; option to filter by project or environment. * `list_jobs_runs`: Lists job runs; option to filter by job, status, or order by field. * `list_projects`: Lists all projects in the dbt Platform account. * `retry_job_run`: Retries a failed job run. * `trigger_job_run`: Triggers a job run; option to override git branch, schema, or other settings. ##### dbt Codegen These tools help automate boilerplate code generation for dbt project files. * `generate_model_yaml`: Generates model YAML with columns; option to inherit upstream descriptions. * `generate_source`: Generates source YAML by introspecting database schemas; option to include columns. * `generate_staging_model`: Generates staging model SQL from a source table. ##### dbt LSP A set of tools that leverage the Fusion engine for advanced SQL compilation and column-level lineage analysis. * `fusion.compile_sql`: Compiles SQL in project context via dbt Platform. * `fusion.get_column_lineage`: Traces column-level lineage via dbt Platform. * `get_column_lineage`: Traces column-level lineage locally (requires dbt-lsp via dbt Labs VSCE). ##### Product Docs Tools for searching and fetching content from the official dbt documentation at docs.getdbt.com. * `get_product_doc_pages`: Fetches the full Markdown content of one or more docs.getdbt.com pages by path or URL. * `search_product_docs`: Searches docs.getdbt.com for pages matching a query; returns titles, URLs, and descriptions ranked by relevance. Use get\_product\_doc\_pages to fetch full content. ##### MCP Server Metadata These tools provide information about the MCP server itself. * `get_mcp_server_branch`: Returns the current git branch of the running dbt MCP server. * `get_mcp_server_version`: Returns the current version of the dbt MCP server. --- ### BigQuery and Apache Iceberg dbt supports materializing models in the Iceberg table format in two ways: * **Simplest:** The model config `table_format = 'iceberg'` instructs dbt to materialize this model as an Iceberg table in BigLake Metastore (managed catalog) * **Extensible:** Define an Iceberg catalog in `catalogs.yml` and configure this model with `catalog_name` #### Creating Iceberg tables dbt supports creating Iceberg tables for two of the BigQuery materializations: * [Table](https://docs.getdbt.com/docs/build/materializations.md#table) * [Incremental](https://docs.getdbt.com/docs/build/materializations.md#incremental) #### Iceberg catalogs BigQuery supports Iceberg tables through its built-in catalog [BigLake Metastore](https://cloud.google.com/bigquery/docs/iceberg-tables#architecture) today. No setup is needed to access the BigLake Metastore. However, you need to have a [storage bucket](https://docs.cloud.google.com/storage/docs/buckets#buckets) and [the required BigQuery roles](https://cloud.google.com/bigquery/docs/iceberg-tables#required-roles) configured prior to creating an Iceberg table. #### BigQuery-specific configs for `biglake_metastore` catalogs The following table outlines the configuration fields required to set up a catalog integration for [BigLake Iceberg tables in BigQuery](https://docs.cloud.google.com/bigquery/docs/iceberg-tables). Supply and nest these additional configurations, unique to BigQuery, under `config.bigquery` (in the new catalog spec) or `adapter_properties` (in the old catalog spec). | Field | Type | Required | Description | Note | | ----------------------- | ------ | ------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | | `file_format` | String | Yes, except for [Lakehouse Runtime Catalog (LRC)](#lakehouse-runtime-catalog-lrc) catalogs | The file format for the Iceberg table. | `parquet` is the only accepted value. | | `external_volume` | String | Yes, except for LRC catalogs | The Cloud Storage bucket where Iceberg table data is written. | For example, `gs://BUCKET_NAME`. | | `lakehouse_catalog` | String | No | The name of the LRC that holds this catalog's tables. | New spec only, and requires dbt v2 (v2). Set it in `catalogs.yml`, not per model. | | `base_location_root` | String | No | If provided, the input overrides the default dbt `base_location` value of `_dbt`. | Can be set in `catalogs.yml`. | | `base_location_subpath` | String | No | An optional suffix to add to the `base_location` path that dbt automatically specifies. | Only configurable per-model. | | `storage_uri` | String | No | If provided, the input overrides the dbt `storage_uri` value. | Only configurable per-model. | * `lakehouse_catalog`: Tells dbt that this catalog's tables live in a [Lakehouse Runtime Catalog](#lakehouse-runtime-catalog-lrc), so dbt addresses them with BigQuery's four-part name. LRC catalogs don't need `external_volume` or `file_format` because the LRC derives the storage location from the namespace. * `base_location_root`: Specifies the prefix of the base location path within the storage bucket where Iceberg table data is written. * `base_location_subpath`: Specifies the suffix of the base location path within the storage bucket where Iceberg table data is written. This property can only be set in model configurations, not in `catalogs.yml`. * `storage_uri`: Completely overrides the storage\_uri, allowing you to specify the full path directly instead of using the catalog integration's external volume and base\_location components. ##### Example 1. Create a `catalogs.yml` at the top level of your dbt project. ##### New spec (beta) catalogs.yml ```yaml catalogs: - name: my_biglake_catalog type: biglake_metastore table_format: iceberg config: bigquery: external_volume: 'gs://mydbtbucket' file_format: parquet ``` ##### Old spec catalogs.yml ```yaml catalogs: - name: my_biglake_catalog active_write_integration: biglake_metastore write_integrations: - name: biglake_metastore external_volume: 'gs://mydbtbucket' table_format: iceberg file_format: parquet catalog_type: biglake_metastore ``` 2. Apply the catalog configuration at either the model, folder, or project level: iceberg\_model.sql ```sql {{ config( materialized='table', catalog_name='my_biglake_catalog' ) }} select * from {{ ref('jaffle_shop_customers') }} ``` 3. Finally, run the model: `dbt run -s my_iceberg_model`. ##### Limitations BigQuery today doesn't support connecting to external Iceberg catalogs. In terms of SQL operations and table management features, refer to the [BigQuery docs](https://cloud.google.com/bigquery/docs/iceberg-tables#limitations) for more information. ##### Base location BigQuery's DDL for creating Iceberg tables requires that a fully qualified `storage_uri` be provided, including the object path. Once the user has provided the bucket name as the `external_volume` in the catalog integration, dbt manages the `storage_uri` input. The default behavior in dbt is to provide an object path, referred to in dbt as the `base_location`, in the form: `_dbt/{SCHEMA_NAME}/{MODEL_NAME}`. We recommend using the default behavior, but if you need to customize the resulting `base_location`, you can configure `base_location` with the model configuration fields `base_location_root` and `base_location_subpath`. * If no inputs are provided, dbt outputs for base\_location `{{ external_volume }}/_dbt/{{ schema }}/{{ model_name }}` * If base\_location\_root = `foo`, dbt outputs `{{ external_volume }}/foo/{{ schema }}/{{ model_name }}` * If base\_location\_subpath = `bar`, dbt outputs `{{ external_volume }}/_dbt/{{ schema }}/{{ model_name }}/bar` * If base\_location\_root = `foo` and base\_location\_subpath = `bar`, dbt outputs `{{ external_volume }}/foo/{{ schema }}/{{ model_name }}/bar` note While you can customize paths with `base_location_root` and `base_location_subpath`, we don't recommend relying on them for environment isolation (such as separating development and production environments). Anyone with repository access can easily modify these configuration values. For true environment isolation, use separate `external_volume` values with infrastructure-level access controls. You can also completely override the `storage_uri` with the model configuration field `storage_uri`. This overrides both the catalog integration path and the other model configuration fields to supply the entire `storage_uri` path directly. An example model with a customized `base_location`: iceberg\_model.sql ```sql {{ config( materialized='table', catalog_name='my_bigquery_iceberg_catalog', base_location_root='foo', base_location_subpath='bar', ) }} select * from {{ ref('jaffle_shop_customers') }} ``` ##### New spec (beta) catalogs.yml ```yaml catalogs: - name: my_bigquery_iceberg_catalog type: biglake_metastore table_format: iceberg config: bigquery: external_volume: 'gs://mydbtbucket' file_format: parquet base_location_root: foo ``` ##### Old spec catalogs.yml ```yaml catalogs: - name: my_bigquery_iceberg_catalog active_write_integration: biglake_metastore write_integrations: - name: biglake_metastore external_volume: 'gs://mydbtbucket' table_format: iceberg file_format: parquet catalog_type: biglake_metastore adapter_properties: base_location_root: foo ``` ###### Rationale By default, dbt manages the full `storage_uri` on behalf of users for ease of use. The `base_location` parameter specifies the location within the storage bucket where the data is written. Without guardrails (for example, if the user forgets to provide a base location root), it's possible for BigQuery to reuse the same path across multiple tables. This behavior could result in future technical debt because it limits the ability to: * Navigate the underlying object store * Read Iceberg tables through an object-store integration * Grant schema-specific access to tables through object store * Use a crawler pointed at the tables within the external storage to build a new catalog with another tool To maintain best practices, dbt enforces an input and, by default, writes your tables within a `_dbt/{SCHEMA_NAME}/{TABLE_NAME}` prefix to ensure easier object-store observability and auditability. (Applies to dbt v2.0 and later) #### Lakehouse Runtime Catalog (LRC) [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt v2 only `lakehouse_catalog` requires the [dbt v2](https://docs.getdbt.com/docs/introduction.md) (v2) with the `use_catalogs_v2` behavior flag enabled. dbt\_project.yml ```yaml flags: use_catalogs_v2: true ``` BigQuery's [Lakehouse Runtime Catalog](https://cloud.google.com/bigquery/docs/blms-rest-catalog) (LRC) addresses a table with four parts — project, catalog, namespace, and table — but BigQuery SQL accepts only three quoted segments. dbt handles this by quoting the catalog and namespace together as the middle segment: ```sql `{project}`.`{catalog}.{namespace}`.`{table}` ``` Set `lakehouse_catalog` on a `biglake_metastore` catalog to tell dbt that its tables live in an LRC. dbt then uses the four-part name and omits the connection clause and the `table_format` option, neither of which BigQuery accepts for LRC tables. ##### Prerequisites Create the LRC catalog and namespace before you run dbt. BigQuery has no SQL statement that creates them, so dbt can create only the table. Because of this, dbt attempts to create the namespace on every run against an LRC catalog, and BigQuery rejects it with the following warning: ```shell [FailedToCreateDatabase (dbt1051)]: Failed to create schema 'sales_catalog.analytics' in database 'my_project' in remote for model.my_project.my_lrc_model: [BigQuery] googleapi: Error 400: Invalid project ID 'my_project.sales_catalog'. ``` Expect this warning — it doesn't fail the run. As long as the catalog and namespace exist, dbt creates the table. ##### LRC example 1. Add a catalog with `lakehouse_catalog` set. An LRC catalog doesn't need `external_volume` or `file_format`. catalogs.yml ```yaml catalogs: - name: my_lrc_catalog type: biglake_metastore table_format: iceberg config: bigquery: lakehouse_catalog: sales_catalog ``` 2. Configure a model with `catalog_name`: my\_lrc\_model.sql ```sql {{ config( materialized='table', catalog_name='my_lrc_catalog' ) }} select * from {{ ref('jaffle_shop_customers') }} ``` 3. Run the model: `dbt run -s my_lrc_model`. dbt generates the following DDL: ```sql create or replace table `my_project`.`sales_catalog.analytics`.`my_lrc_model` OPTIONS() as ( select * from `my_project`.`analytics`.`jaffle_shop_customers` ) ``` ##### LRC limitations BigQuery doesn't expose LRC tables through `INFORMATION_SCHEMA`, so your queries against those views don't return them. This doesn't affect `dbt docs generate`. --- ### Build and view your docs with dbt dbt platform dbt enables you to generate documentation for your project and data platform. The documentation is automatically updated with new information after a fully successful job run, ensuring accuracy and relevance. The default documentation experience in dbt is [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md), available on [Starter, Enterprise, or Enterprise+ plans](https://www.getdbt.com/pricing/). Use [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) to view your project's resources (such as models, tests, and metrics) and their lineage to gain a better understanding of its latest production state. Refer to [documentation](https://docs.getdbt.com/docs/build/documentation.md) for more configuration details. This shift makes [dbt Docs](#dbt-docs) a legacy documentation feature in dbt. dbt Docs is still accessible and offers basic documentation, but it doesn't offer the same speed, metadata, or visibility as Catalog. dbt Docs is available to dbt developer plans or dbt v1 users. #### Set up a documentation job Upcoming change for dbt v2 jobs In a future update, this setup will no longer be applicable for dbt v2 jobs in the dbt platform. Execution commands (`run`, `build`, `seed`, `snapshot`) will automatically trigger metadata generation, so you won't need to add a `dbt docs generate` step or select the **Generate docs on run** option in **Execution settings**. Catalog uses the [metadata](https://docs.getdbt.com/docs/explore/explore-projects.md#generate-metadata) generated after each job run in the production or staging environment, ensuring it always has the latest project results. To view richer metadata, you can set up documentation for a job in dbt when you edit your job settings or create a new job. Configure the job to [generate metadata](https://docs.getdbt.com/docs/explore/explore-projects.md#generate-metadata) when it runs. If you want to view column and statistics for models, sources, and snapshots in Catalog, then this step is necessary. To set up a job to generate docs: 1. In the top left, click **Deploy** and select **Jobs**. 2. Create a new job or select an existing job and click **Settings**. 3. Under **Execution Settings**, select **Generate docs on run** and click **Save**. [![Setting up a job to generate documentation](/img/docs/dbt-platform/using-dbt-platform/documentation-job-execution-settings.png?v=2 "Setting up a job to generate documentation")](#)Setting up a job to generate documentation *Note, for dbt Docs users you need to configure the job to generate docs when it runs, then manually link that job to your project. Proceed to [configure project documentation](#configure-project-documentation) so your project generates the documentation when this job runs.* You can also add the [`dbt docs generate` command](https://docs.getdbt.com/reference/commands/cmd-docs.md) to the list of commands in the job run steps. However, you can expect different outcomes when adding the command to the run steps compared to configuring a job selecting the **Generate docs on run** checkbox. Review the following options and outcomes: | Options | Outcomes | | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Select checkbox** | Select the **Generate docs on run** checkbox to automatically generate updated project docs each time your job runs. If that particular step in your job fails, the job can still be successful if all subsequent steps are successful. | | **Add as a run step** | Add `dbt docs generate` to the list of commands in the job run steps, in whatever order you prefer. If that particular step in your job fails, the job will fail and all subsequent steps will be skipped. | Tip — Documentation-only jobs To create and schedule documentation-only jobs at the end of your production jobs, add the `dbt compile` command in the **Commands** section. #### dbt Docs dbt Docs, available on developer plans or dbt v1 users, generates a website from your dbt project using the `dbt docs generate` command. It provides a central location to view your project's resources, such as models, tests, and lineage — and helps you understand the data in your warehouse. ##### Configure project documentation You configure project documentation to generate documentation when the job you set up in the previous section runs. In the project settings, specify the job that generates documentation artifacts for that project. Once you configure this setting, subsequent runs of the job will automatically include a step to generate documentation. 1. From dbt, click on your account name in the left side menu and select **Account settings**. 2. Navigate to **Projects** and select the project that needs documentation. 3. Click **Edit**. 4. Under **Artifacts**, select the job that should generate docs when it runs and click **Save**. [![Configuring project documentation](/img/docs/dbt-platform/using-dbt-platform/documentation-project-details.png?v=2 "Configuring project documentation")](#)Configuring project documentation Use Catalog for a richer documentation experience For a richer and more interactive experience, try out [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md), available on [Starter, Enterprise, or Enterprise+ plans](https://www.getdbt.com/pricing/). It includes map layers of your DAG, keyword search, interacts with the Studio IDE, model performance, project recommendations, and more. ##### Generating documentation To generate documentation in the Studio IDE, run the `dbt docs generate` command (dbt v1 only) in the **Command Bar** in the Studio IDE. This command will generate the documentation for your dbt project as it exists in development in your IDE session. After running `dbt docs generate` in the Studio IDE, click the icon above the file tree, to see the latest version of your documentation rendered in a new browser window. ##### View documentation Once you set up a job to generate documentation for your project, you can click **Catalog** in the navigation and then click on **dbt Docs**. Your project's documentation should open. This link will always help you find the most recent version of your project's documentation in dbt. These generated docs always show the last fully successful run, which means that if you have any failed tasks, including tests, then you will not see changes to the docs by this run. If you don't see a fully successful run, then you won't see any changes to the documentation. The Studio IDE makes it possible to view [documentation](https://docs.getdbt.com/docs/build/documentation.md) for your dbt project while your code is still in development. With this workflow, you can inspect and verify what your project's generated documentation will look like before your changes are released to production. #### Related docs * [Documentation](https://docs.getdbt.com/docs/build/documentation.md) * [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) --- ### Build your metrics Use MetricFlow in dbt to centrally define your metrics. As a key component of the [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md), MetricFlow is responsible for SQL query construction and defining specifications for dbt semantic models and metrics. It uses familiar constructs like semantic models and metrics to avoid duplicative coding, optimize your development workflow, ensure data governance for company metrics, and guarantee consistency for data consumers. For a complete list of configuration options and property specs, see the [Semantic Layer reference](https://docs.getdbt.com/reference/semantic-layer-reference.md). [![This diagram shows how the dbt Semantic Layer works with your data stack.](/img/docs/dbt-platform/semantic-layer/sl-concept.png?v=2 "This diagram shows how the dbt Semantic Layer works with your data stack.")](#)This diagram shows how the dbt Semantic Layer works with your data stack. MetricFlow allows you to: * Intuitively define metrics in your dbt project * Develop from your preferred environment, whether that's the [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md), [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md), or [dbt v1](https://docs.getdbt.com/docs/local/install-dbt.md) * Use [MetricFlow commands](https://docs.getdbt.com/docs/build/metricflow-commands.md) to query and test those metrics in your development environment * Harness the true magic of the universal Semantic Layer and dynamically query these metrics in downstream tools (Available for dbt [Starter, Enterprise, or Enterprise+](https://www.getdbt.com/pricing/) accounts only). [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/reference/semantic-layer-reference.md) ###### [Semantic Layer reference](https://docs.getdbt.com/reference/semantic-layer-reference.md) [Complete, exhaustive configuration reference for semantic models, metrics, and dimensions.](https://docs.getdbt.com/reference/semantic-layer-reference.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/build/latest-metrics-spec.md) ###### [Migrate to the latest YAML spec](https://docs.getdbt.com/docs/build/latest-metrics-spec.md) [Learn how to migrate from the legacy metrics YAML spec to the latest metrics YAML spec.](https://docs.getdbt.com/docs/build/latest-metrics-spec.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/guides/sl-qs.md) ###### [Quickstart for the dbt Semantic Layer](https://docs.getdbt.com/guides/sl-qs.md) [Use this guide to build and define metrics, set up the dbt Semantic Layer, and query them using downstream tools.](https://docs.getdbt.com/guides/sl-qs.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/build/about-metricflow.md) ###### [About MetricFlow](https://docs.getdbt.com/docs/build/about-metricflow.md) [Understand MetricFlow's core concepts, how to use joins, how to save commonly used queries, and what commands are available.](https://docs.getdbt.com/docs/build/about-metricflow.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/build/semantic-models.md) ###### [Semantic model](https://docs.getdbt.com/docs/build/semantic-models.md) [Use semantic models as the basis for defining data. They act as nodes in the semantic graph, with entities connecting them.](https://docs.getdbt.com/docs/build/semantic-models.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/build/metrics-overview.md) ###### [Metrics](https://docs.getdbt.com/docs/build/metrics-overview.md) [Define metrics in your dbt project using different metric types in YAML files.](https://docs.getdbt.com/docs/build/metrics-overview.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/build/advanced-topics.md) ###### [Advanced topics](https://docs.getdbt.com/docs/build/advanced-topics.md) [Learn about advanced topics for dbt Semantic Layer and MetricFlow, such as data modeling workflows, and more.](https://docs.getdbt.com/docs/build/advanced-topics.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) ###### [About the dbt Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) [Introducing the dbt Semantic Layer, the universal process that allows data teams to centrally define and query metrics](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) ###### [Available integrations](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) [Discover the diverse range of partners that seamlessly integrate with the powerful dbt Semantic Layer, allowing you to query and unlock valuable insights from your data ecosystem.](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) <br /> #### Related docs * [Quickstart guide with the Semantic Layer](https://docs.getdbt.com/guides/sl-qs.md) * [The Semantic Layer: what's next](https://www.getdbt.com/blog/dbt-semantic-layer-whats-next/) blog * [Semantic Layer on-demand course](https://learn.getdbt.com/courses/semantic-layer) * [Semantic Layer FAQs](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-faqs.md) --- ### Cache common queries dbt platform | Enterprise, Enterprise+ The Semantic Layer allows you to cache common queries in order to speed up performance and reduce compute on expensive queries. There are two different types of caching: * [Result caching](#result-caching) leverages your data platform's built-in caching layer. * [Declarative caching](#declarative-caching) allows you to pre-warm the cache using saved queries configuration. While you can use caching to speed up your queries and reduce compute time, knowing the difference between the two depends on your use case: * Result caching happens automatically by leveraging your data platform's cache. * Declarative caching allows you to 'declare' the queries you specifically want to cache. With declarative caching, you need to anticipate which queries you want to cache. * Declarative caching also allows you to dynamically filter your dashboards without losing the performance benefits of caching. This works because filters on dimensions (that are already in a saved query config) will use the cache. #### Prerequisites * dbt [Enterprise or Enterprise+](https://www.getdbt.com/) plans. * dbt environments must be on [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) and not legacy dbt v1 versions. * A successful job run and [production environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md#set-as-production-environment). * For declarative caching, you need to have [exports](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md) defined in your [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md) YAML configuration file. #### Result caching Result caching leverages your data platform’s built-in caching layer and features. [MetricFlow](https://docs.getdbt.com/docs/build/about-metricflow.md) generates the same SQL for multiple query requests, this means it can take advantage of your data platform’s cache. Double-check your data platform's specifications. Here's how caching works, using Snowflake as an example, and should be similar across other data platforms: 1. **Run from cold cache** — When you run a semantic layer query from your BI tool that hasn't been executed in the past 24 hours, the query scans the entire dataset and doesn't use the cache. 2. **Run from warm cache** — If you rerun the same query after 1 hour, the SQL generated and executed on Snowflake remains the same. On Snowflake, the result cache is set per user for 24 hours, which allows the repeated query to use the cache and return results faster. Different data platforms might have different caching layers and cache invalidation rules. Here's a list of resources on how caching works on some common data platforms: * [BigQuery](https://cloud.google.com/bigquery/docs/cached-results) * [DataBricks](https://docs.databricks.com/en/optimizations/disk-cache.html) * [Microsoft Fabric](https://learn.microsoft.com/en-us/fabric/data-warehouse/caching) * [Redshift](https://docs.aws.amazon.com/redshift/latest/dg/c_challenges_achieving_high_performance_queries.html#result-caching) * [Snowflake](https://community.snowflake.com/s/article/Caching-in-the-Snowflake-Cloud-Data-Platform) * [Starburst Galaxy](https://docs.starburst.io/starburst-galaxy/data-engineering/optimization-performance-and-quality/workload-optimization/warp-speed-enabled.html) #### Declarative caching Declarative caching enables you to pre-warm the cache using [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md) by setting the cache config to `true` in your `saved_queries` settings. This is useful for optimizing performance for key dashboards or common ad-hoc query requests. tip Declarative caching also allows you to dynamically filter your dashboards without losing the performance benefits of caching. This works because filters on dimensions (that are already in a saved query config) will use the cache. For example, if you filter a metric by geographical region on a dashboard, the query will hit the cache, ensuring faster results. This also removes the need to create separate saved queries with static filters. For configuration details, refer to [Declarative caching setup](#declarative-caching-setup). How declarative caching works: * Make sure your saved queries YAML configuration file has [exports](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md) defined. * Running a saved query triggers the Semantic Layer to: * Build a cached table from a saved query, with exports defined, into your data platform. * Make sure any query requests that match the saved query's inputs use the cache, returning data more quickly. * Automatically invalidates the cache when it detects new and fresh data in any upstream models related to the metrics in your cached table. * Refreshes (or rebuilds) the cache the next time you run the saved query. 📹 Check out this video demo to see how declarative caching works! This video demonstrates the concept of declarative caching, how to run it using the dbt scheduler, and how fast your dashboards load as a result. Refer to the following diagram, which illustrates what happens when the Semantic Layer receives a query request: [![Overview of the declarative cache query flow](/img/docs/dbt-platform/semantic-layer/declarative-cache-query-flow.jpg?v=2 "Overview of the declarative cache query flow")](#)Overview of the declarative cache query flow ##### Declarative caching setup To populate the cache, you need to configure an export in your saved query YAML file configuration *and* set the `cache config` to `true`. You can't cache a saved query without an export defined. semantic\_model.yml ```yaml saved_queries: - name: my_saved_query ... # Rest of the saved queries configuration. config: cache: enabled: true # Set to true to enable, defaults to false. exports: - name: order_data_key_metrics config: export_as: table ``` To enable saved queries at the project level, you can set the `saved-queries` configuration in the [`dbt_project.yml` file](https://docs.getdbt.com/reference/dbt_project.yml.md). This saves you time in configuring saved queries in each file: dbt\_project.yml ```yaml saved-queries: my_saved_query: config: +cache: enabled: true ``` ##### Run your declarative cache After setting up declarative caching in your YAML configuration, you can now run [exports](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md) with the dbt job scheduler to build a cached table from a saved query into your data platform. * Use [exports to set up a job](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md) to run a saved query dbt. * The dbt Semantic Layer builds a cache table in your data platform in a dedicated `dbt_sl_cache` schema. * The cache schema and tables are created using your deployment credentials. You need to grant read access to this schema for your Semantic Layer user. * The cache refreshes (or rebuilds) on the same schedule as the saved query job. [![Overview of the cache creation flow.](/img/docs/dbt-platform/semantic-layer/cache-creation-flow.jpg?v=2 "Overview of the cache creation flow.")](#)Overview of the cache creation flow. After a successful job run, you can go back to your dashboard to experience the speed and benefits of declarative caching. #### Cache management dbt uses the metadata from your dbt model runs to intelligently manage cache invalidation. When you start a dbt job, it keeps track of the last model runtime and checks the freshness of the metrics upstream of your cache. If an upstream model has data in it that was created after the cache was created, dbt invalidates the cache. This means queries won't use outdated cases and will instead query directly from the source data. Stale, outdated cache tables are periodically dropped and dbt will write a new cache the next time your saved query runs. You can manually invalidate the cache through the [dbt Semantic Layer APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md) using the `InvalidateCacheResult` field. #### FAQs  How does caching interact with access controls? Cached data is stored separately from the underlying models. If metrics are pulled from the cache, we don’t have the security context applied to those tables at query time. In the future, we plan to clone credentials, identify the minimum access level needed, and apply those permissions to cached tables. #### Related docs * [Validate semantic nodes in CI](https://docs.getdbt.com/docs/deploy/ci-jobs.md#semantic-validations-in-ci) * [Saved queries](https://docs.getdbt.com/docs/build/saved-queries.md) * [Semantic Layer FAQs](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-faqs.md) --- ### Changelog 2019 and 2020 note This changelog references dbt versions that are no longer supported and have been removed from the docs. For more information about upgrading to a supported version of dbt in your dbt Cloud environment, read [Upgrade dbt version in Cloud](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md). Welcome to the 2019 and 2020 changelog for the dbt application! You can use this changelog to see the highlights of what was new, fixed, and enhanced during this time period. #### dbt Cloud v1.1.16 (December 23, 2020) This release adds preview support for Databricks Spark in dbt and adds two new permission sets for Enterprise acccounts. ###### Enhancements * Added preview support for Databricks Spark support * Added two new Enterprise permission sets: Account Viewer and Project Creator ###### Fixed * Improve logging infrastructure for dbt run logs * Fix for SSH tunnel logging errors #### dbt Cloud v1.1.15 (December 10, 2020) Lots of great stuff to confer about this go-round: things really coalesced this week! Lots of excitement around adding Spark to the connection family, as well as knocking out some longstanding bugs. ###### Enhancements * Add Spark as an option for database setup ###### Fixed * Fix this one hairy bug where one email could have multiple user accounts * Fix setup-connection react-page routing * Break out group selection logic from license types and group names * Handle JSON errors in v1/v2 body parsing * Handle AuthForbidden and AuthCancelled graciously - ie, not throw 500s * Fix regression with Studio IDE loading spinner #### dbt Cloud v1.1.14 (November 25, 2020) This release adds a few new pieces of connective tissue, notably OAuth for BigQuery and SparkAdapter work. There are also some quality of life improvements and investments for the future, focused on our beloved Studio IDE users, and some improved piping for observability into log management and API usage. ###### Enhancements * Update IP allowlist * User can OAuth for BigQuery in profile credentials * Adding SparkAdapter backend models, mappers, and services * Added BigQuery OAuth integration * Adding db index for owner\_thread\_id ###### Fixed * Fix post /run error rate * Fix bug where bad argument was passed to dbt runs * Log out unhandled error in environment variable context manager * Remove account settings permissions for user integrations #### dbt Cloud v1.1.13 (November 12, 2020) This release adds support for triggering runs with overriden attributes via the [triggerRun](https://docs.getdbt.com/dbt-cloud/api-v2) API endpoint. Additionally, a number of bugs have been squashed and performance improvements have been made. ###### Enhancements * Improve error handling for long-running queries in the Studio IDE * Use S3 client caching to improve log download speed for scheduled runs * Support triggering jobs [with overriden attributes from the API](https://docs.getdbt.com/dbt-cloud/api-v2) * Clarify "upgrade" copy on the billing page ###### Fixed * GitLab groups endpoint now returns all groups and subgroups * Support BigQuery retry configs with value 0 * Prevent web IDE from crashing after running an invalid dbt command * Apply additional log scrubbing to filter short-lived git credentials * Fix older migration to make auth\_url field nullable * Support paths in GitLab instance URL * Fix for auth token request url in GitLab oauth flow #### dbt Cloud v1.1.12 (October 30, 2020) This release adds dbt v.18.1 and 0.19.0b1 to dbt Cloud. Additionally, a number of bugs have been fixed. ###### Enhancements * Update copy on billing page for picking a plan at the end of a trial * Improved authorization for metadata API * Add dbt 0.19.0b1 * Add dbt 0.18.1 ###### Fixed * Fixed an issue where groups from other logged-in accounts appeared in the RBAC UI * Fixed requested GitLab scopes and an issue when encrypting deploy tokens for GitLab auth * Fixed an issue where null characters in logs threw errors in scheduled runs #### dbt Cloud v1.1.11 (October 15, 2020) Release v1.1.11 includes some quality-of-life enhancements, copy tweaks, and error resolutions. It also marks the last time we'll have the same digit four times in a row in a release until v2.2.22. ###### Enhancements * Add InterfaceError exception handling for commands * Rename My Account --> Profile * Add project and connection to admin backend ###### Fixed * Resolve errors from presence of null-characters in logs * Email verifications backend * Undo run.serialize * Fix error while serialized run * Fix logic error in connection setup * Fix a bug with GitLab auth flow for unauthenticated users * Fix bug where Native Okta SSO uses the wrong port #### dbt Cloud v1.1.10 (October 8, 2020) This release adds support for repositories imported via GitLab (Enterprise) and contains a number of bugfixes and improvements in the Studio IDE. ###### Enhancements * Add Gitlab integration (Enterprise) * Add GitLab repository setup to project setup flow (Enterprise) * Add GitLab automated Deploy Token installation (Enterprise) * Add dbt 0.18.1rc1 ###### Fixed * Fix bug where Studio IDE gets stuck after changing project repository * Fix race condition where connections can be added to the wrong project * Fix revoking email invites * Fix a bug in slim CI deferring run search where missing previous run caused the scheduler to raise an error * Fix a source of Studio IDE instability * Gracefully clean up Studio IDE backend on shutdown * Always show SSO mappings on Group Details page #### dbt Cloud v1.1.9 (October 1, 2020) This release adds the ability for admins on the Enterprise plan to configure the Role Based Access Control permissions applied to Projects in their account. Additionally, job execution deferral is now available behind a feature flag, and a number of fixes and improvements were released as well. ###### Enhancements * Add dbt version in the navigation sidebar * Add RBAC Group Permission view, create, and modify UIs * Add personal git auth for Studio IDE error handling modals * Add Develop Requests to backend views * Implemented job execution deferral * Add support for dbt v0.18.1b2 ###### Fixed * Fixed the scenario where interacting with the Refresh Studio IDE button causes an index.lock file to remain in the Studio IDE file system * Validate PR URL for XSS attempts * Address RBAC inconsistencies * Fixed users not being able to update their dbt password in-app * Fix for applying user permissions across multiple accounts after SSO auth * Google API: default to common api endpoint but allow override * Fix for missing email variable in GSuite debug logging * Destroy Studio IDE session when switching projects #### dbt Cloud v1.1.8 (September 17, 2020) This release adds native support for Okta SSO and dbt v0.18.0. It also adds initial support for a GitLab integration and self-service RBAC configuration. ###### Enhancements * Add dbt 0.18.0 * Add native Okta SSO support * Add additional logging for Gsuite and Azure SSO * Add git cloning support via GitLab deploy tokens for scheduled runs (coming soon) * add RBAC Groups Detail Page and Groups List UIs ###### Fixed * Allow `*_proxy` env vars in scheduled runs #### dbt Cloud v1.1.7 \[September 3, 2020] This release adds a Release Candidate for [dbt v0.18.0](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade.md) and includes bugfixes and improvements to the Cloud IDE and job scheduler. ###### Enhancements * Improve scheduler backoff behavior * Add dbt 0.18.0rc1 * Add support for non-standard ssh ports in connection tunnels * Add support for closing the Studio IDE filesystem context menu by clicking outside the menu ###### Fixed * Fix for joining threads in run triggers * Fix thread caching for s3 uploads #### dbt Cloud v1.1.6 (August 20, 2020) This release includes security enhancements and improvements across the entire dbt application. ###### Enhancements * Support for viewing development docs inside of the Studio IDE ([docs](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) * Change CI temporary schema names to be prefixed with `dbt_cloud` instead of `sinter` * Change coloring and iconography to improve accessibility and UX across the application * \[Enterprise] Support the specification of multiple authorized domains in SSO configuration * \[On-premises] Upgrade boto3 to support KIAM authentication ###### Fixed * \[Enterprise] Fix for missing IdP group membership mappings when users belong to >100 Azure AD groups * Disallow the creation of symlinks in the Studio IDE * Improve reliability of background cleanup processes * Improve performance and reliability of artifact management and PR webhook processing #### dbt Cloud v1.1.5 (August 4, 2020) This release adds a major new feature to the Studio IDE: merge conflict resolution! It also includes changes to the job scheduler that cut the time and resource utilization significantly. ###### Enhancements * Add dbt 0.17.2 * Add dbt 0.18.0 beta 2 * Add merge conflict resolution, a merge commit workflow, and merge abort workflow to the IDE * Deprecate dbt versions prior to 0.13.0 * Refactor to cut job scheduler loop time * Reduce extra database calls to account table in job scheduler loop * \[On-premises] Allow clients to disable authentication for SMTP * \[On-premises] Allow disabling of TLS for SMTP * \[On-premises] Making k8s access mode for Studio IDE pods an environment variable * \[Security] Force session cookie to be secure * Make api and admin modules flake8 complaint ###### Fixed * Fix incorrect usage of `region_name` in KMS client * Fix a call to a deprecated Github API * Remove extraneous billing API calls during job scheduler loop * Fix error where refreshing the IDE would leave running dbt processes in a bad state #### dbt Cloud v1.1.4 (July 21, 2020) This release dramatically speeds up the job scheduler. It adds a new stable dbt version (0.17.1) and a new prerelease (0.17.2b1), and it includes a number of bugfixes. ###### Enhancements * Add dbt 0.17.2b1 * Add dbt 0.17.1 and set as default version * Speed up job scheduler by 50% * Added generate docs to rpc service and new view docs route * Queue limiting by account for scheduled jobs ###### Fixed * Fix enterprise SSO configuration when old Auth0 Azure AD is configured * Do not schedule jobs for deleted job definitions or environments * Fix permissions issues * Fix a bug with metadata set in azure storage provider * Fixed error when switching to developer plan from trial * Fix authentication bug where we setup all accounts with same domain * \[Security] Add security check to prevent potentially malicious html files in dbt docs #### dbt Cloud v1.1.3 (July 7, 2020) This release contains a number of IDE features and bugfixes, a new release candidate of dbt, and a brand new Enterprise Single-Sign On method: Azure Active Directory! ###### Enhancements * Add dbt 0.17.1rc3 * Snowflake: Add support for `client_session_keep_alive` config * Enterprise: Native Azure Oauth2 for Enterprise accounts * Studio IDE: Add custom command palette for finding files ###### Fixed * Do not run CI builds for draft PRs in GitHub * Remove race condition when syncing account with stripe billing events * Enterprise: Fixed JIT provisioning bug impacting accounts with shared IdP domains * Studio IDE: Fix a regression with Github git clone method * Studio IDE: Fix a race condition where git clone didn't complete before user entered Studio IDE * Studio IDE: Fix bug with checking out an environment custom branch on Studio IDE refresh * Bigquery: Fix PR schema dropping #### dbt Cloud v1.1.2 (June 23, 2020) This branch includes an important security fix, two new versions of dbt, and some miscellaneous fixes. ###### Enhancements * Add project names to the account settings notifications section * Add dbt 0.17.1 release candidate * Update development dbt version to Marian Anderson * Add remember me to login page and expire user sessions at browser close * Adding Auth Provider and enabling Gsuite SSO for enterprise customers ###### Fixed * \[Security] Fix intra-account API key leakage * Support queries containing unicode characters in the Studio IDE #### dbt Cloud v1.1.1 (June 9, 2020) This release includes dbt 0.17.0 and a number of IDE quality of life improvements. ###### Enhancements * Added dbt 0.17.0 * Added the ability to create a new folder in the IDE * Added gitignore status to file system and display dbt artifacts, including directories dbt\_modules, logs, and target * (Cloud only) Added rollbar and update some various error handling clean up * (On-premises only) Admin site: allow Repository's Pull Request Template field to be blank * (On-premises only) Added AWS KMS support ###### Fixed * Expires old pending password reset codes when a new password reset is requested #### dbt Cloud v1.1.0 (June 2, 2020) This release adds some new admin backend functionality, as well as automatic seat usage reporting. ##### On-Premises Only ###### Added * Added automatic reporting of seat usage. ###### Changed * Admins can now edit remote URLs for repository in the admin backend. * Admins can now edit credentials in the admin backend. *** #### dbt Cloud v1.0.12 (May 27, 2020) This release contains a few bugfixes for the Studio IDE and email notifications, as well as the latest release candidate of 0.17.0. ##### All versions ###### Added * Use the correct starter project tag, based on dbt version, when initializing a new project in the IDE * Added branch filtering to IDE git checkout UI. * Added dbt 0.17.0-rc3. ###### Fixed * Fixed source freshness report for dbt version v0.17.0 * Fixed issue with checking-out git branches * Fixed issue of logs being omitted on long running queries in the Studio IDE * Fixed slack notifications failing to send if email notifications fail ##### On-Premises Only ###### Added * Added an Admin page for deleting credentials. *** #### dbt Cloud v1.0.11 (May 19, 2020) This version adds some new permission sets, and a new release candidate of dbt. ##### All versions ###### Added * Added permission sets for Job Viewer, Job Admin and Analyst. * Added dbt 0.17.0-rc1 *** #### dbt Cloud v1.0.10 (May 11, 2020) ##### All versions ###### Added * Added dbt 0.17.0-b1. * PR Url is now self serve configurable. * Added more granular permissions around creating and deleting permissions. (Account Admin can create new projects by default while both Account Admin and Project Admin can delete the projects they have permissions for by default) * Added an error message to display to users that do not have permissions set up for any projects on an account. ###### Fixed * Removed .sql from CSV download filename * Fixed breaking JobDefinition API with new param custom\_branch\_only * Fixed Studio IDE query table column heading casing *** #### dbt Cloud v1.0.9 (May 5, 2020) This release includes bugfixes around how permissions are applied to runs and run steps, fixes a bug where the scheduler would hang up, and improves performance of the Studio IDE. ##### All versions ###### Fixed * Fixed permission checks around Runs and Run Steps, this should only affect Enterprise accounts with per-project permissions. * Fixed receiving arbitrary remote\_url when creating a git url repository. * Fixed issue when handling non-resource specific errors from RPC server in Studio IDE. * Fixed a bug where the scheduler would stop if the database went away. * Fixed IDE query results table not supporting horizontal scrolling. ###### Changed * Improve Studio IDE query results performance. * Allow configuration on jobs to only run builds when environment target branch is env's custom branch. * Allow configuration of GitHub installation IDs in the admin backend. ##### On-Premises Only ###### Fixed * Fixed logic error for installations with user/password auth enabled in an on-premises context *** #### dbt Cloud v1.0.8 (April 28, 2020) This release adds a new version of dbt (0.16.1), fixes a number of IDE bugs, and fixes some dbt Cloud on-premises bugs. ##### All versions ###### Added * Add dbt 0.16.1 ###### Fixed * Fixed Studio IDE filesystem loading to check for directories to ensure that load and write methods are only performed on files. * Fixed a bug with generating private keys for connection SSH tunnels. * Fixed issue preventing temporary PR schemas from being dropped when PR is closed. * Fix issues with Studio IDE tabs not updating query compile and run results. * Fix issues with query runtime timer in Studio IDE for compile and run query functions. * Fixed what settings are displayed on the account settings page to align with the user's permissions. * Fixed bug with checking user's permissions in frontend when user belonged to more than one project. * Fixed bug with access control around environments and file system/git interactions that occurred when using Studio IDE. * Fixed a bug with Environments too generously matching repository. ###### Changed * Make the configured base branch in the Studio IDE read-only. * Support configuring groups using an account ID in the admin backend. * Use gunicorn webserver in Studio IDE. * Allow any repository with a Github installation ID to use build-on-PR. * Member and Owner Groups are now editable from admin UI. ##### On-Premises Only ###### Fixed * Fixed an issue where account license counts were not set correctly from onprem license file. * Fixed an issue where docs would sometimes fail to load due to a server error. *** #### dbt Cloud v1.0.7 (April 13, 2020) This release rolls out a major change to how permissions are applied in dbt's API. It also adds some minor bugfixes, and some tooling for improved future QA. ##### All versions ###### Added * Added support to permission connections on a per project basis. * Added support to permission credentials on a per project basis. * Added support to permission repositories on a per project basis. * Smoke tests for account signup, user login and basic project setup * Add dbt 0.16.1rc1 * Non-enterprise users can now add new accounts from the Accounts dropdown. ###### Fixed * Fix missing migration for credentials. * Fixed issue with testing connections with a non-default target name specified in the credentials. * Fix issue where Bigquery connections could be created with invalid values for `location`. *** #### dbt Cloud v1.0.6 (March 30, 2020) This release adds UIs to select group permissions in the project settings UI. It also contains bugfixes for the Studio IDE, PR build schema dropping, and adds support for dissociating Github and Slack integrations via the Admin backend. ##### All versions ###### Added * (Enterprise only) Added ability to create group permissions for specific projects in the project settings UI. ###### Fixed * Fix empty state for selecting github repositories * Fixed an issue with the IDE failing to report an invalid project subdirectory for a dbt project * Fix blank loading screen displayed when switching accounts while on account/profile settings page * Fix issue preventing schemas from dropping during PR builds * Fix issue where whitespace in user's name breaks default schema name * Added webhook processing for when a user disassociates github access to their account. * Added slack disassociation capability on user integrations page and on backend admin panel (for notifications). ###### Changed * Declare application store using configureStore from redux-toolkit *** #### dbt Cloud v1.0.5 (March 23, 2020) ##### All versions ###### Added * Add support for authenticating Development and Deployment Snowflake credentials using keypair auth * Add support for checking out tags, render git output in "clone" run step * Add dbt 0.15.3 * Add dbt 0.16.0 ###### Fixed * Git provider urls now built with correct github account and repository directories. * Invalid DateTime Start time in Studio IDE Results Panel KPIs. * Fix a race condition causing the Invite User UI to not work properly. * Incorrect model build times in Studio IDE. ###### Changed * Git: ignore `logs/` and `target/` directories in the IDE. *** #### 1.0.4 (March 16, 2020) This release adds two new versions of dbt, adds Snowflake SSO support for Enterprise accounts, and fixes a number of bugs. ##### All versions ###### Added * Added dbt 0.15.3rc1 * Added dbt 0.16.0rc2 * Add support for cloning private deps in the IDE when using deploy key auth. * Log user that kicked off manual runs. * Enterprise support for authenticating user Snowflake connections using Snowflake single sign-on ###### Fixed * Fixed issue loading accounts for a user if they lack permissions for any subset of accounts they have a user license for. * Fixed issue with showing blank page for user who is not associated with any accounts. * Fixed issue where runs would continue to kick off on a deleted project. * Fixed issue where accounts connected to GitHub integrations with SAML protection could not import repositories * Improved error messages shown to the user if repos are unauthorized in a GitHub integration when importing a repo * Fix colors of buttons in generated emails ##### On-Premises ###### Added * Added Admin backend UIs for managing user permissions. *** #### 1.0.3 (March 1, 2020) This release contains the building blocks for RBAC, and a number of bugfixes and upgrades. ##### All versions ###### Added * Add support for a read replica for reading runs from the API. * Added groups, group permissions, and user groups. * Add email address to email verification screen. * Add Enterprise Permissions. * Allow account-level access to resources for groups with a permission statement of "all resources" for api backwards compatibility. * Add dbt 0.16.0b3 ###### Fixed * Fix issue with loading projects after switching accounts. * Fix broken links to connections from deployment environment settings. * Fix a bug with inviting readonly users. * Fix a bug where permissions were removed from Enterprise users upon login. ###### Changed * Update Django version: 2.2.10 * Update Django admin panel version * Update Social Auth version and the related Django component * Update jobs from using account-based resource permissions to project-based resource permissions * Update modal that shows when trials are expired; fix copy for past-due accounts in modal * Replace formatted string logging with structured logging * Move connection and repository settings from account settings to project settings * Update project setup flow to be used for creating projects * Update develop requests to have a foreign key on projects ##### On-Premises ###### Added * Accounts created from admin backend will come with a default set of groups ###### Changed * Rename "Fishtown Analytics User" to "Superuser" *** #### dbt Cloud v1.0.2 (February 20, 2020) This release contains a number of package upgrades, and a number of bugfixes. ##### All versions ###### Added * Add request context data to logs * Comprehensive logging for git subprocesses ###### Fixed * Fix an issue where the "Cancel Run" button does not work * Fix warnings regarding mutable resource model defaults for jobs and job notifications * Fix bug where users can create multiple connection user credentials through the project setup workflow * Update auth for requests against Github's api from using query parameters to using an Authorization header * Remove unused threads input from deployment environments * Fix issue that prevented user from viewing documentation and data sources * Fix issue rendering code editor panel in the IDE when using Safari * Fix issue with log levels that caused dbt logs to be too chatty ###### Changed * Update Django version: 2.2.10 * Update Django admin panel version * Update Social Auth version and the related Django component * Update jobs from using account-based resource permissions to project-based resource permissions * Update modal that shows when trials are expired; fix copy for past-due accounts in modal * Replace formatted string logging with structured logging * Move connection and repository settings from account settings to project settings * Update project setup flow to be used for creating projects ###### Removed None. *** #### dbt Cloud v1.0.1 (February 4, 2020) This release makes the IDE generally available, and adds two new versions of dbt (0.15.1, 0.15.2). For on-premises customers, there is a new set of configurations in the configuration console: SMTP: You can now configure dbt to send email notifications through your own SMTP server. RSA Encryption: You can now provide your own RSA keypair for dbt to use for encryption. These fields need to be specified for your instance of dbt to function properly. ##### All versions ###### Added * New Team List page * New Team User Detail page * New Invite User page * New dashboard for Read Only users * New dbt version: 0.15.1 * New dbt version: 0.15.2 * Ability to rename files in Studio IDE * New backend service for project-based resource permissions ###### Fixed * Fix an issue where the user has to repeat steps in the onboarding flow * Fix issue where user can get stuck in the onboarding flow * Fix bug where email notifications could be sent to deleted users * Fix UI bug not allowing user to check "Build on pull request?" when creating a job * Fix UI bug in header of the Edit User page * Fix issue that did not take into account pending invites and license seats when re-sending a user invite. * Fix an issue when processing Github webhooks with unconfigured environments * Fix console warning presented when updating React state from unmounted component * Fix issue where closed tabs would continue to be shown, though the content was removed correctly * Fix issue that prevented opening an adjacent tab when a tab was closed * Fix issue creating BigQuery connections causing the account connections list to not load correctly. * Fix for locked accounts that have downgraded to the developer plan at trial end * Fix for not properly showing server error messages on the user invite page ###### Changed * Deployed a number of Studio IDE visual improvements * Batch logs up every 5 seconds instead of every second to improve database performance * Make `retries` profile configuration for BigQuery connections optional * Support `retries` profile configuration for BigQuery connections (new in dbt v0.15.1) * Replace Gravatar images with generic person icons in the top navbar * Remove deprecated account subscription models * Remove external JS dependencies ###### Removed * Remove the "read only" role (this is now a "read only" license type) * Remove the "standard" license type * Remove "beta" tag from Studio IDE * Remove unused frontend code (team page/create repository page and related services) ##### Self-Service ###### Fixed * Fix for locked accounts that have downgraded to the developer plan at trial end ###### Added * New Plans page * Add a 14 day free trial * Add the ability to provision a new repository via dbt * New Invite Team step for project setup process for trial accounts ###### Changed * The "Basic" and "Pro" plans are no longer available. The new "Developer" and "Team" plans are available. * Prorations are now charged immediately, instead of applied to the next billing cycle. * It is no longer possible to downgrade to a plan that does not support the current number of allocated seats. * A "Team" plan that has been cancelled will be locked (closed) at the end of the subscription's period ##### On-Premises ###### Added * Support custom SMTP settings * Support Azure Blob Storage for run logs + artifacts * Optionally disable anonymous usage tracking *** #### dbt Cloud v0.5.0 (December 19, 2019) This release preps dbt for the general Studio IDE release in January. Beta Studio IDE functionality can be turned on by checking "Develop file system" in the Accounts page in the dbt backend. ##### All versions ###### Added * New dbt version: 0.14.2 * New dbt version: 0.14.3 * New dbt version: 0.14.4 * New dbt version: 0.15.0 * New API endpoint: v3/projects * New API endpoint: v3/credentials * New API endpoint: v3/environments * New API endpoint: v3/events * Studio IDE: Add git workflow UI * Studio IDE: Add filesystem management * Studio IDE: Hup the server when files change * Studio IDE: Display server status and task history * Added development and deployment environments and credentials * Support `--warn-error` flag in dbt runs ###### Fixed * Fixed an issue where the run scheduler would hang up when deleting PR schemas * Fixed an issue where the webhook processor would mark a webhook as processed without queuing a run * Fix a bug where SSH tunnels were not created for the Develop Studio IDE * Fix Develop Studio IDE scrolling in Firefox * Fix a bug where requests were timed out too aggressively * Require company name at signup * Fix security issue where IP blacklist could be bypassed using shorthand * Do a better job of handling git errors * Allow users to delete projects ###### Changed * Move account picker to sidebar * Increase require.js timeout from 7s to 30s * Migrate environments to projects * Move some UIs into Account Settings * Make cron scheduling available on the free tier * Apply new styles to Studio IDE * Speed up develop --- ### Changelog 2021 note This changelog references dbt versions that are no longer supported and have been removed from the docs. For more information about upgrading to a supported version of dbt in your dbt environment, read [Upgrade dbt version in Cloud](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md). Welcome to the 2021 changelog for the dbt application! You can use this changelog to see highlights of what was new, fixed, and enhanced. #### dbt Cloud v1.1.41 (December 8, 2021) It's one of the best weeks of the year - it's [Coalesce](https://coalesce.getdbt.com/)! We'll have some exciting product announcements to share! Did somebody say [metrics](https://coalesce.getdbt.com/talks/keynote-metric-system/) and [dbt v1](https://coalesce.getdbt.com/talks/dbt-v10-reveal/)?! ###### New products and features * dbt v1.0 is now available in dbt... nbd. ###### Performance improvements and enhancements * Now whenever you log back into dbt, you'll return to the account and project that you most recently were working in! #### dbt Cloud v1.1.39 (November 10, 2021) We shipped environment variables in dbt. Environment variables create a way to separate code from configuration - allowing you to set config based on context and keep secrets like git tokens securely stored. ###### New products and features * You can now add environment variables to your dbt project. Why does this matter? Environment variables are a fundamental building block of a dbt project, which until now, we only enabled in dbt v1. They power many use cases such as cloning private packages, limiting the amount of data that is processed in development environments, changing your data sources depending on the environment, and more. Read about environment variables in our [blog post](https://blog.getdbt.com/introducing-environment-variables-in-dbt-cloud/) or [docs](https://docs.getdbt.com/docs/build/environment-variables.md). #### dbt Cloud v1.1.38 (October 27, 2021) Have you used the [Metadata API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md) yet? The Metadata API is available to customers on the Team and Enterprise plans, and with it, you can learn tons about your dbt project, if it's running dbt v0.19.0 or later. You can now query information about *any* run, not just the last run of a job. Mo' data, mo' fun! #### dbt Cloud v1.1.37 (October 13, 2021) dbt v0.21 is now available in dbt Cloud. The big change with this release is it introduces the `dbt build` command. `dbt build` logically does everything you'd want to do in your DAG. It runs your models, tests your tests, snapshots your snapshots, and seeds your seeds. It does this, resource by resource, from left to right across your DAG. dbt build is an opinionated task. It’s the culmination of all we’ve built- running models with resilient materializations, prioritizing data quality with tests, updating fixtures with seeds, capturing slowly changing dimensions with snapshot. Give it a try! ###### New products and features * We have a new beta feature, which we're calling Model Bottlenecks. It allows you to visually see how long it takes to build models in each run, so you can see clearly which models are taking the longest. If you're interested in learning more, check out #beta-feedback-model-bottlenecks in the dbt community Slack, and we can add you to the beta. #### dbt Cloud v1.1.36 (September 29, 2021) Check out the release candidate for `dbt v0.21.0`! Also tab switching in the dbt Cloud IDE now keeps track of your scroll position - at last! ###### Bug fixes * Some Redshift customers were experiencing timeouts on runs. We've since fixed this bug by keeping the session alive longer. ###### Performance improvements and enhancements * You won't lose track of the code snippets you were looking at when you switch back and forth between tabs in the dbt Cloud IDE, as we now keep track of your scroll position. #### dbt Cloud v1.1.35 (September 15, 2021) Have you ever been working in the Studio IDE, taken a several hour break from developing, and when you returned to your work, the Studio IDE started behaving in unexpected ways? Your develop session became inactive, without any notification. Well, that silent failure won’t happen anymore! dbt now will let you know when you have to refresh your Studio IDE so you can continue to pick up work where you last left off. ###### New products and features * dbt v0.20.2 is released in dbt. ###### Performance improvements and enhancements * Set default threads to 4 for new jobs and in development creds. ###### Bug fixes * The user is now prompted to refresh the page when in a disconnected Studio IDE state. * dbt tasks that fail or error are now correctly ordered in the run drawer history. #### dbt Cloud v1.1.34 (September 1, 2021) We just launched our beta for supporting environment variables in dbt. Environment variables are exciting because they allow you to clone private packages. If you’re interested in joining the beta, check out the #beta-feedback-for-env-vars channel in dbt Slack for more information. ###### Performance improvements and enhancements Our Studio IDE SQL drawer got a fresh new look, and it now has improved accessibility. #### dbt Cloud v1.1.33 (August 18, 2021) We added a DAG in the Studio IDE, so that you can see your model dependencies as you develop! If you haven’t seen the DAG visualization yet, take a moment to spin up the Studio IDE, navigate to the Lineage tab, and click-click-click around in there — it is legitimately a brand new modality for developing dbt projects, and it’s something worth being excited about! ###### New products and features * [Dashboard Status Tiles](https://docs.getdbt.com/docs/explore/data-tile.md) can now be embedded on dashboards (or anywhere you can embed an iFrame) to give immediate insight into data freshness and quality. This helps dbt project maintainers build trust internally about the data that end users are seeing. * We shipped DAG in the Studio IDE to GA! * Support for `dbt v0.20.1` in Cloud. ###### Bug fixes * Databricks users will now be able to see and update the token/schema for deployment environments. * Some Github users were experiencing a broken profile image in dbt. This should be fixed if users disconnect and reconnect their Github accounts. #### dbt Cloud v1.1.32 (August 4, 2021) The Metadata API is now in GA! When dbt invokes certain commands like run, test, seed, etc, dbt generates metadata in the form of [artifacts](https://docs.getdbt.com/reference/artifacts/dbt-artifacts.md). These artifacts give you tons of information about project set up, run times, test details, compiled SQL, and so much more. Now dbt serves a GraphQL API which supports arbitrary queries over these artifacts, so you can retrieve the metadata you want almost instantaneously. ###### New products and features * The Metadata API is the start of our metadata product suite. Learn more about how to use the Metadata API [here](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md). * dbt Enterprise customers using GitHub now get better fine-grained access control in their dbt projects. dbt will enforce git permissions for every developer to ensure that read / write policies in GitHub carry through to the IDE. #### dbt Cloud v1.1.31 (July 21, 2021) We’ve improved the tabbing experience in the Studio IDE. Tabs now work much more intuitively, and you don’t have to worry about losing your work anymore! ###### New products and features * We are working to release a DAG directly in the IDE, so that when you’re developing, you have a clear idea of where the model you’re working on sits in the dependency graph. If you’re interested in testing out the feature early, head over to the `#beta-feedback-for-ide-dag` channel in the dbt Slack, and we’ll get the new product feature-flagged on your account! * Added dbt 0.20.0 to Cloud ###### Bug fixes * Users will now be able to initialize any project that doesn't contain a `dbt_project.yml` file, regardless of whether or not there are pre-existing files and/or commits to that repo. ###### Performance improvements and enhancements * We've been working on some nice improvements to tabs in our Studio IDE. We’ve fixed deficiencies with tabs that caused users to lose work if they didn’t hit save regularly enough. Additionally, opening, closing, and the order of the tabs work much more smoothly. * You may have noticed that there is now a source freshness checkbox in your execution settings when you configure a job on dbt Cloud. Selecting this checkbox will run `dbt source freshness` as the first step in your job, but it will not break subsequent steps if it fails. Updated source freshness documentation available [here](https://docs.getdbt.com/docs/deploy/source-freshness.md). * Added a new endpoint to allow API key rotation via `POST https://cloud.getdbt.com/api/v2/users/{user-id}/apikey` #### dbt Cloud v1.1.30 (July 7, 2021) We shipped a resizable folder pane in the Studio IDE, and we're hearing great things! "My quality of life has greatly increased with this little update!" Hope this helps everyone else enjoy the Studio IDE a little more too. ###### New products and features * Resizable folder pane in the Studio IDE: Have you ever developed in the Studio IDE and not been able to see the full name of your model because you couldn’t adjust the width of the file pane? Yeah, us too. Now you’ll be able to adjust your project’s file tree width to be as wide or as narrow as you’d like. It’s these small things that make developing in the Studio IDE so much easier. ###### Bug fixes * Made some changes to GitLab webhooks so that the status of the dbt run gets properly updated in GitLab. * Resolved an issue where users saw a blank screen rather than the SSO reauthentication page. ###### Performance improvements and enhancements * Refreshed the design of the repository import page. #### dbt Cloud v1.1.29 (June 23, 2021) We're heads down working on a handful of new features that we're going to share at the end of this month. The finish line is in sight. In the meantime, check out our latest release candidates for dbt v1. The biggest changes are better tests, providing consistency, configurability, and persistence. ###### New products and features * Add support for latest Core release candidates to dbt: v0.19.2-rc2 and v0.20.0-rc1 ###### Bug fixes * Add a safeguard for the SSO reauth page to avoid 401 interceptors ###### Performance improvements and enhancements * Ensure navigation bar is in dark mode when Studio IDE is set to dark mode #### dbt Cloud v1.1.28 (June 9, 2021) We shipped a far better experience for GitLab users. Be sure to check out new CI features that are now available for customers using GitLab. Additionally, all developers should test out Slim CI which will speed up their model builds. ###### New products and features * `Slim CI`: We’ve made Slim CI available for all our cloud customers! With Slim CI, you don't have to rebuild and test all your models; you can instruct dbt Cloud to run jobs on only modified or new resources. If you are a GitHub or GitLab user, try creating a new job that runs on pull requests and you can signal to dbt to run only on these modified resources by including the `state:modified+` argument. Read more about Slim CI [here](https://docs.getdbt.com/docs/deploy/continuous-integration.md). * Native GitLab authentication for dbt Developer and Team Tiers: We’ve shipped native GitLab auth into GA. You can now import new GitLab repos with a couple clicks, trigger CI builds when Merge Requests are opened in GitLab, and carry GitLab permissions through to Studio IDE's git actions. Read how to set up native GitLab auth [here](https://docs.getdbt.com/docs/platform/git/connect-gitlab.md). ###### Bug fixes * Allow users to select artifacts from a job that runs source freshness on jobs with the source freshness execution settings set to `ON`. * Resolve `RUN ONLY ON CUSTOM BRANCH?` button to toggle on and off properly. * Retain information in a `Statement` tab when the page is refreshed. * Unsaved changes in the Studio IDE are now saved when committing work. * Drop temporary schemas in the data warehouse for closed or merged GitLab merge requests. ###### Performance improvements and enhancements * Behind the scenes, we’ve been moving off of Angular and onto React. We’ve started the process of migrating the central pieces of our UI over - the first of which is the main navigation. We think this will have a big impact on our ability to reduce UI bugs and improve user experience. * Added support for dbt 0.19.2rc2 + 0.20.0rc1 in dbt. #### dbt Cloud v1.1.27 (May 26, 2021) A lot of improvements coming for GitLab webhooks and native auth. We also fixed a number of bugs in the Studio IDE. Our goal is for you to never see an infinite spinner again! ###### Enhancements * Add dbt v0.19.2rc1 and v0.20.0b1 * Add an open/closable overlay for the DAG * Disable department dropdown * Add DAG flags, button, and tab context * Add run source freshness option to jobs * Implement conditional redirecting after GitLab app integration * Add Develop Pod Support for Rook and Ceph file storage * Show all common actions for valid top level commands ###### Fixed * Fix link to documentation * Disable the "Restart Studio IDE" Button while the Studio IDE is loading * Continue canceling runs when we run into deleted accounts * Fix SSO re-auth page * Fix blank verify email page * Resolve git refresh regression * Fix missing "Run on Merge" button in Job creation/edit form- * Warn users they have unsaved changes * Updates test command suggestions and regex for common action suggestions * Updates order of stylesheet import to fix missing border bug * Fix GitLab PR link for Run Page * Fix infinite spinner for missing environment or user credentials * Fix infinite spinner when user is missing dev credentials * Do not try to push if awaiting a merge * Fix deleting schemas * Fix favicon reference #### dbt Cloud v1.1.26 (May 12, 2021) If you haven't seen it yet, spin up the Studio IDE: the command bar now has recent actions (you can up-arrow like on the command line) as well as some hardcoded suggestions that will auto-populate your active model, if there is one. Check it out! Other fixes and adjustments as well, as we all get ready for Staging this Thursday - exciting week for the Product org over at ol' Fishtown! ###### Enhancements * Made dbt default version on env 0.19.1 * Rolled out new command line experience to all customers * Post webhook triggered run status back to gitlab * Temporary tabs can also populate the model from manifest * Check command line content is minimally valid * Allow user to restart server when develop pod crashes * Prevent overflow of menu items ###### Fixed * Handle validation error for improper remote URLs in the Scheduler * Refactor exception logging out of GitRepo and into exception handlers * Required tags returning null from core no longer causing infinite spinner * Removed deleted repos while fetching repository for sending commit statuses * Refactor git provider service * Resolve files with special characters becoming forever dirty * Disable input when RPC command running & add button when command bar is empty * Updating button for the Cancel/Enter button on commandline * Fix connection setup to always use the project referenced in the route * Fix "View data sources" URL in environment page * Add support for clicking on previously run commands and updating the text inside of the commandline * Fix sources URL in environments page * Fix metadata token not allowed API response #### dbt Cloud v1.1.25 (April 28, 2021) Exciting things coming down the pipe - ongoing enhancements to the command bar experience in the Studio IDE, doing some work to ensure that more git providers are presented with a first class experience in Cloud, as well as assorted bug fixes - "I must have bug fixes, always and always" - that was Monet I think ###### Enhancements * Made a grip of visual updates to the new command bar work * Moved to using the active model name instead of a placeholder in command bar work * Added user ability to delete connections, remove association from a given project. * Added verification of dbt version for command bar beta feature flag ###### Fixed * Removed testing prop that keeps drawer open * Added double encoding to handle Snowflake roles with spaces * Fixed account switching in user notifications * Handled invalid Azure SSO group responses * Fixed error which only showed common actions when run drawer was closed * Allowed unencrypted adapter fields to be edited * Fixed bugs with file and folder renaming, alongside associated tab state #### dbt Cloud v1.1.24 (April 14, 2021) Phew! As our company grows, so too does our changelog! Look at all these! The big chunks you'll see here are related to some ongoing in-Studio IDE work, focused on the command bar experience, as well as some partner & connection work (see the Gits, Databricks, and so forth), and of course ongoing longer-term bets around metadata! ###### Enhancements * Added onFocus and onBlur properties to populate and remove "dbt" in command bar * Enabled executing command on enter if user's cursor is in the command bar * Added Metadata API access button to account settings * Added feature flag for displaying only recent actions * Added dbt 0.19.1 * Added regex validation to Databrick's hostname web-form field * Updated Connection Edit to allow adapter editing * Enabled self-service Github and GitLab integration disconnection * Added link to docs for license map & handle duplicate error gracefully * Moved deferred job execution to execution settings. * Recorded user command history * Enabled new file creation flow ###### Fixed * Added styling class to popup to ensure text is readable * Fixed sourcemaps syntax for dev commands * Added timeout and retry to dbt deps * Updated databricks schema field type and add error handling to ConnectionSetup * Fixed Bigquery private keys & convert text to textarea * Fixed last used datetime in the service token UI * Added missing token URI to Bigquery connection edit * Prevent multiple develop sessions for one user * Fixed SchemaForm validating non-displayed fields * Fixed required fields for Bigquery connection JSON uploads * Fixed self selection as deferred job * Always create a Monaco model on tab open if no matching model exists * Fixed tab dirty indicator on open tab * Fixed password reset flow * Fixed docs and sources links in dashboard page for read only users * Fixed truncating first\_name to 30 characters #### dbt Cloud v1.1.23 (March 31, 2021) Some backend work, some frontend work, some bug fixes: a nice mix for this release. A few user facing changes you may have noticed already are the persistence of dark/light mode settings across refresh (no more blinding Studio IDE!), branches in the Studio IDE being categorized by Active vs. Removed from Remote, and a tidier new file creation flow, with the file tree expanding to show the new file and opening a new tab to populate the said file! ###### Enhancements * Splitting Local-only and Remote branches into different sections of the dropdown selector * Update Profile Integrations to include SSO info * Upgrade to Tailwind 2.0 and FUI 0.0.5 * Allow users to create metadata tokens from the UI * Support manually-managed group memberships * SSO: resolve bug w/ first & last names acting up * Integrate Delighted for NPS surveys * Add dbt 0.19.1rc1 to Cloud * Add an account-level setting to require users to re-authenticate via SSO * Read-only metadata ServiceToken for Cloud * Persist Studio IDE light mode / dark mode across refresh * Categorize & order git branches * Improve new file creation flow ###### Fixed * Check for an empty repository before checking matching remote * Increase wait if run was finished recently * Support default branches through git when a custom branch is not specified * Don't download logs for skipped steps * API Gateway is no longer flooded with errors due to Studio IDE blindly polling dead Develop pod * Fix user license creation via admin interface * Adjusted addition of global .gitignore #### dbt Cloud v1.1.22 (March 17, 2021) Rolling out a few long-term bets to ensure that our beloved dbt does not fall over for want of memory, as well as a grip of bug fixes and error messaging improvements (error messages should be helpful, not scolding or baffling, after all!) ###### Enhancements * Release Scribe to 100% of multi-tenant accounts * Update language for SQL drawer empty state * Reduce Scribe memory usage ###### Fixed * Fix NoSuchKey error * Guarantee unique notification settings per account, user, and type * Fix for account notification settings * Don't show deleted projects on notifications page * Fix unicode error while decoding last\_chunk * Show more relevant errors to customers * Groups are now editable by non-sudo requests * Normalize domain names across inputs/outputs * Redirect auth failed errors back to appropriate page with error description #### dbt Cloud v1.1.21 (March 3, 2021) This changelog wraps up work on what we've been calling the SQL Drawer in the Studio IDE - some design nudges, some interface adjustments, overall a cleaner and snappier experience. If you haven't dipped into the Studio IDE in a while it's worth taking a look! Some back-end work as well, making SSO and role based admin easier and more broadly available for Enterprise level folks, along with your usual assortment of bug squashes and iterations. ###### Enhancements * Styling and copy adjustments in the Cloud Studio IDE * Open self-service role based access control to all Enterprise customers * Update AuthProvider UI to enable SAML and Okta * Add a SAML auth redirect URL ###### Fixed * Add param to admin project mapper to included soft deleted projects * Fix delaying logs when we are waiting for a model to finish executing * Saving GSuite auth provider form triggers an authorize * Scribe populates truncated debug logs when runs are executing * Delay attempts for non-200 status codes * Add logic to support select fields in adapter UI * Undo clobbering groups #### dbt Cloud v1.1.20 (February 17, 2021) Continued stability and quality of life improvements for folks with multiple accounts and projects - no longer will you have to remember the chronological order of birth of your accounts and projects, as they'll be ordered by the much easier to parse (for human brains anyway) alphabetical order. We're also shipping some experience improvements in the SQL Drawer at the bottom half of the Studio IDE. ###### Enhancements * Deleted Info and Logs Studio IDE Tabs, logs will now be displayed in Results Tab * Removed service token feature flag * List Jobs dropdown in alphabetical order * List Account and Project dropdowns in alphabetica order * Pre-join Job Definition results to speed up scheduler * Combine scheduler queries to speedup runtime by about 30% ###### Fixed * Fix issue with source freshness for 0.19.0 #### dbt Cloud v1.1.19 (February 3, 2021) The latest release of dbt (Oh Nineteen Oh) is now available for your enjoyment on dbt! We're also releasing some service token pieces here, though they're not quite ready for wide release yet. Moving forward, Oh Nineteen Oh will probably end up being the minimum version required to run the Metadata API & Metadata Toolkit, so, this is a big release! ###### Enhancements * Added dbt 0.19.0 😻 * Allowed account-wide service tokens to create connections * Added integration for service token UI and API * Authorized requests that supply a service token ###### Fixed * Added logic to show the entered service token name prior to the request completing * Fixed endlessly running rpc queries with non-working cancel button on Studio IDE refresh #### dbt Cloud v1.1.18 (January 20, 2021) Most notable things here are around foundational work toward future feature releases, as well as strong assurances of future stability for dbt, and ensuring future sales tax compliance (which we understand turns out to be quite important!) - turns out to be a quite future-looking release! ###### Enhancements * Add service tokens UI (stubbed) behind a feature flag * Fixing and Upgrading social-auth * Add dbt Spark 0.19.0rc1 * Adds the reconciliation of persisted file content and tab state when navigating into the Studio IDE * Adds the reconciliation of persisted file content and tab state between Studio IDE sessions * Read logs from scribe and stop logging to db * Upgrade social auth 3.3.3 * Add warning logs for social auth failures * Add dbt 0.19.0rc1 ###### Fixed * Prevent social-auth from updating first or last name * Page through Stripe results when listing subscriptions * Prevent enqueueing runs in deleted projects * Fix Studio IDE git actions causing open tab contents to be lost on Studio IDE re-entry * Add DBT\_CLOUD\_CONTEXT environment variable * Add logic to hide IP whitelist message for on-prem customers * fix 0.19.0rc1 run image dependencies --- ### Column-level lineage dbt platform | Enterprise, Enterprise+ Catalog now offers column-level lineage (CLL) for the resources in your dbt project. Analytics engineers can quickly and easily gain insight into the provenance of their data products at a more granular level. For each column in a resource (model, source, or snapshot) in a dbt project, Catalog provides end-to-end lineage for the data in that column given how it's used. CLL is available to all dbt Enterprise plans that can use Catalog. [![Overview of column level lineage](/img/docs/collaborate/dbt-explorer/example-overview-cll.png?v=2 "Overview of column level lineage")](#)Overview of column level lineage On-demand learning If you enjoy video courses, check out our [dbt Catalog on-demand course](https://learn.getdbt.com/courses/dbt-catalog) and learn how to best explore your dbt project(s)! #### Access the column-level lineage There is no additional setup required for CLL if your account is on an Enterprise plan that can use Catalog. You can access the CLL by expanding the column card in the **Columns** tab of an Catalog [resource details page](https://docs.getdbt.com/docs/explore/explore-projects.md#view-resource-details) for a model, source, or snapshot. dbt updates the lineage in Explorer after each run that's executed in the production or staging environment. At least one job in the production or staging environment must run `dbt docs generate`. Refer to [Generating metadata](https://docs.getdbt.com/docs/explore/explore-projects.md#generate-metadata) for more details. [![Example of the Columns tab and where to expand for the CLL](/img/docs/collaborate/dbt-explorer/example-cll.png?v=2 "Example of the Columns tab and where to expand for the CLL")](#)Example of the Columns tab and where to expand for the CLL #### Column evolution lens You can use the column evolution lineage lens to determine when a column is transformed vs. reused (passthrough or rename). The lens helps you distinguish when and how a column is actually changed as it flows through your dbt lineage, informing debugging workflows in particular. [![Example of the Column evolution lens](/img/docs/collaborate/dbt-explorer/example-evolution-lens.png?v=2 "Example of the Column evolution lens")](#)Example of the Column evolution lens ##### Inherited column descriptions A reused column, labeled as **Passthrough** or **Rename** in the lineage, automatically inherits its description from the source and upstream model columns. The inheritance goes as far back as possible. As long as the column isn't transformed, you don't need to manually define the description; it'll automatically propagate downstream. Passthrough and rename columns are clearly labeled and color-coded in the lineage. In the following `dim_salesforce_accounts` model example (located at the end of the lineage), the description for a column inherited from the `stg_salesforce__accounts` model (located second to the left) indicates its origin. This helps developers quickly identify the original source of the column, making it easier to know where to make documentation changes. [![Example of lineage with propagated and inherited column descriptions.](/img/docs/collaborate/dbt-explorer/example-prop-inherit.png?v=2 "Example of lineage with propagated and inherited column descriptions.")](#)Example of lineage with propagated and inherited column descriptions. #### Column-level lineage use cases Learn more about why and how you can use CLL in the following sections. ##### Root cause analysis When there is an unexpected breakage in a data pipeline, column-level lineage can be a valuable tool to understand the exact point where the error occurred in the pipeline. For example, a failing data test on a particular column in your dbt model might've stemmed from an untested column upstream. Using CLL can help quickly identify and fix breakages when they happen. ##### Impact analysis During development, analytics engineers can use column-level lineage to understand the full scope of the impact of their proposed changes. This knowledge empowers them to create higher-quality pull requests that require fewer edits, as they can anticipate and preempt issues that would've been unchecked without column-level insights. ##### Collaboration and efficiency When exploring your data products, navigating column lineage allows analytics engineers and data analysts to more easily navigate and understand the origin and usage of their data, enabling them to make better decisions with higher confidence. #### Caveats Refer to the following CLL caveats or limitations as you navigate Catalog. ##### Column usage Column-level lineage reflects the lineage from `select` statements in your models' SQL code. It doesn't reflect other usage like joins and filters. ##### SQL parsing Column-level lineage relies on SQL parsing. Errors can occur when parsing fails or a column's origin is unknown (like with JSON unpacking, lateral joins, and so on). In these cases, lineage may be incomplete and dbt will provide a warning about it in the column lineage. [![Example of warning in the full lineage graph](/img/docs/collaborate/dbt-explorer/example-parsing-error-pill.png?v=2 "Example of warning in the full lineage graph")](#)Example of warning in the full lineage graph To review the error details: 1. Click the **Expand** icon in the upper right corner to open the column's lineage graph 2. Select the node to open the column’s details panel Possible error cases are: * **Parsing error** — Error occurs when the SQL is ambiguous or too complex for parsing. An example of ambiguous parsing scenarios are *complex* lateral joins. * **Python error** — Error occurs when a Python model is used within the lineage. Due to the nature of Python models, it's not possible to parse and determine the lineage. * **Unknown error** — Error occurs when the lineage can't be determined for an unknown reason. An example of this would be if a dbt best practice is not being followed, like using hardcoded table names instead of `ref` statements. --- ### Community adapters Community adapters are adapter plugins contributed and maintained by members of the community. We welcome and encourage [adapter plugins contributions](https://docs.getdbt.com/docs/contribute-dbt-adapters.md#contribute-to-a-pre-existing-adapter) from the dbt community. Please be mindful that these [community maintainers](https://docs.getdbt.com/docs/connect-adapters.md#maintainers) are intrepid volunteers who donate their time and effort — so be kind, understanding, and help out where you can! In v2, [DuckDB](https://docs.getdbt.com/docs/local/connect-data-platform/duckdb-setup.md) is available as a community adapter. There are also several trusted adapters available: Apache Spark, BigQuery, Databricks, DuckDB, Redshift, Snowflake. For the full list, refer to [trusted adapters](https://docs.getdbt.com/docs/trusted-adapters.md?version=2.0). If you'd like to contribute a community adapter for v2, refer to [creating a v2 adapter](https://docs.getdbt.com/guides/adapter-creation-v2.md?step=1). Refer to the following available community adapter(s): | Data platform | | | | ------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | | [Confluent Cloud](https://docs.getdbt.com/docs/local/connect-data-platform/confluent-setup.md) | [CrateDB](https://docs.getdbt.com/docs/local/connect-data-platform/cratedb-setup.md) | [Databend Cloud](https://docs.getdbt.com/docs/local/connect-data-platform/databend-setup.md) | | [DeltaStream](https://docs.getdbt.com/docs/local/connect-data-platform/deltastream-setup.md) | [Doris & SelectDB](https://docs.getdbt.com/docs/local/connect-data-platform/doris-setup.md) | [DuckDB](https://docs.getdbt.com/docs/local/connect-data-platform/duckdb-setup.md) | | [Extrica](https://docs.getdbt.com/docs/local/connect-data-platform/extrica-setup.md) | [Hologres](https://docs.getdbt.com/docs/local/connect-data-platform/hologres-setup.md) | [IBM watsonx.data - Spark](https://docs.getdbt.com/docs/local/connect-data-platform/watsonx-spark-setup.md) | | [Infer](https://docs.getdbt.com/docs/local/connect-data-platform/infer-setup.md) | [iomete](https://docs.getdbt.com/docs/local/connect-data-platform/iomete-setup.md) | [MaxCompute](https://docs.getdbt.com/docs/local/connect-data-platform/maxcompute-setup.md) | | [MindsDB](https://docs.getdbt.com/docs/local/connect-data-platform/mindsdb-setup.md) | [MySQL](https://docs.getdbt.com/docs/local/connect-data-platform/mysql-setup.md) | [RisingWave](https://docs.getdbt.com/docs/local/connect-data-platform/risingwave-setup.md) | | [Rockset](https://docs.getdbt.com/docs/local/connect-data-platform/rockset-setup.md) | [Sail](https://docs.getdbt.com/docs/local/connect-data-platform/sail-setup.md) | [SingleStore](https://docs.getdbt.com/docs/local/connect-data-platform/singlestore-setup.md) | | [SQL Server & Azure SQL](https://docs.getdbt.com/docs/local/connect-data-platform/mssql-setup.md) | [SQLite](https://docs.getdbt.com/docs/local/connect-data-platform/sqlite-setup.md) | [Starrocks](https://docs.getdbt.com/docs/local/connect-data-platform/starrocks-setup.md) | | [TiDB](https://docs.getdbt.com/docs/local/connect-data-platform/tidb-setup.md) | [TimescaleDB](https://dbt-timescaledb.debruyn.dev/) | [Upsolver](https://docs.getdbt.com/docs/local/connect-data-platform/upsolver-setup.md) | | [Vertica](https://docs.getdbt.com/docs/local/connect-data-platform/vertica-setup.md) | [Watsonx-Presto](https://docs.getdbt.com/docs/local/connect-data-platform/watsonx-presto-setup.md) | [Yellowbrick](https://docs.getdbt.com/docs/local/connect-data-platform/yellowbrick-setup.md) | --- ### Configure BYOK for dbt Wizard Beta ### Configure BYOK for dbt Wizard [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Local development You can use the dbt Wizard CLI with bring-your-own-key (BYOK), which means you supply your own credentials from a supported AI provider instead of using dbt Labs' infrastructure. The following BYOK instructions on this page apply to the CLI only. For dbt platform BYOK setup, refer to [Configure BYOK for dbt Wizard in dbt platform](https://docs.getdbt.com/docs/platform/wizard-byok-platform.md). If you'd rather skip that upkeep, the [dbt Labs-managed option](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md#configure-ai-provider) is ready to use with no setup, and comes with [trial and monthly usage credits](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/trial-and-billing.md). The "key" in BYOK is whatever credential your chosen provider uses to authenticate API requests — an API key for OpenAI or Anthropic, a bearer token for AWS Bedrock, or a token/PAT for Snowflake Cortex. When you configure a provider with that credential, dbt Wizard calls the provider's API directly using it, so: * Usage costs appear on your provider account, not your dbt Labs account. * Token costs are billed by whichever provider you choose. dbt Wizard supports [managed models](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/overview.md#dbt-managed-providers) (billed by dbt Labs, no key to manage) and [bring-your-own-key (BYOK)](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) models (billed directly by your provider). Here are the following AI providers supported depending on where you work. Refer to [Model Provider Rate table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) for the full list of available models. ##### dbt platform | Provider | Access | | ---------------------------------------------------------------------------- | ------------------- | | [OpenAI](https://openai.com/policies/row-terms-of-use/) (default) | dbt managed or BYOK | | [Anthropic](https://www.anthropic.com/legal/consumer-terms)† | dbt managed or BYOK | | Open weight models (like DeepSeek, Kimi, and so on). | dbt managed | | [Azure AI Foundry](https://www.microsoft.com/licensing/terms) / Azure OpenAI | BYOK | ##### Locally (CLI) | Provider | Access | | ---------------------------------------------------------------------------- | ------------------- | | [OpenAI](https://openai.com/policies/row-terms-of-use/) | dbt managed or BYOK | | [Anthropic](https://www.anthropic.com/legal/consumer-terms)† | dbt managed or BYOK | | Open weight models (like DeepSeek, Kimi, and so on). | dbt managed | | [Azure AI Foundry](https://www.microsoft.com/licensing/terms) / Azure OpenAI | BYOK | | [AWS Bedrock](https://aws.amazon.com/service-terms/) | BYOK | | [Google Gemini](https://ai.google.dev/gemini-api/terms) | BYOK | | [Snowflake Cortex](https://www.snowflake.com/en/legal/terms-of-service/) | BYOK | | [Databricks Unity AI Gateway](https://www.databricks.com/legal/mcsa) | BYOK | You can also connect a personal OpenAI ChatGPT subscription instead of a key. †Anthropic enterprise and subscription licenses (such as Claude Enterprise) aren't supported per Anthropic's [terms of service](https://www.anthropic.com/legal/consumer-terms). See it in action and share your feedback Want to see dbt Wizard in action? Check out the [demo video](https://www.youtube.com/watch?v=-lIzh1xQWMA). We'd love to hear how dbt Wizard is working for you. Share your feedback by either running the `/feedback` slash command in your interactive terminal session or by going to the [#dbt-wizard](https://getdbt.slack.com/archives/C0B6KLW6T26) channel in the [dbt Community Slack](https://docs.getdbt.com/community/join?version=2.0). Thanks so much for your help in improving dbt Wizard and dbt data development! #### Configure a provider You can configure a provider in one of the following ways: * [**Terminal commands**:](#configure-in-the-terminal) Best for users who want to configure providers from the shell. * [**Interactive session**:](#configure-in-the-tui) Best for most users working in the dbt Wizard text based user interface (TUI). Use the `/providers` slash command. * [**Environment variables**:](#set-your-api-key) Best for headless runs, such as `wizard exec`, automation, or temporary local sessions. ##### Configure in the terminal In the terminal, use the `providers` subcommand to list, configure, and enable providers: ```bash wizard providers list wizard providers configure PROVIDER_NAME wizard providers enable PROVIDER_NAME ``` Replace `PROVIDER_NAME` with the name of a supported provider, such as `openai`, `anthropic`, `bedrock`, `azure`, `gemini`, `snowflake`, or `databricks`. Then, follow the prompts to enter your credentials. The `wizard providers list` command shows you the currently configured providers and their status: ```bash ➜ jaffle-shop git:(mwong-fusion) wizard providers list provider enabled route auth models dbt true remote dbt 3 openai false local missing 3 openai_subscription false local missing 1 anthropic false local missing 3 bedrock true local configured 15 azure false local missing 3 snowflake false local missing 3 gemini false local missing 3 ``` The `configure` command prompts you to enter credentials for the selected provider. To use environment variables or set a key without echoing it in your shell history, refer to [Set your API key](#set-your-api-key). So for example, if you're using OpenAI, you would run and follow the prompts to configure it: ```bash wizard providers configure openai wizard providers enable openai ``` To store an API key without echoing it in your shell history: ```bash printf '%s' 'sk-...' | wizard providers set-key PROVIDER_NAME ``` Credentials are stored in `~/.dbt/wizard/provider-auth.json`. Provider settings are stored in `~/.dbt/wizard/providers.json`. ##### Configure in the TUI You can configure providers from an active CLI TUI session with the `/providers` slash command: ```bash /providers ``` From the provider menu, you can: * Enable or disable a provider. * Add or update credentials. * Select available models. * Check whether a provider is authenticated and active. Example provider menu: ```bash /providers Model Providers Select a provider to inspect or update it. › 1. dbt enabled; 3/3 models selected; uses dbt login; active 2. openai disabled; 3/3 models selected; missing credentials; needs setup 3. openai_subscription disabled; 1/1 models selected; not connected; needs setup 4. anthropic disabled; 3/3 models selected; authenticated; needs setup 5. bedrock enabled; 15/15 models selected; authenticated; active 6. azure disabled; 3/3 models selected; missing credentials; needs setup 7. snowflake disabled; 3/3 models selected; missing credentials; needs setup 8. gemini disabled; 3/3 models selected; missing credentials; needs setup ``` #### Set your API key ##### Interactive session The first time you start dbt Wizard in a project, onboarding prompts you to choose a provider. 1. At the **Configure a Provider** prompt, select your provider. 2. Paste your API key or provider credentials when prompted. 3. Choose an AI model to finish setup. To add or switch providers later from an active session, type `/providers` in the TUI. When you configure a provider through dbt Wizard, credentials are stored in `~/.dbt/wizard/provider-auth.json`. ##### Environment variable Set the key as an environment variable if you want to: * Run dbt Wizard in headless mode, such as with `wizard exec`. * Use a key for the current terminal [session](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md#sessions) only. * Reuse the same key across different terminal sessions. * Avoid storing credentials in the dbt Wizard config directory. ##### OpenAI ```bash export OPENAI_API_KEY="sk-..." ``` ##### Anthropic ```bash export ANTHROPIC_API_KEY="sk-ant-..." ``` ##### Amazon Bedrock ```bash export AWS_BEARER_TOKEN_BEDROCK="ABSK..." ``` ##### Azure AI Foundry ```bash export AZURE_API_KEY="..." ``` ##### Google Gemini ```bash export GOOGLE_API_KEY="..." ``` ##### Snowflake Cortex ```bash export SNOWFLAKE_API_KEY="..." ``` ##### Databricks ```bash export DATABRICKS_API_KEY="dapi..." export DATABRICKS_API_BASE="https://adb-1234567890.azuredatabricks.net" ``` To make an environment variable available across terminal sessions, add it to your shell profile, such as `.zshrc`, `.bashrc`, or equivalent. Avoid committing API keys to version control, project files, or shared configuration. To persist provider credentials, use one of the following options: * `wizard providers configure PROVIDER_NAME` * `wizard providers set-key PROVIDER_NAME` * The provider's environment variable #### Choose an AI model dbt Wizard CLI uses dbt Labs-managed models out of the box. You can switch to a BYOK model in the following ways: | Method | Description | Example | | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | Interactive session | Change the AI model interactively in the TUI. | `/model` | | Invocation flag | Change the AI model at invocation with the `-m` flag. | `wizard -m gpt-4o "refactor stg_orders to use incremental materialization"` | | Config file | Set a default AI model in the config file (`~/.dbt/wizard/config.toml`); applies to all future sessions. Run `wizard debug models` to list available model IDs. | `model = "gpt-4o"` | Restart the dbt Wizard CLI after changing the model. #### Examples The following examples use the same provider configuration flow described earlier, with provider-specific credential requirements. ##### AWS Bedrock AWS Bedrock is supported in the CLI only. dbt Wizard currently supports Bedrock through an Amazon Bedrock API key, not the full AWS credential chain. Ensure your AWS account has access to the Bedrock models you plan to use and that your Bedrock API key has permission to invoke them. ```bash export AWS_BEARER_TOKEN_BEDROCK="ABSK..." wizard providers enable bedrock wizard providers bedrock set-region us-east-1 wizard providers list wizard debug models ``` To set a default Bedrock model, add the model ID to `~/.dbt/wizard/config.toml`: ```toml model = "BEDROCK_MODEL_ID" ``` ##### Snowflake Cortex [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Snowflake Cortex BYOK support in the CLI is in preview. Availability and setup steps may change. Ensure your Snowflake account has the privileges required for Cortex large language model (LLM) functions. Refer to the [Snowflake Cortex documentation](https://docs.snowflake.com/en/user-guide/snowflake-cortex/overview). You can [authenticate](#authentication-options) with Snowflake Cortex using an API token or a Programmatic Access Token (PAT) (for SSO and Okta users). ```bash wizard providers list wizard providers enable snowflake wizard providers configure snowflake wizard providers list wizard debug models ``` The `wizard providers configure snowflake` command walks you through the following prompts: ```text Enable this provider? [Y/n]: Models to enable [1]: Snowflake account ID: Snowflake API base override (optional): Paste API key/token, or press enter to configure it later: ``` | Prompt | What to enter | | ------------------------------------------ | ---------------------------------------------------------------------------------------------------------- | | **Snowflake account ID** | Your Snowflake account identifier (for example, `myorg-myaccount`) | | **Snowflake API base override (optional)** | Leave blank — this is only needed for custom or private Snowflake endpoints | | **Paste API key/token** | Your authentication token — refer to [Authentication options](#authentication-options) in the next section | ###### Authentication options The key/token field accepts a regular API token or a Programmatic Access Token (PAT), depending on how your Snowflake account is configured. Both are entered in the same place — the **Paste API key/token** prompt in the terminal, or **Set key/token** (option 3) in the TUI. 1. In your Snowflake account, [generate your API or PAT token](https://docs.snowflake.com/en/user-guide/programmatic-access-tokens#label-pat-generate). 2. Select **Generate token** and copy the token value. 3. Go back to dbt Wizard and paste the PAT at the key/token prompt. To set a default Snowflake Cortex model, add the model ID to `~/.dbt/wizard/config.toml`: ```toml model = "SNOWFLAKE_CORTEX_MODEL_ID" ``` ##### Databricks Unity AI Gateway [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt Wizard connects to Databricks through the [Unity Catalog AI Gateway](https://docs.databricks.com/aws/en/ai-gateway/), so you bring your own models served from your Databricks workspace. Make sure the [serving endpoints](https://docs.databricks.com/en/machine-learning/model-serving/index.html) you plan to use are deployed and that your Databricks token has permission to query them. ```bash export DATABRICKS_API_KEY="dapi..." wizard providers configure databricks wizard providers enable databricks wizard providers list wizard debug models ``` The `wizard providers configure databricks` command first asks for your workspace URL, then for the serving endpoint name behind each model you enable: ```text Enable this provider? [Y/n]: Models to enable [1]: Databricks workspace URL (e.g. https://adb-1234567890.azuredatabricks.net): Endpoint name for databricks/claude-sonnet-4-6: Paste API key/token, or press enter to configure it later: ``` | Prompt | What to enter | | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | **Databricks workspace URL** | Your Databricks workspace URL (for example, `https://adb-1234567890.azuredatabricks.net`) | | **Endpoint name** | The serving endpoint name for each model (for example, `databricks-claude-sonnet-4-6`). dbt Wizard suggests a default endpoint name for each model | | **Paste API key/token** | Your Databricks personal access token (PAT) | By default, dbt Wizard maps the following models to Databricks serving endpoints. You can override any endpoint name during configuration: | Model | Default endpoint name | | ------------------- | ------------------------------ | | `claude-sonnet-4-6` | `databricks-claude-sonnet-4-6` | | `claude-opus-4-7` | `databricks-claude-opus-4-7` | | `claude-haiku-4-5` | `databricks-claude-haiku-4-5` | | `gpt-5.5` | `databricks-gpt-5-5` | | `gpt-5.4` | `databricks-gpt-5-4` | | `gpt-5.4-mini` | `databricks-gpt-5-4-mini` | To set a default Databricks model, add the model ID to `~/.dbt/wizard/config.toml`: ```toml model = "databricks/claude-sonnet-4-6" ``` #### Related docs * [Install dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-cli.md) * [Configuration reference](https://docs.getdbt.com/docs/dbt-ai/wizard-config.md) * [dbt Wizard in Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) --- ### Configure environment variables Local development dbt v2 automatically loads environment variables from a `.env` file in your current working directory (the folder you `cd` into and run dbt commands from in your terminal). This helps you manage credentials and settings without hardcoding them in your `profiles.yml` or exposing them in your shell history. #### Using a `.env` file 1. Create a `.env` file in your current working directory (typically at the root of your dbt project): ```env DBT_MY_DATABASE=my_database DBT_MY_SCHEMA=my_schema DBT_SECRET_KEY=my_secret_value ``` 2. Reference these variables in your `profiles.yml` using the [`env_var` Jinja function](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md): ```yaml my_profile: target: dev outputs: dev: type: snowflake account: my_account database: "{{ env_var('DBT_MY_DATABASE') }}" schema: "{{ env_var('DBT_MY_SCHEMA') }}" ``` 3. Run dbt commands normally. dbt v2 will automatically load the variables from the `.env` file. For example, running `dbtf debug` will show your connection using the values from `.env`: ```shell dbtf debug ... Debugging connection: "authenticator": "my_authenticator", "account": "my_account", "user": "my_user", "database": "my_database", # Loaded from DBT_MY_DATABASE in .env "schema": "my_schema", # Loaded from DBT_MY_SCHEMA in .env ``` note We recommend placing your `.env` file in the project root and running dbt commands from that location because the file is loaded *only* from your current working directory. It doesn't support the `--project-dir` flag or (Applies to dbt v1.11 and later) `DBT_ENGINE_PROJECT_DIR` environment variable, and dbt won't search your project root if you're running commands from a different directory location. ##### Precedence order When the same environment variable is defined in multiple places, dbt v2 uses the following precedence order (highest to lowest): 1. Shell environment — Variables set directly in your shell (for example, `export DBT_MY_VAR=value`) 2. `.env` file — Variables defined in the `.env` file in your current working directory This means environment variables set in your shell always override values from the `.env` file. tip Add `.env` to your `.gitignore` file to prevent sensitive credentials from being committed to version control. The `dbtf init` command automatically includes `.env` in the generated `.gitignore` file. For more details on managing environment variables locally, refer to [Configure your local environment](https://docs.getdbt.com/docs/configure-dbt-extension.md#set-environment-variables-locally). --- ### Configure incremental models Learn how to configure and optimize incremental models when developing in dbt. Snowflake column size change [Snowflake plans to increase](https://docs.snowflake.com/en/release-notes/bcr-bundles/un-bundled/bcr-2118) the default column size for string and binary data types in September 2026. `dbt-snowflake` versions below v1.10.6 may fail to build certain incremental models when this change is deployed.  Assess impact and required actions If you're using a `dbt-snowflake` version below v1.10.6 or have not yet migrated to a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) in the dbt platform, your adapter version is incompatible with this change and may fail to build incremental models that meet *both* of the following conditions: * Contain string columns with collation defined * Use the `on_schema_change='sync_all_columns'` config To check whether this change affects your project, run the following [list](https://docs.getdbt.com/reference/commands/list.md) command: ```bash dbt ls -s config.materialized:incremental,config.on_schema_change:sync_all_columns --resource-type model ``` * If the command returns `No nodes selected!`, no action is required. * If the command returns one or more models (for example, `Found 1000 models, 644 macros`), you may be impacted if those models have string columns that don't specify a width. In that case, upgrade to a version that includes the fix: * **dbt v1**: `dbt-snowflake` v1.10.6 or later. For upgrade instructions, refer to [Upgrade adapters](https://docs.getdbt.com/docs/local/install-dbt.md) in the dbt v1 installation instructions. * **dbt platform**: Any release track (**v1 Latest**, **v1 Compatible**, **v1 Extended**, or **v1 Fallback**). * **dbt v2**: v2.0.0. This ensures your incremental models can safely handle schema changes while maintaining required collation settings. Incremental models are built as tables in your data warehouse. The first time a model is run, the table is built by transforming *all* rows of source data. On subsequent runs, dbt transforms *only* the rows in your source data that you tell dbt to filter for, inserting them into the target table which is the table that has already been built. Often, the rows you filter for on an incremental run will be the rows in your source data that have been created or updated since the last time dbt ran. As such, on each dbt run, your model gets built incrementally. Using an incremental model limits the amount of data that needs to be transformed, vastly reducing the runtime of your transformations. This improves warehouse performance and reduces compute costs. #### Configure incremental materializations Like the other materializations built into dbt, incremental models are defined with `select` statements, with the materialization defined in a config block. ```sql {{ config( materialized='incremental' ) }} select ... ``` To use incremental models, you also need to tell dbt: * How to filter the rows on an incremental run * The unique key of the model (if any) ##### Understand the is\_incremental() macro The `is_incremental()` macro powers incremental materializations. It will return `True` if *all* of the following conditions are met: * The model must already exist as a table in the database * The `full-refresh` flag *is not* passed * The running model is configured with `materialized='incremental'` Note that the SQL in your model needs to be valid whether `is_incremental()` evaluates to `True` or `False`. ##### Filtering rows on an incremental run To tell dbt which rows it should transform on an incremental run, wrap valid SQL that filters for these rows in the `is_incremental()` macro. Often, you'll want to filter for "new" rows, as in, rows that have been created since the last time dbt ran this model. The best way to find the timestamp of the most recent run of this model is by checking the most recent timestamp in your target table. dbt makes it easy to query your target table by using the "[{{ this }}](https://docs.getdbt.com/reference/dbt-jinja-functions/this.md)" variable. Also common is wanting to capture both new and updated records. For updated records, you'll need to [define a unique key](#defining-a-unique-key-optional) to ensure you don't bring in modified records as duplicates. Your `is_incremental()` code will check for rows created *or modified* since the last time dbt ran this model. For example, a model that includes a computationally slow transformation on a column can be built incrementally, as follows: models/stg\_events.sql ```sql {{ config( materialized='incremental' ) }} select *, my_slow_function(my_column) from {{ ref('app_data_events') }} {% if is_incremental() %} -- this filter will only be applied on an incremental run -- (uses >= to include records whose timestamp occurred since the last run of this model) -- (If event_time is NULL or the table is truncated, the condition will always be true and load all records) where event_time >= (select coalesce(max(event_time),'1900-01-01') from {{ this }} ) {% endif %} ``` Optimizing your incremental model For more complex incremental models that make use of Common Table Expressions (CTEs), you should consider the impact of the position of the `is_incremental()` macro on query performance. In some warehouses, filtering your records early can vastly improve the run time of your query! ##### About incremental\_predicates `incremental_predicates` is an advanced use of incremental models, where data volume is large enough to justify additional investments in performance. This config accepts a list of any valid SQL expression(s). dbt does not check the syntax of the SQL statements. This an example of a model configuration in a `yml` file you might expect to see on Snowflake: ```yml models: - name: my_incremental_model config: materialized: incremental unique_key: id # this will affect how the data is stored on disk, and indexed to limit scans cluster_by: ['session_start'] incremental_strategy: merge # this limits the scan of the existing table to the last 7 days of data incremental_predicates: ["DBT_INTERNAL_DEST.session_start > dateadd(day, -7, current_date)"] # `incremental_predicates` accepts a list of SQL statements. # `DBT_INTERNAL_DEST` and `DBT_INTERNAL_SOURCE` are the standard aliases for the target table and temporary table, respectively, during an incremental run using the merge strategy. ``` Alternatively, here are the same configurations configured within a model file: ```sql -- in models/my_incremental_model.sql {{ config( materialized = 'incremental', unique_key = 'id', cluster_by = ['session_start'], incremental_strategy = 'merge', incremental_predicates = [ "DBT_INTERNAL_DEST.session_start > dateadd(day, -7, current_date)" ] ) }} ... ``` This will template (in the `dbt.log` file) a `merge` statement like: ```sql merge into <existing_table> DBT_INTERNAL_DEST from <temp_table_with_new_records> DBT_INTERNAL_SOURCE on -- unique key DBT_INTERNAL_DEST.id = DBT_INTERNAL_SOURCE.id and -- custom predicate: limits data scan in the "old" data / existing table DBT_INTERNAL_DEST.session_start > dateadd(day, -7, current_date) when matched then update ... when not matched then insert ... ``` Limit the data scan of *upstream* tables within the body of their incremental model SQL, which will limit the amount of "new" data processed/transformed. ```sql with large_source_table as ( select * from {{ ref('large_source_table') }} {% if is_incremental() %} where session_start >= dateadd(day, -3, current_date) {% endif %} ), ... ``` ##### Defining a unique key Defining the optional [`unique_key` parameter](https://docs.getdbt.com/reference/resource-configs/unique_key.md) enables updating existing rows instead of just appending new rows. If new information arrives for an existing `unique_key`, that new information can replace the current information instead of being appended to the table. If a duplicate row arrives, it can be ignored. Refer to [strategy specific configs](https://docs.getdbt.com/docs/build/incremental-strategy.md#strategy-specific-configs) for more options on managing this update behavior, like choosing only specific columns to update. If you don't specify a `unique_key`, most adapters will result in `append`-only behavior, which means dbt inserts all rows returned by the model's SQL into the preexisting target table without regard for whether the rows represent duplicates. The optional `unique_key` parameter specifies a field (or combination of fields) that defines the grain of your model. That is, the field(s) identify a single unique row. You can define `unique_key` in a configuration block at the top of your model, and it can be a single column name or a list of column names. The `unique_key` should be supplied in your model definition as a string representing a single column or a list of single-quoted column names that can be used together, for example, `['col1', 'col2', …])`. Columns used in this way should not contain any nulls, or the incremental model may fail to match rows and generate duplicate rows. Either ensure that each column has no nulls (for example with `coalesce(COLUMN_NAME, 'VALUE_IF_NULL')`) or define a single-column [surrogate key](https://www.getdbt.com/blog/guide-to-surrogate-key) (for example with [`dbt_utils.generate_surrogate_key`](https://github.com/dbt-labs/dbt-utils#generate_surrogate_key-source)). tip In cases where you need multiple columns in combination to uniquely identify each row, we recommend you pass these columns as a list (`unique_key = ['user_id', 'session_number']`), rather than a string expression (`unique_key = 'concat(user_id, session_number)'`). By using the first syntax, which is more universal, dbt can ensure that the columns will be templated into your incremental model materialization in a way that's appropriate to your database. When you pass a list in this way, please ensure that each column does not contain any nulls, or the incremental model run may fail. Alternatively, you can define a single-column [surrogate key](https://www.getdbt.com/blog/guide-to-surrogate-key), for example with [`dbt_utils.generate_surrogate_key`](https://github.com/dbt-labs/dbt-utils#generate_surrogate_key-source). When you define a `unique_key`, you'll see this behavior for each row of "new" data returned by your dbt model: * If the same `unique_key` is present in the "new" and "old" model data, dbt will update/replace the old row with the new row of data. The exact mechanics of how that update/replace takes place will vary depending on your database, [incremental strategy](https://docs.getdbt.com/docs/build/incremental-strategy.md), and [strategy specific configs](https://docs.getdbt.com/docs/build/incremental-strategy.md#strategy-specific-configs). * If the `unique_key` is *not* present in the "old" data, dbt will insert the entire row into the table. Please note that if there's a unique\_key with more than one row in either the existing target table or the new incremental rows, the incremental model may fail depending on your database and [incremental strategy](https://docs.getdbt.com/docs/build/incremental-strategy.md). If you're having issues running an incremental model, it's a good idea to double check that the unique key is truly unique in both your existing database table and your new incremental rows. You can [learn more about surrogate keys here](https://www.getdbt.com/blog/guide-to-surrogate-key). info While common incremental strategies, such as `delete+insert` + `merge`, might use `unique_key`, others don't. For example, the `insert_overwrite` strategy does not use `unique_key`, because it operates on partitions of data rather than individual rows. For more information, see [About incremental\_strategy](https://docs.getdbt.com/docs/build/incremental-strategy.md). ###### `unique_key` example Consider a model that calculates the number of daily active users (DAUs), based on an event stream. As source data arrives, you will want to recalculate the number of DAUs for both the day that dbt last ran, and any days since then. The model would look as follows: models/staging/fct\_daily\_active\_users.sql ```sql {{ config( materialized='incremental', unique_key='date_day' ) }} select date_trunc('day', event_at) as date_day, count(distinct user_id) as daily_active_users from {{ ref('app_data_events') }} {% if is_incremental() %} -- this filter will only be applied on an incremental run -- (uses >= to include records arriving later on the same day as the last run of this model) where date_day >= (select coalesce(max(date_day), '1900-01-01') from {{ this }}) {% endif %} group by 1 ``` Building this model incrementally without the `unique_key` parameter would result in multiple rows in the target table for a single day – one row for each time dbt runs on that day. Instead, the inclusion of the `unique_key` parameter ensures the existing row is updated instead. #### How do I rebuild an incremental model? If your incremental model logic has changed, the transformations on your new rows of data may diverge from the historical transformations, which are stored in your target table. In this case, you should rebuild your incremental model. To force dbt to rebuild the entire incremental model from scratch, use the `--full-refresh` flag on the command line. This flag will cause dbt to drop the existing target table in the database before rebuilding it for all-time. ```bash $ dbt run --full-refresh --select my_incremental_model+ ``` The trailing `+` in the command above will also run all downstream models that depend on `my_incremental_model`. If any of those downstream dependencies are also incremental models, they will be fully refreshed as well. You can optionally use the [`full_refresh config`](https://docs.getdbt.com/reference/resource-configs/full_refresh.md) to set a resource to always or never full-refresh at the project or resource level. If specified as true or false, the `full_refresh` config will take precedence over the presence or absence of the `--full-refresh` flag. For detailed usage instructions, check out the [dbt run](https://docs.getdbt.com/reference/commands/run.md) documentation. #### What if the columns of my incremental model change? Incremental models can be configured to include an optional `on_schema_change` parameter to enable additional control when incremental model columns change. These options enable dbt to continue running incremental models in the presence of schema changes, resulting in fewer `--full-refresh` scenarios and saving query costs. You can configure the `on_schema_change` setting as follows. dbt\_project.yml ```yaml models: +on_schema_change: "sync_all_columns" ``` models/staging/fct\_daily\_active\_users.sql ```sql {{ config( materialized='incremental', unique_key='date_day', on_schema_change='fail' ) }} ``` The possible values for `on_schema_change` are: * `ignore`: Default behavior (see below). * `fail`: Triggers an error message when the source and target schemas diverge * `append_new_columns`: Append new columns to the existing table. Note that this setting does *not* remove columns from the existing table that are not present in the new data. * `sync_all_columns`: Adds any new columns to the existing table, and removes any columns that are now missing. Note that this is *inclusive* of data type changes. On BigQuery, changing column types requires a full table scan; be mindful of the trade-offs when implementing. **Note**: None of the `on_schema_change` behaviors backfill values in old records for newly added columns. If you need to populate those values, we recommend running manual updates, or triggering a `--full-refresh`. `on_schema_change` tracks top-level changes Currently, `on_schema_change` only tracks top-level column changes. It does not track nested column changes. For example, on BigQuery, adding, removing, or modifying a nested column will not trigger a schema change, even if `on_schema_change` is set appropriately. ##### Default behavior This is the behavior of `on_schema_change: ignore`, which is set by default. If you add a column to your incremental model, and execute a `dbt run`, this column will *not* appear in your target table. If you remove a column from your incremental model and execute a `dbt run`, `dbt run` will fail. Instead, whenever the logic of your incremental changes, execute a full-refresh run of both your incremental model and any downstream models. --- ### Configure your local environment Local development Whether you currently use dbt platform or self-host dbt, follow the instructions on this page to: * [Prepare your local setup](#prepare-your-local-setup) * [Set environment variables locally](#set-environment-variables-locally) * [Configure the dbt extension](#configure-the-dbt-extension) If you're new to dbt or getting started with a new project, you can skip this page and check out our [Quickstart for dbt v2](https://docs.getdbt.com/guides/dbt.md?step=1) to get started with the dbt extension. The steps differ slightly depending on whether you use dbt platform or self host dbt. * dbt platform — You'll mirror your dbt platform environment locally to unlock powerful features like Mesh, deferral, and so on. If your project has environment variables, you'll also set them locally to leverage the VS Code extension's features. * Self-hosted — When you self-host dbt, you'll most likely already have a local setup and environment variables. Use this page to confirm that your existing local setup and environment variables work seamlessly with the VS Code extension. #### Prerequisites * Downloaded and installed the dbt VS Code extension * Basic understanding of [Git workflows](https://docs.getdbt.com/docs/platform/git/version-control-basics.md) and [dbt project structure](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md) * [Developer or analyst license](https://www.getdbt.com/pricing)\* if you're using dbt platform. Note, the [Analyst license type](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md?version=1.12#licenses) is not available for new purchase. #### Prepare your local setup In this section, we'll walk you through the steps to prepare your local setup for the dbt VS Code extension. If you're a dbt platform user that installed the VS Code extension, follow these steps. If you're a self-hosted user, you most likely already have a local setup and environment variables but can confirm using these steps. 1. [Clone](https://code.visualstudio.com/docs/sourcecontrol/overview#_cloning-a-repository) your dbt project repository from your Git provider to your local machine. If you use dbt platform, clone the same repo connected to your project. 2. Ensure you have a dbt [`profiles.yml` file](https://docs.getdbt.com/docs/local/profiles.yml.md). This file defines your data warehouse connection. If you don't have one, run `dbt init` in the terminal to configure your adapter. 3. Validate your `profiles.yml` and project configuration by running `dbt debug`. 4. Add a `dbt_cloud.yml` file from the dbt platform Account settings: * Navigate to **Your profile** -> **VS Code Extension** -> **Download credentials**. * Place the downloaded file in your `~/.dbt/` directory. This registers and connects the extension to dbt platform and enables platform features such as Mesh and deferral. Refer to [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md) for the file's structure and required fields. * Check the `project-id` in your [`dbt_project.yml` `dbt-cloud` block](https://docs.getdbt.com/reference/dbt_cloud.yml.md#the-dbt-cloud-block-in-dbt_projectyml) matches the project you're working on. 5. Confirm connection from your workstation (like running `dbt debug` in the terminal). Your local computer connects directly to your data warehouse and Git. * dbt platform users: Ensure your laptop/VPN is allowed; dbt platform IPs no longer apply. Check with your admin if you have any issues. * dbt v1 users: This has likely already been configured. 6. (Optional) If your project uses environment variables, [find them](https://docs.getdbt.com/docs/build/environment-variables.md#setting-and-overriding-environment-variables) in the dbt platform and [set them](#set-environment-variables-locally) in VS Code or Cursor. * dbt platform users: Copy any environment variables from **Deploy → Environments → Environment variables** tab in dbt platform. Masked secrets are hidden. Work with your admin to get those values. [![Environment variables tab](</img/docs/dbt-platform/using-dbt-platform/Environment Variables/navigate-to-env-vars.png?v=2> "Environment variables tab")](#)Environment variables tab #### Set environment variables locally Environment variables are used for authentication and configuration. This section is most relevant for [dbt VS Code extension](https://docs.getdbt.com/docs/about-dbt-extension.md) and dbt platform users who have environment variables configured as part of their workspace setup. If you're using dbt locally, you can also install the VS Code extension and use its features and actions — you just may not need to configure these variables unless your setup specifically requires them. The following table shows the different options and when to use them: | Location | Affects | Session state | When to use | | --------------------------------------------------------------------------- | --------------------- | ----------------------------- | ------------------------------------------------------------------------ | | [**Shell profile**](#configure-at-the-os-or-shell-level) | Terminal | ✅ Permanent | Variables remain active globally and available across terminal sessions. | | [**VS Code/Cursor settings**](#configure-in-the-vs-code-extension-settings) | Extension menus + LSP | ✅ Per VS Code/Cursor profile | Editor-only workflows using the extension menu actions. | | [**Terminal session**](#configure-in-the-terminal-session) | Current terminal only | ❌ Temporary | One off testing. | tip If you want to use both the VS Code extension menus and terminal to run dbt commands, define your variables in the `shell` profile and VS Code/Cursor settings so they remain active in the terminal globally and in VS Code/Cursor. ##### Configure at the OS or shell level Define variables once at the OS or shell level to ensure they're available to all terminal sessions. Even if you close a terminal window, the variables will remain available to you. ##### Mac / Linux 1. Open your shell configuration file in a text editor using the following commands (If the file does not exist, create it using a text editor using `vi ~/.zshrc` or `vi ~/.bashrc`): ```bash open -e ~/.zshrc ## for zsh (macOS) nano ~/.bashrc ## for bash (Linux or older macOS) ``` 2. A file will open up and you can add your environment variables to the file. For example: * For zsh (macOS): ```bash ## ~/.zshrc export DBT_ENV_VAR1="my_value" export DBT_ENV_VAR2="another_value" ``` * For bash (Linux or older macOS): ```bash ## ~/.bashrc or ~/.bash_profile export DBT_ENV_VAR1="my_value" export DBT_ENV_VAR2="another_value" ``` 3. Save the file. 4. Start a new shell session by closing and reopening the terminal or running `source ~/.zshrc` or `source ~/.bashrc` in the terminal. 5. Verify the variables by running `echo $DBT_ENV_VAR1` and `echo $DBT_ENV_VAR2` in the terminal. <br /> If you see the value printed back in the terminal, you're all set! These variables will now be available: * In all future terminal sessions * For all dbt commands run in the terminal ##### Windows There are two ways to create persistent environment variables on Windows: through PowerShell or the System Properties. The following steps will explain how to configure environment variables using PowerShell. **PowerShell** 1. Run the following commands in PowerShell: ```powershell [Environment]::SetEnvironmentVariable("DBT_ENV_VAR1","my_value","User") [Environment]::SetEnvironmentVariable("DBT_ENV_VAR2","another_value","User") ``` 1. This saves the variables permanently for your user account. To make them available system-wide for all users, replace "User" with "Machine" (requires admin rights). 2. Then, restart VS Code or select **Developer: Reload Window** for changes to take effect. 3. Verify the changes by running `echo $DBT_ENV_VAR1` and `echo $DBT_ENV_VAR2` in the terminal. **System properties (Environment Variables)** 1. Press **Start** → search for **Environment Variables** → open **Edit the system environment variables**. 2. From the **Advanced** tab of the System Properties, click **Environment Variables…**. 3. Under **User variables**, click **New…**. 4. Add the variables and values. For example: * Variable name: `DBT_ENV_VAR1` * Variable value: `my_value` 5. Repeat for any others, then click **OK**. 6. Restart VS Code or Cursor. 7. Verify the changes by running `echo $DBT_ENV_VAR1` and `echo $DBT_ENV_VAR2` in the terminal. ###### About `.env` file support [dbt](https://docs.getdbt.com/docs/local/install-dbt.md) and the dbt VS Code extension can automatically read environment variables from a `.env` file in your current working directory (the folder you `cd` into and run dbt commands from in your terminal), if one exists. The environment variables you define in the `.env` file are available when running dbt commands in the terminal and when using the extension's menu actions. Here are some considerations when defining environment variables in the `.env` file: * The `.env` file provides a convenient way to set environment variables that work across both the CLI and the VS Code extension. * We recommend placing your `.env` file in the project root and running dbt commands from that location because the file is loaded *only* from your current working directory. It doesn't support the `--project-dir` flag or (Applies to dbt v1.11 and later) `DBT_ENGINE_PROJECT_DIR` environment variable, and dbt won't search your project root if you're running commands from a different directory location. * Add `.env` to your `.gitignore` file to prevent sensitive credentials from being committed to your repository. * Environment variables set directly in your shell (such as `export DBT_ENV_VAR=value`) take precedence over values defined in the `.env` file. ##### Configure in the VS Code extension settings To use the dbt extension menu actions/buttons, you can configure environment variables directly in the [VS Code User Settings](vscode://settings/dbt.environmentVariables) interface or in a `.env` file in your current working directory. This includes both your custom variables and any automatic [dbt platform variables](https://docs.getdbt.com/docs/build/environment-variables.md) (like `DBT_CLOUD_ENVIRONMENT_NAME`) that your project depends on. * Configure variables in the VS Code **User Settings** or in a `.env` file to have them recognized by the extension. For example, when using LSP -powered features, "Show build menu," and more. * VS Code does not inherit variables set by the VS Code terminal or external shells. * The terminal uses system environmental variables, and does not inherit variables set in the dbt VS Code extension config. For example, running a dbt command in the terminal won't fetch or use the dbt VS Code extension variables. To configure environment variables in VS Code/Cursor: ##### Open User Settings 1. Open the [Command Palette](https://code.visualstudio.com/docs/configure/settings#_user-settings) (Cmd + Shift + P for Mac, Ctrl + Shift + P for Windows/Linux). 2. Then select either **Preferences: Open User Settings** in the dropdown menu. 3. Open the [VS Code user settings page](vscode://settings/dbt.environmentVariables). 4. Search for `dbt.environmentVariables`. 5. In the **dbt:Environment Variables** section, add your item and value for the environment variables. 6. Click **Ok** to save the changes. 7. Reload the VS Code extension to apply the changes. Open the Command Palette and select **Developer: Reload Window**. 8. Verify the changes by running a dbt command and checking the output. ##### Open .env file 1. Create a `.env` file in your current working directory (typically at the root level of your dbt project, same level as your `dbt_project.yml` file). 2. Add your environment variables to the file. For example: ```env DBT_ENV_VAR1=my_value DBT_ENV_VAR2=another_value ``` 3. Save the file. 4. Reload the VS Code extension to apply the changes. 5. Verify the changes by running a dbt command using the extension menu button on the top right corner and checking the output. For example, running `dbtf debug` will show your connection using the values from `.env`: ```shell dbtf debug ... Debugging connection: "authenticator": "my_authenticator", "account": "my_account", "user": "my_user", "database": "my_database", # Loaded from DBT_MY_DATABASE in .env "schema": "my_schema", # Loaded from DBT_MY_SCHEMA in .env ``` ##### Configure in the terminal session Configure environment variables in the terminal session using the `export` command. Something to keep in mind: * Doing so will make variables visible to commands that run in that terminal session only. * It lasts only for the current session and opening a new terminal will lose the values. * The built-in dbt VS Code extension buttons and menus will not pick these up. To configure environment variables in the terminal session: 1. Run the following command in the terminal, replacing `DBT_ENV_VAR1` and `test1` with your own variable and value. ##### Mac / Linux ```bash export DBT_ENV_VAR1=test1 ``` ##### Windows Cmd Refer to [Microsoft's documentation](https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/set_1) for more information on the `set` command. ```bash set DBT_ENV_VAR1=test1 ``` ##### Windows PowerShell Refer to [Microsoft's documentation](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_environment_variables?view=powershell-7.5#use-the-variable-syntax) for more information on the `$env:` syntax. ```bash $env:DBT_ENV_VAR1 = "test1" ``` 2. Verify the changes by running a dbt command and checking the output. #### dbt extension settings After installing the dbt extension and configuring your local setup, you may want to configure it to better fit your development workflow: 1. Open the VS Code settings by pressing `Ctrl+,` (Windows/Linux) or `Cmd+,` (Mac). 2. Search for `dbt`. On this page, you can adjust the extension's configuration options to fit your needs. [![dbt extension settings within the VS Code settings.](/img/docs/extension/dbt-extension-settings.png?v=2 "dbt extension settings within the VS Code settings.")](#)dbt extension settings within the VS Code settings. The following settings are the most relevant when you install or manage dbt yourself. Most users never need to set these because the extension can download and manage v2 automatically. | Setting | Description | | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `dbt.fusionPath` | Path to the v2 binary. The extension invokes the language server through this binary (`dbt-fusion lsp`). Set this when you install v2 manually — for example, in an [air-gapped environment](https://docs.getdbt.com/docs/dbt-versions/dbt-version-compatibility.md#verify-binaries-for-manual-and-air-gapped-installs) — instead of letting the extension download it. | | `dbt.badReleasesManifestPath` | Path to a local copy of the [known-bad-releases manifest](https://docs.getdbt.com/docs/dbt-versions/dbt-version-compatibility.md#known-bad-releases). Use this if you don't have outbound network access and distribute the manifest alongside your binary bundle (for example, air-gapped installations). | | `dbt.environmentVariables` | Environment variables the extension passes to dbt. Refer to [Configure in the VS Code extension settings](#configure-in-the-vs-code-extension-settings). | Upgrading from a separate language server binary Earlier extension versions used separate `dbt.cliPath` and `dbt.lspPath` settings for two distinct binaries. Current versions use a single `dbt.fusionPath` because the CLI and language server ship as one binary in dbt v2. If you previously configured `dbt.lspPath`, migrate that path to `dbt.fusionPath`. Refer to [Version compatibility](https://docs.getdbt.com/docs/dbt-versions/dbt-version-compatibility.md) for details. #### File associations and other extensions The dbt extension doesn't depend on your `.sql` [file associations](https://code.visualstudio.com/docs/languages/identifiers). It activates on your `dbt_project.yml`, so LSP features — like autocomplete, go-to-definition, and database-aware IntelliSense — work whether your files are set to `sql` or `jinja-sql`. The only difference between those two is syntax highlighting and the file icon. Where the association does matter is for *other* extensions. Any extension that claims the `sql` file type — whether that's Snowflake, SQL Server, or another database tool — can intercept your dbt files and override the IntelliSense you'd otherwise get from the dbt extension. For example, the Snowflake extension only picks up files associated with `snowflake-sql`. That makes `snowflake-sql` the wrong association for your dbt files — it hands them to the Snowflake extension instead of treating them as dbt files. If you have multiple SQL-aware extensions installed, it's worth checking which one is handling your `.sql` files to avoid unexpected behavior. If you want to use both extensions, keep your ad hoc query files in a separate, gitignored directory and scope the Snowflake association to just that path: ```json { "files.associations": { "**/snowflake_sandbox/*.sql": "snowflake-sql", "*.sql": "sql" } } ``` #### Next steps Now that you've configured your local environment, you can start using the dbt extension to streamline your dbt development workflows. Check out the following resources to get started: * [About the dbt extension](https://docs.getdbt.com/docs/about-dbt-extension.md) * [dbt extension features](https://docs.getdbt.com/docs/dbt-extension-features.md) * [Sign in or register](https://docs.getdbt.com/docs/sign-in-dbt-extension.md) --- ### Configuring deferral in dbt State Preview ### Configuring deferral in dbt State [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Login required | Usage-based By default, dbt State defers to your production environment. Both sections on this page are optional; configure them if you want to customize the defaults. #### Configure deferral By default, dbt State defers to your production environment. You only need to configure this if you want to change that behavior: * **dbt platform**: To defer to an environment other than the default (for example, staging), add `defer-env-id` to the `dbt-cloud` block in `dbt_project.yml`. Refer to [Configure the dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) for more information. dbt\_project.yml ```yaml dbt-cloud: project-id: <your-project-id> defer-env-id: <your-environment-id> ``` * **Self-managed deployments**: If you can't access a production or deployment manifest, you can set [`defer_to_target`](https://docs.getdbt.com/reference/resource-configs/defer-to-target.md) in `profiles.yml` for best-effort auto-deferral. Note that this approach has known limitations; refer to [Caveats to dbt State without a manifest](https://docs.getdbt.com/reference/resource-configs/defer-to-target.md#caveats-to-dbt-state-without-a-manifest). profiles.yml ```yaml my_project: outputs: uat: type: snowflake # ... connection settings defer_to_target: staging ``` You can also pass `--state` or `--defer-state` to explicitly point dbt State to a specific `manifest.json`. note If you've overridden `generate_*_name()` macros with runtime values (such as environment variables, file paths, or dates), provide a `manifest.json` file so dbt State can locate objects correctly. Without one, it infers object locations from your macros and profile target, which may be incorrect in these cases. Refer to [Caveats to dbt State without a manifest](https://docs.getdbt.com/reference/resource-configs/defer-to-target.md#caveats-to-dbt-state-without-a-manifest). #### Specify your project or org If you have multiple projects or orgs that use dbt State, configure the `dbt-cloud` block in `dbt_project.yml` so dbt State knows which one to use: * **For dbt platform users with multiple projects**: Add `project-id` to identify which project dbt State should use. dbt\_project.yml ```yaml dbt-cloud: project-id: <your-project-id> ``` * **For self-managed deployments with multiple dbt State orgs**: Add `state-org-id` to identify which org dbt State should use. dbt\_project.yml ```yaml dbt-cloud: state-org-id: <your-org-id> ``` #### Related docs * [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) * [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) * [dbt State configs](https://docs.getdbt.com/reference/resource-configs/dbt-state-configs.md) --- ### Connect dbt MCP server to dbt platform dbt platform This quickstart uses the local MCP server: it runs on your machine using `uvx dbt-mcp`, connects to your dbt platform for Semantic Layer, Discovery, and SQL, and optionally runs local dbt. For self-hosted CLI only (with or without a dbt platform account), see [Run self-hosted dbt](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-cli.md) or [Run self-hosted dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md). To configure or disable specific tools, see the [Environment variables reference](https://docs.getdbt.com/docs/dbt-ai/mcp-environment-variables.md). #### Prerequisites * [Install uv](https://docs.astral.sh/uv/getting-started/installation/) * A [dbt platform account](https://www.getdbt.com/signup) * For OAuth connections: * MCP OAuth is available for Starter, Enterprise, and Enterprise+ plans. #### Step 1: Choose your auth method and configure ##### OAuth *MCP OAuth is available in public beta for Starter, Enterprise, and Enterprise+ accounts.* OAuth is the fastest setup for dbt platform accounts, no tokens to copy or manage. A browser window opens to authenticate the first time you connect. For OAuth *without* a self-hosted installation, use the [remote MCP server](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-remote.md). Remote MCP OAuth is available in public beta for Starter, Enterprise, and Enterprise+ accounts. If your client does not support OAuth or you need token-based access, use [token-based authentication](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#token-based-authentication). Static subdomains required Only accounts with static subdomains (for example, `abc123` in `abc123.us1.dbt.com`) can use OAuth with MCP servers. Follow [these](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) instructions to find your account subdomain. If your account does not have a subdomain, contact support for more information. ###### Find your Access URL 1. Log in to your dbt platform account. 2. Go to **Account settings** and copy your **Access URL** (for example, `abc123.us1.dbt.com`). Multi-cell and DBT\_HOST format * The `DBT_HOST` field accepts both `abc123.us1.dbt.com` and `https://abc123.us1.dbt.com`. * If your Access URL is `abc123.us1.dbt.com`, split it into two variables: * `DBT_HOST=us1.dbt.com` * `MULTICELL_ACCOUNT_PREFIX=abc123` Don't include the account prefix in `DBT_HOST`. For more details, see [multi-cell configuration examples](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#api-and-sql-tool-settings). ###### Add the config to your MCP client ##### Claude Desktop **Option A: Quick install (recommended)** 1. Go to the [latest dbt MCP release](https://github.com/dbt-labs/dbt-mcp/releases/latest) and download `dbt-mcp.mcpb`. 2. Double-click the file to open it in Claude Desktop. 3. Enter your **Access URL** as the dbt platform Host. 4. Enable the server. **Option B: Manual config** 1. In Claude Desktop, go to **Settings** → **Developer** tab → **Edit Config**. 2. Paste the following configuration, replacing `YOUR-ACCESS-URL` with your Access URL: ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "YOUR-ACCESS-URL" } } } } ``` 3. Save and restart Claude Desktop. Config file location: * macOS: `~/Library/Application Support/Claude/claude_desktop_config.json` * Windows: `%APPDATA%\Claude\claude_desktop_config.json` ##### Claude Code Run this command, replacing `YOUR-ACCESS-URL` with your Access URL: ```shell claude mcp add dbt \ -e DBT_HOST=YOUR-ACCESS-URL \ -- uvx dbt-mcp ``` For example, if your Access URL is `abc123.us1.dbt.com`: ```shell claude mcp add dbt \ -e DBT_HOST=abc123.us1.dbt.com \ -- uvx dbt-mcp ``` ##### Cursor Click a link below with Cursor open to auto-configure, then replace the placeholder with your Access URL: * [dbt platform only (OAuth)](cursor://anysphere.cursor-deeplink/mcp/install?name=dbt\&config=eyJlbnYiOnsiREJUX0hPU1QiOiJZT1VSLUFDQ0VTUy1VUkwiLCJESVNBQkxFX0RCVF9DTEkiOiJ0cnVlIn0sImNvbW1hbmQiOiJ1dngiLCJhcmdzIjpbImRidC1tY3AiXX0%3D) — platform features only, no CLI * [dbt platform + CLI (OAuth)](cursor://anysphere.cursor-deeplink/mcp/install?name=dbt\&config=eyJlbnYiOnsiREJUX0hPU1QiOiJZT1VSLUFDQ0VTUy1VUkwiLCJEQlRfUFJPSkVDVF9ESVIiOiIvcGF0aC90by9wcm9qZWN0IiwiREJUX1BBVEgiOiJwYXRoL3RvL2RidC9leGVjdXRhYmxlIn0sImNvbW1hbmQiOiJ1dngiLCJhcmdzIjpbImRidC1tY3AiXX0%3D) — platform features + self-hosted dbt CLI commands After clicking, replace `YOUR-ACCESS-URL` with your actual Access URL (for example, `abc123.us1.dbt.com`) and save. ##### VS Code 1. Open **Settings** → **Features** → **Chat** and ensure **MCP** is enabled. 2. Open the Command Palette (`Ctrl/Cmd + Shift + P`) and select **MCP: Open User Configuration**. 3. Add the following configuration to `mcp.json`: VS Code uses `"servers"`, not `"mcpServers"` ```json { "servers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "YOUR-ACCESS-URL" } } } } ``` Replace `YOUR-ACCESS-URL` with your Access URL (for example, `abc123.us1.dbt.com`) and save. ###### Optional: Add self-hosted dbt CLI commands To also run dbt platform CLI commands (`dbt run`, `dbt build`, `dbt test`, and more), add these two variables to your `env` block: ```json "DBT_PROJECT_DIR": "/path/to/your/dbt/project", "DBT_PATH": "/path/to/your/dbt/executable" ``` Find `DBT_PATH` by running `which dbt` (macOS/Linux) or `where dbt` (Windows). `DBT_PROJECT_DIR` is the folder containing your `dbt_project.yml`. ##### Tokens Token-based auth gives you more control and is better for shared or team setups. You'll need a service token or Personal Access Token (PAT). Which token should I use? * **PAT (Personal Access Token):** Required if you want to use `execute_sql`. Tied to your user account. * **Service token:** Works for all other platform toolsets. Better for shared or team setups. See [Choosing an auth method](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#choose-your-auth-method) for full guidance. ##### Find your paths and IDs You need the following values. See [Finding your IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md) for step-by-step instructions. | Variable | Where to find it | | ----------------- | -------------------------------------------------------------------------- | | `DBT_HOST` | Your dbt platform hostname, found in **Account settings** → **Access URL** | | `DBT_TOKEN` | A service token or PAT from **Account settings** → **API tokens** | | `DBT_PROD_ENV_ID` | Your production environment ID, found in **Deploy** → **Environments** | | `DBT_DEV_ENV_ID` | Your development environment ID (required for `execute_sql`) | | `DBT_USER_ID` | Your numeric user ID (required for `execute_sql`) | | `DBT_ACCOUNT_ID` | Your account ID (required for Admin API tools) | Use values only, not full URLs These variables expect hostnames or numeric IDs — not full URLs: ```bash # ✅ Correct DBT_HOST=cloud.getdbt.com # https://cloud.getdbt.com also works DBT_PROD_ENV_ID=54321 DBT_USER_ID=123 # ❌ Wrong — IDs must be numeric, not full URLs DBT_PROD_ENV_ID=https://cloud.getdbt.com/deploy/12345/projects/67890/environments/54321 DBT_USER_ID=https://cloud.getdbt.com/settings/profile ``` Multi-cell accounts If your Access URL is `abc123.us1.dbt.com`, split it into two variables: * `DBT_HOST=us1.dbt.com` * `MULTICELL_ACCOUNT_PREFIX=abc123` Don't include the account prefix in `DBT_HOST`. For more details, see [multi-cell configuration examples](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#api-and-sql-tool-settings). ##### Add the config to your MCP client Use the configuration below, replacing the placeholder values with your IDs from above. Include only the variables you need: ##### Claude Desktop 1. In Claude Desktop, go to **Settings** → **Developer** tab → **Edit Config**. 2. Paste the following configuration, replacing the placeholder values: ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "cloud.getdbt.com", "DBT_TOKEN": "your-token-here", "DBT_PROD_ENV_ID": "12345", "DBT_DEV_ENV_ID": "67890", "DBT_USER_ID": "123", "DBT_ACCOUNT_ID": "99999" } } } } ``` 3. Save and restart Claude Desktop. Config file location: * macOS: `~/Library/Application Support/Claude/claude_desktop_config.json` * Windows: `%APPDATA%\Claude\claude_desktop_config.json` ##### Claude Code Run this command, replacing the placeholders with your actual values: ```bash claude mcp add dbt \ -e DBT_HOST=cloud.getdbt.com \ -e DBT_TOKEN=your-token-here \ -e DBT_PROD_ENV_ID=12345 \ -- uvx dbt-mcp ``` Add `-e DBT_DEV_ENV_ID=...` and `-e DBT_USER_ID=...` if you use `execute_sql`; add `-e DBT_ACCOUNT_ID=...` for Admin API. ##### Cursor 1. In Cursor, open **Settings** → **MCP** → **Edit config** (or your config file). 2. Paste the following configuration, replacing the placeholder values: ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "cloud.getdbt.com", "DBT_TOKEN": "your-token-here", "DBT_PROD_ENV_ID": "12345", "DBT_DEV_ENV_ID": "67890", "DBT_USER_ID": "123", "DBT_ACCOUNT_ID": "99999" } } } } ``` 3. Save the configuration. ##### VS Code 1. Open **Settings** → **Features** → **Chat** and ensure **MCP** is enabled. 2. Open the Command Palette (`Ctrl/Cmd + Shift + P`) and select **MCP: Open User Configuration**. 3. Add the following configuration to `mcp.json`: VS Code uses `"servers"`, not `"mcpServers"` ```json { "servers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "cloud.getdbt.com", "DBT_TOKEN": "your-token-here", "DBT_PROD_ENV_ID": "12345", "DBT_DEV_ENV_ID": "67890", "DBT_USER_ID": "123", "DBT_ACCOUNT_ID": "99999" } } } } ``` 4. Save `mcp.json` and restart VS Code.  Optional: add self-hosted dbt CLI commands To also run dbt commands (`dbt run`, `dbt build`, `dbt test`, and more), add these two variables to your `env` block: ```json "DBT_PROJECT_DIR": "/path/to/your/dbt/project", "DBT_PATH": "/path/to/your/dbt/executable" ``` Find `DBT_PATH` by running `which dbt` (macOS/Linux) or `where dbt` (Windows). `DBT_PROJECT_DIR` is the folder containing your `dbt_project.yml`. #### Step 2: Authenticate ##### OAuth The first time you connect, dbt MCP opens a browser window to complete OAuth. After signing in, your session is saved and future connections are automatic. If authentication doesn't start, close your client and run: * macOS/Linux: `rm -f ~/.dbt/mcp.yml ~/.dbt/mcp.lock` * Windows: `Remove-Item -Force $env:USERPROFILE\.dbt\mcp.yml, $env:USERPROFILE\.dbt\mcp.lock` Then restart your client. ##### Tokens No additional authentication step is needed — your token is already in the configuration. The server connects automatically when your MCP client starts. #### Step 3: Test your setup Ask your AI assistant a data-related question (for example, *"What models are in my dbt project?"* or *"What metrics are defined in my Semantic Layer?"*). If dbt MCP is working, the response will use your dbt metadata. #### What's available With the platform setup, your AI assistant can use: * Semantic Layer queries * Metadata Discovery (model lineage, test results, source freshness) * Admin API (trigger jobs, list runs, get artifacts) * SQL execution and text-to-SQL (requires a [PAT](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md)) * All dbt commands if you added `DBT_PROJECT_DIR` and `DBT_PATH` For the complete tool list, see [Available tools](https://docs.getdbt.com/docs/dbt-ai/mcp-available-tools.md). Looking for self-hosted dbt CLI only? If you only need to run dbt commands locally (with or without a dbt platform account), see [Run self-hosted dbt](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-cli.md). #### Troubleshooting  Can't find the uvx executable  Can't find the uvx executable **Symptoms:** Error messages like `Could not connect to MCP server dbt-mcp`, `Error: spawn uvx ENOENT`, or `spawn uvx ENOENT` in your MCP client. **Cause:** Your MCP client (like Claude desktop) can't find `uvx` in its PATH because it starts with a limited environment. **Solution:** Use the full path to `uvx` in your configuration. 1. Find the full path: * macOS/Linux: Run `which uvx` in Terminal. * Windows: Run `where uvx` in Command Prompt or PowerShell. 2. Replace `"command": "uvx"` with the full path: ```json { "mcpServers": { "dbt": { "command": "/full/path/to/uvx", "args": ["dbt-mcp"], "env": { } } } } ``` Example on macOS with Homebrew: `"command": "/opt/homebrew/bin/uvx"` For VS Code (`mcp.json`), the same fix applies — replace `uvx` with its full path in the `command` field.  OAuth login not initiating  OAuth login not initiating **Symptoms:** The OAuth browser window never opens, or authentication appears to hang. **Cause:** dbt MCP uses a lock file to avoid repeated authentication. If a previous session left the lock file in place, it can block new authentication attempts. **Solution:** 1. Close your MCP client (Claude Desktop, Cursor, VS Code, etc.). 2. Delete the self-hosted dbt MCP config files: * macOS/Linux: `rm -f ~/.dbt/mcp.yml ~/.dbt/mcp.lock` * Windows: `Remove-Item -Force $env:USERPROFILE\.dbt\mcp.yml, $env:USERPROFILE\.dbt\mcp.lock` 3. Restart your client and try connecting again. If these steps don't resolve the issue, confirm that AI features are enabled on your account. An account admin can enable them in **Account settings** → **Edit** → toggle on **Enable account access to dbt Wizard features**. Refer to [Enable dbt Wizard](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md).  Server not starting  Server not starting **Symptoms:** The MCP server shows as disconnected or unavailable in your client. **Diagnosis:** Check the server logs: * **VS Code:** Open the Command Palette (`Ctrl/Cmd + Shift + P`) → `MCP: List Servers` → click the dbt server to see detailed logs. * **Claude Desktop:** Check `~/Library/Logs/Claude` (macOS) or `%APPDATA%\Claude\logs` (Windows). * **All clients:** Set `DBT_MCP_LOG_LEVEL=DEBUG` in your environment variables to get more verbose output. **Common causes:** * Missing or incorrect `DBT_PROJECT_DIR` or `DBT_PATH` — verify the paths exist and are absolute paths. * Invalid or expired authentication tokens — generate a new token and update your config. * Missing required environment variables for the toolset you're trying to use — see [Tool requirements](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#tool-requirements-at-a-glance).  execute\_sql tool not working  execute\_sql tool not working **Symptoms:** The `execute_sql` tool returns an authentication error or is unavailable. **Cause:** How you fix this depends on how you connected: * **Remote MCP with OAuth:** Sign in through your MCP client when prompted. You don't need a personal access token. * **Remote MCP with token-based auth:** You need a [personal access token (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md). Service tokens won't work. You also need `x-dbt-dev-environment-id` and `x-dbt-user-id` headers. The user ID must match the user who created the PAT. * **Local MCP:** Set `DBT_TOKEN` to a personal access token (PAT). Service tokens won't work. **Solution:** 1. If you use OAuth for remote MCP, confirm you completed sign-in and consent in your MCP client. 2. If you use token-based auth or local MCP, create a [personal access token (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) in **Account settings** → **API tokens** → **Personal tokens**. 3. For token-based remote MCP, add the PAT to the `Authorization` header as `Token YOUR_PAT`. For local MCP, set the PAT as your `DBT_TOKEN` value. 4. Also ensure `DBT_DEV_ENV_ID` and `DBT_USER_ID` are set (local MCP) or `x-dbt-dev-environment-id` and `x-dbt-user-id` headers are set (remote MCP). Refer to [Finding your IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md) for details.  Toolset unavailable or showing as disabled  Toolset unavailable or showing as disabled **Symptoms:** A toolset (Semantic Layer, Discovery, Admin API) is not available in your AI client even though you've configured credentials. **Cause:** Either the required variables are missing, or the toolset has been explicitly disabled. **Solution:** 1. Check that all required variables for the toolset are set — see [Tool requirements](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#tool-requirements-at-a-glance). 2. Check whether you have any `DISABLE_*` variables set to `true` that might be turning off the toolset. 3. If you're using enable mode (`DBT_MCP_ENABLE_*`), make sure the toolset you need is listed. 4. Set `DBT_MCP_LOG_LEVEL=DEBUG` to see which toolsets are active at startup.  Pasting full URLs instead of IDs  Pasting full URLs instead of IDs **Symptoms:** Authentication errors, unexpected behavior, or the server failing to connect to the right environment. **Cause:** Environment variables like `DBT_PROD_ENV_ID`, `DBT_USER_ID`, and `DBT_ACCOUNT_ID` expect numeric integers, not full browser URLs. **Solution:** ```bash # ✅ Correct DBT_HOST=cloud.getdbt.com # https://cloud.getdbt.com also works DBT_PROD_ENV_ID=54321 DBT_USER_ID=123 # ❌ Wrong — IDs must be numeric, not full URLs DBT_PROD_ENV_ID=https://cloud.getdbt.com/deploy/12345/projects/67890/environments/54321 DBT_USER_ID=https://cloud.getdbt.com/settings/profile ``` See [Finding your IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md) for step-by-step instructions.  Multi-cell account connection issues  Multi-cell or static subdomain account connection issues **Symptoms:** Connection errors when your account URL includes a prefix (for example, `abc123.us1.dbt.com`). **Solution (as of v1.14.0):** Set `DBT_HOST` to the full hostname including the prefix. If you're using PAT-based auth, also set `DBT_ACCOUNT_ID`. ```bash # ✅ Correct DBT_HOST=abc123.us1.dbt.com DBT_ACCOUNT_ID=12345 # required for PAT-based auth ``` You no longer need to set `MULTICELL_ACCOUNT_PREFIX` or `DBT_HOST_PREFIX`. If you have these set from an older configuration, remove them. For all troubleshooting topics, see [MCP troubleshooting](https://docs.getdbt.com/docs/dbt-ai/mcp-troubleshooting.md). #### Next steps * Run dbt commands locally: see [Run self-hosted dbt](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-cli.md) * Configure specific toolsets: see the [Environment variables reference](https://docs.getdbt.com/docs/dbt-ai/mcp-environment-variables.md) * Understand toolset requirements: see [Set up self-hosted MCP](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#tool-requirements-at-a-glance) --- ### Connect to adapters Adapters are an essential component of dbt. At their most basic level, they are how dbt connects with the various supported data platforms. At a higher-level, adapters strive to give analytics engineers more transferrable skills as well as standardize how analytics projects are structured. Gone are the days where you have to learn a new language or flavor of SQL when you move to a new job that has a different data platform. That is the power of adapters in dbt — for more detail, refer to the [Build, test, document, and promote adapters](https://docs.getdbt.com/guides/adapter-creation.md) guide. This section provides more details on different ways you can connect dbt to an adapter, and explains what a maintainer is. ##### Set up in dbt Explore the fastest and most reliable way to deploy dbt by using the dbt platform, a hosted architecture that runs dbt across your organization. The dbt platform lets you seamlessly [connect](https://docs.getdbt.com/docs/platform/about-platform-setup.md) with a variety of [trusted](https://docs.getdbt.com/docs/supported-data-platforms.md) data platform providers directly in the dbt UI. ##### Install self-hosted dbt Install self-hosted dbt locally using the command line. dbt communicates with a number of different data platforms by using a dedicated adapter plugin for each. When you install dbt locally, you'll also need to configure the specific adapter for your database, [connect dbt v2 to dbt v1](https://docs.getdbt.com/docs/local/install-dbt.md), and set up a `profiles.yml` file. (Applies to dbt v2.0 and later) Trusted adapters ship with dbt v2 — when you [install dbt](https://docs.getdbt.com/docs/local/install-dbt.md), the supported data platforms are available out of the box, with no separate `pip install` per adapter needed. Refer to [adapter creation](https://docs.getdbt.com/guides/adapter-creation-v2.md?step=1) for more info. #### Footnotes 1. Use the PyPI package name when installing with `pip` | Adapter repo name | PyPI package name | | ----------------- | -------------------- | | `dbt-layer` | `dbt-layer-bigquery` | [↩](#user-content-fnref-1) --- ### Connect to the remote dbt MCP server dbt platform The remote MCP server connects to dbt platform using HTTP. No self-hosted installation is required — you configure your MCP client with a URL and headers instead of running `uvx dbt-mcp`. [![Remote dbt MCP server architecture](/img/mcp/remote-dbt-mcp.jpg?v=2 "Remote dbt MCP server architecture")](#)Remote dbt MCP server architecture #### When to use remote MCP Remote MCP is a good fit when: * You don't want to or can't install software (`uvx`, dbt-mcp) on your machine. * Your use case is *consumption-based*: querying metrics, exploring metadata, viewing lineage, or running SQL on the platform. * You need Semantic Layer, Administrative, and Discovery APIs access without a local dbt project. Self-hosted development requires self-hosted MCP Self-hosted development and agentic workflows (for example, running dbt commands like `dbt run` or `dbt build` from your AI assistant) require the **self-hosted** MCP server. Remote MCP does not support the self-hosted dbt v1 or dbt v2 CLI or local project access. Use [Connect to dbt platform](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-oauth.md) or [Run self-hosted dbt](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-cli.md) for those workflows. #### Set up remote MCP Follow these steps to set up the remote MCP server. ##### 1. Manage AI features AI features are enabled by default. Admins can [turn them off or back on anytime](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md). ##### 2. Get your credentials Obtain the following information from dbt platform: * **dbt platform host**: Form the URL as `https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/` (for example, `https://cloud.getdbt.com/api/ai/v1/mcp/`). For multi-cell accounts, the host is in the format `ACCOUNT_PREFIX.us1.dbt.com`. Refer to [Access, Regions, & IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). * **Production environment ID**: From **Orchestration** → **Environments** in dbt platform. You will use it as the `x-dbt-prod-environment-id` header (token-based setup only). Refer to [How to find your dbt MCP IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md#dbt-prod-env-id) for step-by-step instructions. * **Token** — PAT or service token with Semantic Layer and Developer permissions (token-based setup only). * **If you use `execute_sql` with token-based auth:** You must use a PAT, plus your development environment ID and user ID. Refer to [How to find your dbt MCP IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md) for details. With OAuth, you only need your MCP URL. ##### 3. Choose authentication: OAuth or tokens | Type | Info | | ---------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **[OAuth (remote)](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#oauth-remote-mcp)** | No API tokens in your client config. Requires an OAuth-capable MCP client. Supports `execute_sql`.<br /><br />Available in public beta for Starter, Enterprise, and Enterprise+ accounts. | | **Token-based** | PAT or service token in the `Authorization` header. Works with any client and is required for shared/CI setups. For `execute_sql`, you must use a PAT (service tokens do not work). | info Remote MCP OAuth is available in public beta for Starter, Enterprise, and Enterprise+ accounts. ##### 4. Get your MCP URL and IDs You can copy your full **MCP URL** from **Account settings** → **Access URLs** → **MCP Endpoint URL** in dbt platform, and paste it directly into your AI tool.  Build your own MCP URL We recommend using the MCP URL from **Account settings** → **Access URLs** → **MCP Endpoint URL** in dbt platform. However, if you want to build your own MCP URL, use your **Access URL** from **Account settings** in dbt platform. The remote MCP endpoint is `https://YOUR_DBT_HOST_URL/api/ai/v1/mcp`. Replace `YOUR_DBT_HOST_URL` with your hostname only (no `https://`). For default hosts, multi-cell accounts, and regions, see [Access, Regions, & IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). Depending on your auth method, you may also need: * **Production environment ID**: From **Orchestration** → **Environments** in dbt platform. Used as the `x-dbt-prod-environment-id` header for token-based setup. Refer to [How to find your dbt MCP IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md#dbt-prod-env-id) for step-by-step instructions. * **Token** — PAT or service token with Semantic Layer and Developer permissions (token-based setup only). * **If you use `execute_sql` with token-based auth:** You must use a PAT, plus your development environment ID and user ID. Refer to [How to find your dbt MCP IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md) for details. With OAuth, you only need your MCP URL. info Only [`text_to_sql`](https://docs.getdbt.com/docs/dbt-ai/mcp-available-tools.md) consumes your dbt Copilot action allotment. Other MCP tools do not. When your account runs out of dbt Copilot actions, the remote MCP server blocks every tool that runs through it, including tools invoked from a self-hosted MCP server and [proxied](https://github.com/dbt-labs/dbt-mcp/blob/main/src/dbt_mcp/tools/toolsets.py#L24) to remote MCP, such as SQL and remote dbt v2 tools. If you reach your dbt Copilot actions limit, remote MCP tools remain unavailable until the limit resets. If you need help, contact your account manager. ##### 5. Configure your MCP client Configure your MCP client with the MCP URL from the previous step. If you use token-based authentication, also add the required headers. ##### OAuth *Available for Starter, Enterprise, and Enterprise+ accounts* **Before you connect** * Your MCP client must support OAuth for HTTP-based MCP servers. If it doesn't, use [token-based authentication](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#token-based-authentication) instead. * On first connect, your client opens a browser for sign-in. dbt then shows a consent screen with the scopes (the specific permissions the client is allowed to use) it's requesting — see [Scopes and consent](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#scopes-and-consent) for what each scope means. * Most modern MCP clients self-register on first connect via [dynamic registration (RFC 7591)](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#dynamic-registration). Clients that don't support it need an admin to register them in **Account settings → Integrations → App integrations**. See [Manual registration](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#manual-registration). For the full flow, sessions, and limitations, refer to [OAuth (remote MCP)](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#oauth-remote-mcp). Configure your client with the MCP URL from the previous step. On first connect, your client opens a browser for sign-in and consent. You don't need a personal access token or extra headers to use `execute_sql`. Some tools (like Claude Desktop) let you add dbt as a custom connector through their UI instead of editing a config file: The following steps show how to connect dbt as a custom connector in Claude Desktop. The exact UI varies by tool, but the flow is the same: add a custom connector with your MCP URL, complete the OAuth consent flow, then connect. 1. In your AI tool, go to its connector settings and choose to add a custom connector (in Claude Desktop, go to **Chat → Customize → Connectors**, then click **Add custom connector**). 2. Enter a name (for example, `dbt`) and paste your dbt platform MCP URL (for example, `https://abc123.us1.dbt.com/api/ai/v1/mcp`), then click **Add**. [![Custom connector dialog showing the dbt MCP URL](/img/docs/dbt-cloud/oauth-add-custom-connector.png?v=2 "Custom connector dialog showing the dbt MCP URL")](#)Custom connector dialog showing the dbt MCP URL 3. Click **Connect**. The tool redirects you to dbt to complete the OAuth consent flow, where you can approve or deny individual [scopes](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#scopes-and-consent). [![OAuth consent screen showing requested scopes and project access](/img/docs/dbt-cloud/oauth-consent-screen.png?v=2 "OAuth consent screen showing requested scopes and project access")](#)OAuth consent screen showing requested scopes and project access 4. After you approve, the connector is added to the **Custom connectors** table and shows as connected. [![Adding a custom dbt connector in an AI tool's connector settings](/img/docs/dbt-cloud/oauth-connectors-page.png?v=2 "Adding a custom dbt connector in an AI tool's connector settings")](#)Adding a custom dbt connector in an AI tool's connector settings 5. That's it 🎉! Ask your tool a data question like *"What is the total revenue for the last 30 days?"* to confirm the connection. For tools configured with a JSON file, use the tab that matches your client: ##### Claude Code Add this to `.mcp.json` at your project root: ```json { "mcpServers": { "dbt": { "type": "http", "url": "https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/" } } } ``` ##### Cursor Add this to `.cursor/mcp.json` (or use the Cursor deeplink in [Integrate Cursor with MCP](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-cursor.md#set-up-with-remote-dbt-mcp-server)): ```json { "mcpServers": { "dbt": { "url": "https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/" } } } ``` ##### VS Code Add this to `mcp.json` (run **MCP: Open Workspace Folder MCP Configuration** from the command palette). VS Code uses the `servers` key, not `mcpServers`: ```json { "servers": { "dbt": { "type": "http", "url": "https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/" } } } ``` ##### Token-based Set the server `url` to `https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/` and add the required headers: * **Required:** `Authorization` (value `Token YOUR_TOKEN` or `Bearer YOUR_TOKEN`), `x-dbt-prod-environment-id` * **For `execute_sql` with token-based auth or dbt v2 tools:** Also add `x-dbt-dev-environment-id` and `x-dbt-user-id` * Use numeric IDs in headers, not full URLs copied from your browser. ##### Claude Code ```json { "mcpServers": { "dbt": { "type": "http", "url": "https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/", "headers": { "Authorization": "Token YOUR_DBT_ACCESS_TOKEN", "x-dbt-prod-environment-id": "DBT_PROD_ENV_ID", "x-dbt-user-id": "DBT_USER_ID", "x-dbt-dev-environment-id": "DBT_DEV_ENV_ID" } } } } ``` ##### Cursor ```json { "mcpServers": { "dbt": { "url": "https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/", "headers": { "Authorization": "Token YOUR_DBT_ACCESS_TOKEN", "x-dbt-prod-environment-id": "DBT_PROD_ENV_ID", "x-dbt-user-id": "DBT_USER_ID", "x-dbt-dev-environment-id": "DBT_DEV_ENV_ID" } } } } ``` ##### VS Code VS Code uses the `servers` key, not `mcpServers`: ```json { "servers": { "dbt": { "type": "http", "url": "https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/", "headers": { "Authorization": "Token YOUR_DBT_ACCESS_TOKEN", "x-dbt-prod-environment-id": "DBT_PROD_ENV_ID", "x-dbt-user-id": "DBT_USER_ID", "x-dbt-dev-environment-id": "DBT_DEV_ENV_ID" } } } } ``` ##### Gemini Gemini uses the `httpUrl` key instead of `url`: ```json { "mcpServers": { "dbt": { "httpUrl": "https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/", "headers": { "Authorization": "Token YOUR_DBT_ACCESS_TOKEN", "x-dbt-prod-environment-id": "DBT_PROD_ENV_ID", "x-dbt-user-id": "DBT_USER_ID", "x-dbt-dev-environment-id": "DBT_DEV_ENV_ID" } } } } ``` * For the complete list of headers, Cursor and other client examples, and optional headers, refer to [Set up remote MCP](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md). * For self-hosted MCP, configuration uses environment variables; check out the [Environment variables reference](https://docs.getdbt.com/docs/dbt-ai/mcp-environment-variables.md) for more information. Once you have configured your MCP client, you can test your setup by asking your AI assistant a data-related question (for example, *"What models are in my dbt project?"* or *"What metrics are defined in my Semantic Layer?"*). If dbt MCP is working, the response will use your dbt metadata. --- ### Connection profiles Local development When you invoke dbt from the command line, dbt parses your `dbt_project.yml` and obtains the `profile` name, which dbt needs to connect to your data warehouse. dbt\_project.yml ```yaml # Example dbt_project.yml file name: 'jaffle_shop' profile: 'jaffle_shop' ... ``` dbt then checks your `profiles.yml` file for a profile with the same name. A profile contains all the details required to connect to your data warehouse. dbt will search the current working directory for the `profiles.yml` file and will default to the `~/.dbt/` directory if not found. This file generally lives outside of your dbt project to avoid sensitive credentials being checked in to version control, but `profiles.yml` can be safely checked in when [using environment variables](#advanced-using-environment-variables) to load sensitive credentials. \~/.dbt/profiles.yml ```yaml # example profiles.yml file jaffle_shop: target: dev outputs: dev: type: postgres host: localhost user: alice password: <password> port: 5432 dbname: jaffle_shop schema: dbt_alice threads: 4 prod: # additional prod target type: postgres host: prod.db.example.com user: alice password: <prod_password> port: 5432 dbname: jaffle_shop schema: analytics threads: 8 ``` To add an additional target (like `prod`) to your existing `profiles.yml`, you can add another entry under the `outputs` key. #### The `env_var` function The `env_var` function can be used to incorporate environment variables from the system into your dbt project. You can use the `env_var` function in your `profiles.yml` file, the `dbt_project.yml` file, the `sources.yml` file, your `schema.yml` files, and in model `.sql` files. Essentially, `env_var` is available anywhere dbt processes Jinja code. When used in a `profiles.yml` file (to avoid putting credentials on a server), it can be used like this: profiles.yml ```yaml profile: target: prod outputs: prod: type: postgres host: 127.0.0.1 # IMPORTANT: Make sure to quote the entire Jinja string here user: "{{ env_var('DBT_USER') }}" password: "{{ env_var('DBT_PASSWORD') }}" .... ``` #### About the `profiles.yml` file In your `profiles.yml` file, you can store as many profiles as you need. Typically, you would have one profile for each warehouse you use. Most organizations only have one profile. #### About profiles A profile consists of *targets*, and a specified *default target*. Each *target* specifies the type of warehouse you are connecting to, the credentials to connect to the warehouse, and some dbt-specific configurations. The credentials you need to provide in your target varies across warehouses — sample profiles for each supported warehouse are available in the [Supported Data Platforms](https://docs.getdbt.com/docs/supported-data-platforms.md) section. **Pro Tip:** You may need to surround your password in quotes if it contains special characters. More details [here](https://stackoverflow.com/a/37015689/10415173). #### Setting up your profile To set up your profile, copy the correct sample profile for your warehouse into your `profiles.yml` file and update the details as follows: * Profile name: Replace the name of the profile with a sensible name – it’s often a good idea to use the name of your organization. Make sure that this is the same name as the `profile` indicated in your `dbt_project.yml` file. * `target`: This is the default target your dbt project will use. It must be one of the targets you define in your profile. Commonly it is set to `dev`. * Populating your target: * `type`: The type of data warehouse you are connecting to * Warehouse credentials: Get these from your database administrator if you don’t already have them. Remember that user credentials are very sensitive information that should not be shared. * `schema`: The default schema that dbt will build objects in. * `threads`: The number of threads the dbt project will run on. You can find more information on which values to use in your targets below. Use the [debug](https://docs.getdbt.com/reference/dbt-jinja-functions/debug-method.md) command to validate your warehouse connection. Run `dbt debug` from within a dbt project to test your connection. #### Understanding targets in profiles dbt supports multiple targets within one profile to encourage the use of separate development and production environments as discussed in [dbt environments](https://docs.getdbt.com/docs/local/dbt-environments.md). A typical profile for an analyst using dbt locally will have a target named `dev`, and have this set as the default. You may also have a `prod` target within your profile, which creates the objects in your production schema. However, since it's often desirable to perform production runs on a schedule, we recommend deploying your dbt project to a separate machine other than your local machine. Most dbt users only have a `dev` target in their profile on their local machine. If you do have multiple targets in your profile, and want to use a target other than the default, you can do this using the `--target` flag when running a dbt command. For example, to run against your `prod` target instead of the default `dev` target: ```bash dbt run --target prod ``` You can use the `--target` flag with any dbt command, such as: ```bash dbt build --target prod dbt test --target dev dbt compile --target qa ``` ##### Overriding profiles and targets When running dbt commands, you can specify which profile and target to use from the CLI using the `--profile` and `--target` [flags](https://docs.getdbt.com/reference/global-configs/about-global-configs.md#available-flags). These flags override what’s defined in your `dbt_project.yml` as long as the specified profile and target are already defined in your `profiles.yml` file. To run your dbt project with a different profile or target than the default, you can do so using the followingCLI flags: * `--profile` flag — Overrides the profile set in `dbt_project.yml` by pointing to another profile defined in `profiles.yml`. * `--target` flag — Specifies the target within that profile to use (as defined in `profiles.yml`). These flags help when you're working with multiple profiles and targets and want to override defaults without changing your files. ```bash dbt run --profile my-profile-name --target dev ``` In this example, the `dbt run` command will use the `my-profile-name` profile and the `dev` target. #### Understanding warehouse credentials We recommend that each dbt user has their own set of database credentials, including a separate user for production runs of dbt – this helps debug rogue queries, simplifies ownerships of schemas, and improves security. To ensure the user credentials you use in your target allow dbt to run, you will need to ensure the user has appropriate privileges. While the exact privileges needed varies between data warehouses, at a minimum your user must be able to: * read source data * create schemas¹ * read system tables Running dbt without create schema privileges If your user is unable to be granted the privilege to create schemas, your dbt runs should instead target an existing schema that your user has permission to create relations within. #### Understanding target schemas The target schema represents the default schema that dbt will build objects into, and is often used as the differentiator between separate environments within a warehouse. Schemas in BigQuery dbt uses the term "schema" in a target across all supported warehouses for consistency. Note that in the case of BigQuery, a schema is actually a dataset. The schema used for production should be named in a way that makes it clear that it is ready for end-users to use for analysis – we often name this `analytics`. In development, a pattern we’ve found to work well is to name the schema in your `dev` target `dbt_<username>`. Suffixing your name to the schema enables multiple users to develop in dbt, since each user will have their own separate schema for development, so that users will not build over the top of each other, and ensuring that object ownership and permissions are consistent across an entire schema. Note that there’s no need to create your target schema beforehand – dbt will check if the schema already exists when it runs, and create it if it doesn’t. While the target schema represents the default schema that dbt will use, it may make sense to split your models into separate schemas, which can be done by using [custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md). #### Understanding threads When dbt runs, it creates a directed acyclic graph (DAG) of links between models. The number of threads represents the maximum number of paths through the graph dbt may work on at once – increasing the number of threads can minimize the run time of your project. The default value for threads in user profiles is 4 threads. For more information, check out [using threads](https://docs.getdbt.com/docs/running-a-dbt-project/using-threads.md). #### Advanced: Customizing a profile directory ##### dbt v2 dbt v2 determines the parent directory for `profiles.yml` using the following precedence: 1. `--profiles-dir` option 2. Project root directory 3. `~/.dbt/` directory Note that dbt v2 doesn't currently support the (Applies to dbt v1.11 and later) `DBT_ENGINE_PROFILES_DIR` environment variable or setting the `profiles.yml` in the current working directory. ##### dbt v1 dbt v1 determines the parent directory for `profiles.yml` using the following precedence: 1. `--profiles-dir` option 2. (Applies to dbt v1.11 and later) `DBT_ENGINE_PROFILES_DIR` environment variable 3. current working directory 4. `~/.dbt/` directory To check the expected location of your `profiles.yml` file for your installation of dbt, you can run the following: ```bash $ dbt debug --config-dir To view your profiles.yml file, run: open /Users/alice/.dbt ``` You may want to have your `profiles.yml` file stored in a different directory than `~/.dbt/` – for example, if you are [using environment variables](#advanced-using-environment-variables) to load your credentials, you might choose to include this file in the root directory of your dbt project. Note that the file always needs to be called `profiles.yml`, regardless of which directory it is in. There are multiple ways to direct dbt to a different location for your `profiles.yml` file: ##### 1. Use the `--profiles-dir` option when executing a dbt command This option can be used as follows: ```text $ dbt run --profiles-dir path/to/directory ``` If using this method, the `--profiles-dir` option needs to be provided every time you run a dbt command. ##### 2. Use the (Applies to dbt v1.11 and later) `DBT_ENGINE_PROFILES_DIR` environment variable to change the default location (dbt v1 only) Setting this environment variable tells dbt v1 to look for your `profiles.yml` file in the specified directory instead of the default location. You can specify this by running: (Applies to dbt v1.11 and later) ```text $ export DBT_ENGINE_PROFILES_DIR=path/to/directory ``` Note: This environment variable isn't supported in dbt v2. #### Advanced: Using environment variables Credentials can be placed directly into the `profiles.yml` file or loaded from environment variables. Using environment variables is especially useful for production deployments of dbt. You can find more information about environment variables [here](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md). #### Related docs * [About `profiles.yml`](https://docs.getdbt.com/docs/local/profiles.yml.md) --- ### Consume metrics from your Semantic Layer dbt platform | Starter, Enterprise, Enterprise+ After [deploying](https://docs.getdbt.com/docs/use-dbt-semantic-layer/deploy-sl.md) your Semantic Layer, the next important (and fun!) step is querying and consuming the metrics you’ve defined. This page links to key resources that guide you through the process of consuming metrics across different integrations, APIs, and tools, using various different [query syntaxes](https://docs.getdbt.com/docs/dbt-apis/sl-jdbc.md#querying-the-api-for-metric-metadata). Once your Semantic Layer is deployed, you can start querying your metrics using a variety of tools and APIs. Here are the main resources to get you started: ##### Available integrations Integrate the Semantic Layer with a variety of business intelligence (BI) tools and data platforms, enabling seamless metric queries within your existing workflows. Explore the following integrations: * [Available integrations](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) — Review a wide range of partners such as Tableau, Google Sheets, Microsoft Excel, and more, where you can query your metrics directly from the Semantic Layer. ##### Query with APIs To leverage the full power of the Semantic Layer, you can use the Semantic Layer APIs for querying metrics programmatically: * [Semantic Layer APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md) — Learn how to use the Semantic Layer APIs to query metrics in downstream tools, ensuring consistent and reliable data metrics. * [JDBC API query syntax](https://docs.getdbt.com/docs/dbt-apis/sl-jdbc.md#querying-the-api-for-metric-metadata) — Dive into the syntax for querying metrics with the JDBC API, with examples and detailed instructions. * [GraphQL API query syntax](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md#querying) — Learn the syntax for querying metrics via the GraphQL API, including examples and detailed instructions. * [Python SDK](https://docs.getdbt.com/docs/dbt-apis/sl-python.md#usage-examples) — Use the Python SDK library to query metrics programmatically with Python. ##### Query during development For developers working within the dbt ecosystem, it’s essential to understand how to query metrics during the development phase using MetricFlow commands: * [MetricFlow commands](https://docs.getdbt.com/docs/build/metricflow-commands.md) — Learn how to use MetricFlow commands to query metrics directly during the development process, ensuring your metrics are correctly defined and working as expected. #### Next steps After understanding the basics of querying metrics, consider optimizing your setup and ensuring the integrity of your metric definitions: * [Optimize querying performance](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-cache.md) — Improve query speed and efficiency by using declarative caching techniques. * [Validate semantic nodes in CI](https://docs.getdbt.com/docs/deploy/ci-jobs.md#semantic-validations-in-ci) — Ensure that any changes to dbt models don’t break your metrics by validating semantic nodes in Continuous Integration (CI) jobs. * [Build your metrics and semantic models](https://docs.getdbt.com/docs/build/build-metrics-intro.md) — If you haven’t already, learn how to define and build your metrics and semantic models using your preferred development tool. --- ### Continuous deployment in dbt dbt platform To help you improve data transformations and ship data products faster, you can run [merge jobs](https://docs.getdbt.com/docs/deploy/merge-jobs.md) to implement a continuous deployment (CD) workflow in dbt. Merge jobs can automatically build modified models whenever a pull request (PR) merges, making sure the latest code changes are in production. You don't have to wait for the next scheduled job to run to get the latest updates. [![Workflow of continuous deployment in dbt](/img/docs/dbt-platform/using-dbt-platform/cd-workflow.png?v=2 "Workflow of continuous deployment in dbt")](#)Workflow of continuous deployment in dbt You can also implement continuous integration (CI) in dbt, which can further reduce the time it takes to push changes to production and improve code quality. To learn more, refer to [Continuous integration in dbt](https://docs.getdbt.com/docs/deploy/continuous-integration.md). #### Trigger jobs with automation or APIs [Merge jobs](https://docs.getdbt.com/docs/deploy/merge-jobs.md) and [deploy jobs](https://docs.getdbt.com/docs/deploy/deploy-jobs.md) start from your [Git provider](https://docs.getdbt.com/docs/platform/git/configure-git.md) or from other triggers you configure inside dbt. Merge jobs that react to merges use the webhook flow summarized in [How merge jobs work](#how-merge-jobs-work) below and detailed under [Set up job trigger on Git merge](https://docs.getdbt.com/docs/deploy/merge-jobs.md#set-up-merge-jobs). To start the same merge or deployment job from your own tooling, use the [Administrative API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md) and [Trigger Job Run](https://docs.getdbt.com/dbt-cloud/api-v2#/operations/Trigger%20Job%20Run). Configure and validate the job in dbt first, note the account and job identifiers from the deployment URL or job settings, authenticate with [service tokens](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) or [personal access tokens](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md), assemble the HTTPS request headers and JSON body from [Trigger Job Run](https://docs.getdbt.com/dbt-cloud/api-v2#/operations/Trigger%20Job%20Run), then monitor the resulting run alongside any webhook-triggered executions in job history. * **CI triggered through Administrative API payloads** behaves differently than rerunning deployment or merge jobs. Follow [Trigger a CI job with the API](https://docs.getdbt.com/docs/deploy/ci-jobs.md#trigger-a-ci-job-with-the-api) on the [CI jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md) page. * **Integrations** with schedulers and platforms (for example [Airflow and dbt](https://docs.getdbt.com/guides/airflow-and-dbt-cloud.md), Prefect, Databricks) are collected on [Deployment tools](https://docs.getdbt.com/docs/deploy/deployment-tools.md). #### How merge jobs work When you set up merge jobs, dbt listens for notifications from your [Git provider](https://docs.getdbt.com/docs/platform/git/configure-git.md) indicating that a PR has been merged. When dbt receives one of these notifications, it enqueues a new run of the merge job. You can set up merge jobs to perform one of the following when a PR merges: | Command to run | Usage description | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `dbt build --select state:modified+` | (Default) Build the modified data with every merge.<br /><br />dbt builds only the changed data models and anything downstream of it, similar to CI jobs. This helps reduce computing costs and ensures that the latest code changes are always pushed to production. | | `dbt compile` | Refresh the applied state for performant (the slimmest) CI job runs.<br /><br />dbt generates the executable SQL (from the source model, test, and analysis files) but does not run it. This ensures the changes are reflected in the manifest for the next time a CI job is run and keeps track of only the relevant changes. | --- ### Continuous integration in dbt dbt platform To implement a continuous integration (CI) workflow in dbt, you can set up automation that tests code changes by running [CI jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md) before merging to production. dbt tracks the state of what’s running in your production environment. When you run a CI job, only the modified data assets in your pull request (PR) and their downstream dependencies are built and tested in a staging schema. You can also view the status of the CI checks (tests) directly from within the PR; this information is posted to your Git provider as soon as a CI job completes. Additionally, you can enable settings in your Git provider that allow PRs only with successful CI checks to be approved for merging. For workflows that promote changes after a PR merges (merge jobs or other deployment triggers, including triggering jobs manually or through APIs), refer to [Continuous deployment in dbt](https://docs.getdbt.com/docs/deploy/continuous-deployment.md), including [merge jobs](https://docs.getdbt.com/docs/deploy/merge-jobs.md). [![Workflow of continuous integration in dbt](/img/docs/dbt-platform/using-dbt-platform/ci-workflow.png?v=2 "Workflow of continuous integration in dbt")](#)Workflow of continuous integration in dbt Using CI helps: * Provide increased confidence and assurances that project changes will work as expected in production. * Reduce the time it takes to push code changes to production, through build and test automation, leading to better business outcomes. * Allow organizations to make code changes in a standardized and governed way that ensures code quality without sacrificing speed. #### How CI works When you [set up CI jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md#set-up-ci-jobs), dbt listens for a notification from your Git provider indicating that a new PR has been opened or updated with new commits. When dbt receives one of these notifications, it enqueues a new run of the CI job. dbt builds and tests models, semantic models, metrics, and saved queries affected by the code change in a temporary schema, unique to the PR. This process ensures that the code builds without error and that it matches the expectations as defined by the project's dbt tests. The unique schema name follows the naming convention `dbt_cloud_pr_<job_id>_<pr_id>` (for example, `dbt_cloud_pr_1862_1704`) and can be found in the run details for the given run, as shown in the following image: [![Viewing the temporary schema name for a run triggered by a PR](/img/docs/dbt-platform/using-dbt-platform/using_ci_dbt_cloud.png?v=2 "Viewing the temporary schema name for a run triggered by a PR")](#)Viewing the temporary schema name for a run triggered by a PR When the CI run completes, you can view the run status directly from within the pull request. dbt updates the pull request in GitHub, GitLab, or Azure DevOps with a status message indicating the results of the run. The status message states whether the models and tests ran successfully or not. dbt deletes the temporary schema from your data warehouse when you close or merge the pull request. If your project has schema customization using the [generate\_schema\_name](https://docs.getdbt.com/docs/build/custom-schemas.md#how-does-dbt-generate-a-models-schema-name) macro, dbt might not drop the temporary schema from your data warehouse. For more information, refer to [Troubleshooting](https://docs.getdbt.com/docs/deploy/ci-jobs.md#troubleshooting). #### Availability of features by Git provider * If your git provider has a [native dbt integration](https://docs.getdbt.com/docs/platform/git/configure-git.md), you can seamlessly set up [continuous integration (CI)](https://docs.getdbt.com/docs/deploy/ci-jobs.md) jobs directly within dbt. * For providers without native integration, you can still use the [Git clone method](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md) to import your git URL and leverage the [dbt Administrative API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md) to trigger a CI job to run. The following table outlines the available integration options and their corresponding capabilities. | **Git provider** | **Native dbt integration** | **Automated CI job** | **Git clone** | **Information** | **Supported plans** | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------- | -------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------- | | [Azure DevOps](https://docs.getdbt.com/docs/platform/git/connect-azure-devops.md)<br /> | ✅ | ✅ | ✅ | Organizations on the Starter and Developer plans can connect to Azure DevOps using a deploy key. Note, you won’t be able to configure automated CI jobs but you can still develop. | Enterprise, Enterprise+ | | [GitHub](https://docs.getdbt.com/docs/platform/git/connect-github.md)<br /> | ✅ | ✅ | ✅ | | All dbt plans | | [GitLab](https://docs.getdbt.com/docs/platform/git/connect-gitlab.md)<br /> | ✅ | ✅ | ✅ | | All dbt plans | | All other git providers using [Git clone](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md) ([BitBucket](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md#bitbucket), [AWS CodeCommit](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md#aws-codecommit), and others) | ❌ | ❌ | ✅ | Refer to the [Customizing CI/CD with custom pipelines](https://docs.getdbt.com/guides/custom-cicd-pipelines.md?step=1) guide to set up continuous integration and continuous deployment (CI/CD). | | #### Differences between CI jobs and other deployment jobs The [dbt scheduler](https://docs.getdbt.com/docs/deploy/job-scheduler.md) executes CI jobs differently from other deployment jobs in these important ways: * [**Concurrent CI checks**](#concurrent-ci-checks) — The scheduler can execute multiple CI runs from the same dbt CI job concurrently (in parallel) when appropriate. * [**Smart cancellation of stale builds**](#smart-cancellation-of-stale-builds) — The scheduler cancels stale in-flight CI runs when you push new commits to the PR. * [**Run slot treatment**](#run-slot-treatment) — CI runs don't consume a run slot. * [**SQL linting**](#sql-linting) — When enabled, linting runs on all SQL files in your project as a step before your CI job builds. ##### Concurrent CI checks [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") When you collaborate on a dbt project with your team and open pull requests in the same dbt repository, the same CI job can run for each qualifying event. Each CI run writes to a dedicated, temporary schema that is tied to its pull request. That isolation lets dbt execute CI runs *concurrently* instead of *sequentially*. This differs from deployment dbt jobs. You don't have to wait for someone else’s CI run to finish before your own check can start. Concurrent CI checks help your whole team test and integrate dbt code faster. The following describes the conditions when CI checks are run concurrently and when they’re not: * CI runs with different PR numbers execute concurrently. * CI runs with the *same* PR number and *different* commit SHAs execute serially because they’re building into the same schema. dbt will run the latest commit and cancel any older, stale commits. For details, refer to [Smart cancellation of stale builds](#smart-cancellation). * CI runs with the same PR number and same commit SHA can still execute concurrently when they come from different dbt projects. This can happen when two CI jobs are set up in different dbt projects that share the same dbt repository. ##### Smart cancellation of stale builds [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") When you push a new commit to a PR, dbt enqueues a new CI run for the latest commit and cancels any CI run that is (now) stale and still in flight. This can happen when you’re pushing new commits while a CI build is still in process and not yet done. By canceling runs in a safe and deliberate way, dbt helps improve productivity and reduce data platform spend on wasteful CI runs. [![Example of an automatically canceled run](/img/docs/dbt-platform/using-dbt-platform/example-smart-cancel-job.png?v=2 "Example of an automatically canceled run")](#)Example of an automatically canceled run ##### Run slot treatment [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") CI runs don't consume run slots. This guarantees a CI check will never block a production run. ##### SQL linting [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") Available on [dbt release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) and dbt Starter or Enterprise-tier accounts. When [enabled for your CI job](https://docs.getdbt.com/docs/deploy/ci-jobs.md#set-up-ci-jobs), dbt lints the changed SQL files in your project. On v1 runs, dbt invokes [SQLFluff](https://sqlfluff.com/), a modular, configurable SQL linter. On v2 runs, dbt uses the built-in `dbt lint` command instead (see below). Linting warns you about complex functions, syntax, formatting, and compilation errors. Linting on dbt v2 CI jobs that run on v2 automatically use the built-in [`dbt lint`](https://docs.getdbt.com/reference/commands/lint.md?version=2.0) command instead of SQLFluff. `dbt lint` is SQLFluff-compatible and it reads your existing `.sqlfluff` config, uses the same rule codes, and respects `-- noqa` suppression comments. For parity expectations between `dbt lint` and SQLFluff, refer to [Rule parity with SQLFluff](https://docs.getdbt.com/reference/commands/lint.md?version=2.0#rule-parity-with-sqlfluff). By default, SQL linting lints all the changed SQL files in your project, compared to the last deferred production state. Note that [snapshots](https://docs.getdbt.com/docs/build/snapshots.md) can be defined in YAML *and* `.sql` files. Their SQL isn't lintable and can cause errors during linting. To prevent SQLFluff from linting snapshot files, add the snapshots directory to your `.sqlfluffignore` file (for example `snapshots/`). Refer to [snapshot linting](https://docs.getdbt.com/docs/platform/studio-ide/lint-format.md#snapshot-linting) for more information. If the linter runs into errors, you can specify whether dbt should stop running the job on error or continue running it on error. When failing jobs, it helps reduce compute costs by avoiding builds for pull requests that don't meet your SQL code quality CI check. ###### To configure SQLFluff linting You can optionally configure SQLFluff linting rules to override default linting behavior. * Use [SQLFluff Configuration Files](https://docs.sqlfluff.com/en/stable/configuration/setting_configuration.html#configuration-files) to override the default linting behavior in dbt. * Create a `.sqlfluff` configuration file in your project, add your linting rules to it, and dbt will use them when linting. * When configuring, you can use `dbt` as the templater (for example, `templater = dbt`) * If you’re using the Studio IDE, dbt CLI, or any other editor, refer to [Customize linting](https://docs.getdbt.com/docs/platform/studio-ide/lint-format.md#customize-linting) for guidance on how to add the dbt-specific (or dbtonic) linting rules we use for our own project. * For complete details, refer to [Custom Usage](https://docs.sqlfluff.com/en/stable/gettingstarted.html#custom-usage) in the SQLFluff documentation. --- ### Continuous integration jobs in dbt dbt platform You can set up [continuous integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md) (CI) jobs to run when someone opens a new pull request (PR) in your Git repository. By running and testing only *modified* models, dbt keeps these jobs as resource-conscious as possible on your data platform. Triggering CI jobs in monorepos If you have a monorepo with several dbt projects, opening a single pull request in one of your projects will trigger jobs for all projects connected to the monorepo. To address this, you can use separate target branches per project (for example, `main-project-a`, `main-project-b`) to separate CI triggers. #### Prerequisites * You have a dbt account. * CI features: * For both the [concurrent CI checks](https://docs.getdbt.com/docs/deploy/continuous-integration.md#concurrent-ci-checks) and [smart cancellation of stale builds](https://docs.getdbt.com/docs/deploy/continuous-integration.md#smart-cancellation) features, your dbt account must be on the [Starter, Enterprise, or Enterprise+ plan](https://www.getdbt.com/pricing/). * [SQL linting](https://docs.getdbt.com/docs/deploy/continuous-integration.md#sql-linting) is available on [dbt release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) and to dbt [Starter, Enterprise, or Enterprise+](https://www.getdbt.com/pricing/) accounts. Refer to [Configure SQLFluff linting](https://docs.getdbt.com/docs/deploy/continuous-integration.md#to-configure-sqlfluff-linting) when you add SQLFluff to your project. Linting on dbt v2 CI jobs that run on v2 automatically use the built-in [`dbt lint`](https://docs.getdbt.com/reference/commands/lint.md?version=2.0) command instead of SQLFluff. `dbt lint` is SQLFluff-compatible and it reads your existing `.sqlfluff` config, uses the same rule codes, and respects `-- noqa` suppression comments. * [Advanced CI](https://docs.getdbt.com/docs/deploy/advanced-ci.md) features: * For the [compare changes](https://docs.getdbt.com/docs/deploy/advanced-ci.md#compare-changes) feature, your dbt account must be on an [Enterprise-tier plan](https://www.getdbt.com/pricing/) and have enabled Advanced CI features. Please ask your [dbt administrator to enable](https://docs.getdbt.com/docs/platform/account-settings.md#account-access-to-advanced-ci-features) this feature for you. After enablement, the **dbt compare** option becomes available in the CI job settings. * Set up a [connection with your Git provider](https://docs.getdbt.com/docs/platform/git/configure-git.md). This integration lets dbt run jobs on your behalf for job triggering. * If you're using a native [GitLab](https://docs.getdbt.com/docs/platform/git/connect-gitlab.md) integration, you need a paid or self-hosted account that includes support for GitLab webhooks and [project access tokens](https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html). If you're using GitLab Free, merge requests will trigger CI jobs but CI job status updates (success or failure of the job) will not be reported back to GitLab. #### Availability of features by Git provider * If your git provider has a [native dbt integration](https://docs.getdbt.com/docs/platform/git/configure-git.md), you can seamlessly set up [continuous integration (CI)](https://docs.getdbt.com/docs/deploy/ci-jobs.md) jobs directly within dbt. * For providers without native integration, you can still use the [Git clone method](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md) to import your git URL and leverage the [dbt Administrative API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md) to trigger a CI job to run. The following table outlines the available integration options and their corresponding capabilities. | **Git provider** | **Native dbt integration** | **Automated CI job** | **Git clone** | **Information** | **Supported plans** | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------- | -------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------- | | [Azure DevOps](https://docs.getdbt.com/docs/platform/git/connect-azure-devops.md)<br /> | ✅ | ✅ | ✅ | Organizations on the Starter and Developer plans can connect to Azure DevOps using a deploy key. Note, you won’t be able to configure automated CI jobs but you can still develop. | Enterprise, Enterprise+ | | [GitHub](https://docs.getdbt.com/docs/platform/git/connect-github.md)<br /> | ✅ | ✅ | ✅ | | All dbt plans | | [GitLab](https://docs.getdbt.com/docs/platform/git/connect-gitlab.md)<br /> | ✅ | ✅ | ✅ | | All dbt plans | | All other git providers using [Git clone](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md) ([BitBucket](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md#bitbucket), [AWS CodeCommit](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md#aws-codecommit), and others) | ❌ | ❌ | ✅ | Refer to the [Customizing CI/CD with custom pipelines](https://docs.getdbt.com/guides/custom-cicd-pipelines.md?step=1) guide to set up continuous integration and continuous deployment (CI/CD). | | #### Set up CI jobs dbt Labs recommends that you create your CI job in a dedicated dbt [deployment environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md#create-a-deployment-environment) that's connected to a staging database. A separate CI environment improves isolation between your temporary CI schemas and production builds. You can trigger CI jobs for pull requests targeting a branch other than `main`. For example, if you use a long-lived staging branch, you can associate a CI environment with that branch via [custom branch](https://docs.getdbt.com/faqs/Environments/custom-branch-settings.md) settings. Then the job in that environment runs only when someone opens PRs against that branch. To learn more, refer to [Get started with CI tests](https://docs.getdbt.com/guides/set-up-ci.md). To make CI job creation easier, many options on the **CI job** page are set to default values that dbt Labs recommends that you use. If you don't want to use the defaults, you can change them. 1. On your deployment environment page, click **Create job** > **Continuous integration job** to create a new CI job. 2. Options in the **Job settings** section: * **Job name** — Specify the name for this CI job. * **Description** — Provide a description about the CI job. * **Environment** — By default, this will be set to the environment you created the CI job from. Use the dropdown to change the default setting. 3. Options in the **Git trigger** section: * **Triggered by pull requests** — By default, it’s enabled. Every time a developer opens up a pull request or pushes a commit to an existing pull request, this job will get triggered to run. * **Run on draft pull request** — Enable this option if you want to also trigger the job to run every time a developer opens up a draft pull request or pushes a commit to that draft pull request. 4. Options in the **Execution settings** section: * **Commands** — By default, this includes the `dbt build --select state:modified+` command. This informs dbt to build only new or changed models and their downstream dependents. Importantly, state comparison can only happen when there is a deferred environment selected to compare state to. Click **Add command** to add more [commands](https://docs.getdbt.com/docs/deploy/job-commands.md) that you want to be invoked when this job runs. * **Linting** — Enable this option for dbt to [lint the SQL files](https://docs.getdbt.com/docs/deploy/continuous-integration.md#sql-linting) in your project as the first step in `dbt run`. If this check runs into an error, dbt can either **Stop running on error** or **Continue running on error**. * **dbt compare**[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") — Enable this option to compare the last applied state of the production environment (if one exists) with the latest changes from the pull request, and identify what those differences are. To enable record-level comparison and primary key analysis, you must add a [primary key constraint](https://docs.getdbt.com/reference/resource-properties/constraints.md) or [uniqueness test](https://docs.getdbt.com/reference/resource-properties/data-tests.md#unique). Otherwise, you'll receive a "Primary key missing" error message in dbt. To review the comparison report, navigate to the [Compare tab](https://docs.getdbt.com/docs/deploy/run-visibility.md#compare-tab) in the job run's details. A summary of the report is also available from the pull request in your Git provider (refer to the [CI report example](#example-ci-report)). Optimization tip When you enable the **dbt compare** checkbox, you can customize the comparison command to optimize your CI job. For example, if you have large models that take a long time to compare, you can exclude them to speed up the process using the [`--exclude` flag](https://docs.getdbt.com/reference/node-selection/exclude.md). Refer to [compare changes custom commands](https://docs.getdbt.com/docs/deploy/job-commands.md#compare-changes-custom-commands) for more details. Additionally, if you set [`event_time`](https://docs.getdbt.com/reference/resource-configs/event-time.md) in your models/seeds/snapshots/sources, it allows you to compare matching date ranges between tables by filtering to overlapping date ranges. This is useful for faster CI workflows or custom sampling setups. * **Compare changes against an environment (Deferral)** — By default, it’s set to the **Production** environment if you created one. This option allows dbt to check the state of the code in the PR against the code running in the deferred environment, so as to only check the modified code, instead of building the full table or the entire DAG. Comparison manifests The latest successful run in the environment can come from *any* job that updates artifacts there. If many jobs run with different settings, your CI comparison state can change in ways that are hard to predict. Even when you defer to Production, a merge or deploy that refreshes the manifest while other pull requests are open can make those runs pick up unrelated `state:modified` nodes until branches are updated. * **Run timeout** — Cancel the CI job if the run time exceeds the timeout value. You can use this option to help ensure that a CI check doesn't consume too much of your warehouse resources. If you enable the **dbt compare** option, the timeout value defaults to `3600` (one hour) to prevent long-running comparisons. * **Enable dbt State** [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") — [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) reduces unnecessary model rebuilds by reusing nodes when neither the logic nor the data has changed. For more details, refer to [Setting up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) and [Enabling dbt State on individual jobs](https://docs.getdbt.com/docs/deploy/dbt-state-enable-jobs.md). 5. (optional) Options in the **Advanced settings** section: * **Environment variables** — Define [environment variables](https://docs.getdbt.com/docs/build/environment-variables.md) to customize the behavior of your project when this CI job runs. You can specify that a CI job is running in a *Staging* or *CI* environment by setting an environment variable and modifying your project code to behave differently, depending on the context. It's common for teams to process only a subset of data for CI runs, using environment variables to branch logic in their dbt project code. * **Target name** — Define the [target name](https://docs.getdbt.com/docs/build/custom-target-names.md). Similar to **Environment Variables**, this option lets you customize the behavior of the project. You can use this option to specify that a CI job is running in a *Staging* or *CI* environment by setting the target name and modifying your project code to behave differently, depending on the context. * **dbt version** — By default, it’s set to inherit the [dbt version](https://docs.getdbt.com/docs/dbt-versions.md) from the environment. dbt Labs strongly recommends that you don't change the default setting. This option to change the version at the job level is useful only when you upgrade a project to the next dbt version; otherwise, mismatched versions between the environment and job can lead to confusing behavior. * **Threads** — By default, it’s set to 4 [threads](https://docs.getdbt.com/docs/local/profiles.yml.md#understanding-threads). Increase the thread count to increase model execution concurrency. * **Generate docs on run** — Enable this if you want to [generate project docs](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) when this job runs. This is disabled by default since testing doc generation on every CI check is not a recommended practice. * **Run source freshness** — Enable this option to invoke the `dbt source freshness` command before running this CI job. Refer to [Source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) for more details. [![Example of CI Job page in the dbt UI](/img/docs/dbt-platform/using-dbt-platform/create-ci-job.png?v=2 "Example of CI Job page in the dbt UI")](#)Example of CI Job page in the dbt UI ##### Example of CI check in pull request The following is an example of a CI check in a GitHub pull request. The green checkmark means the dbt build and tests were successful. Clicking on the dbt section takes you to the relevant CI run in dbt. [![Example of CI check in GitHub pull request](/img/docs/dbt-platform/using-dbt-platform/example-github-pr.png?v=2 "Example of CI check in GitHub pull request")](#)Example of CI check in GitHub pull request ##### Example of CI report in pull request [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") The following is an example of a CI report in a GitHub pull request, which is shown when the **dbt compare** option is enabled for the CI job. It displays a high-level summary of the models that changed from the pull request. [![Example of CI report comment in GitHub pull request](/img/docs/dbt-platform/using-dbt-platform/example-github-ci-report.png?v=2 "Example of CI report comment in GitHub pull request")](#)Example of CI report comment in GitHub pull request #### Trigger a CI job with the API [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") If you're not using dbt’s native Git integration with [GitHub](https://docs.getdbt.com/docs/platform/git/connect-github.md), [GitLab](https://docs.getdbt.com/docs/platform/git/connect-gitlab.md), or [Azure DevOps](https://docs.getdbt.com/docs/platform/git/connect-azure-devops.md), you can use the [Administrative API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md) to trigger a CI job to run. However, dbt will not automatically delete the temporary schema for you. This is because automatic deletion relies on incoming webhooks from Git providers, which is only available through the native integrations. If you instead need workflows that run after a merge (not CI checks on an open pull request), refer to [Continuous deployment in dbt](https://docs.getdbt.com/docs/deploy/continuous-deployment.md) and [Merge jobs](https://docs.getdbt.com/docs/deploy/merge-jobs.md). ##### Prerequisites * You have a dbt account. * You have a dbt [Enterprise or Enterprise+ plan](https://www.getdbt.com/pricing/). Legacy Team plans also retain access. * For the [Concurrent CI checks](https://docs.getdbt.com/docs/deploy/continuous-integration.md#concurrent-ci-checks) and [Smart cancellation of stale builds](https://docs.getdbt.com/docs/deploy/continuous-integration.md#smart-cancellation) features, your dbt account must be on the [Enterprise or Enterprise+ plan](https://www.getdbt.com/pricing/), and legacy Team plans. Starter plans do not have access to these features when triggering a CI job with the API. 1. Set up a CI job with the [Create Job](https://docs.getdbt.com/dbt-cloud/api-v2#/operations/Create%20Job) API endpoint using `"job_type": ci` or from the [dbt UI](#set-up-ci-jobs). 2. Call the [Trigger Job Run](https://docs.getdbt.com/dbt-cloud/api-v2#/operations/Trigger%20Job%20Run) API endpoint to trigger the CI job. You must include both of these fields to the payload: * Provide the pull request (PR) ID using one of these fields: * `github_pull_request_id` * `gitlab_merge_request_id` * `azure_devops_pull_request_id` * `non_native_pull_request_id` (for example, Bitbucket) * Provide the `git_sha` or `git_branch` to target the correct commit or branch to run the job against. #### Semantic validations in CI [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") Automatically test your semantic nodes (metrics, semantic models, and saved queries) during code reviews by adding warehouse validation checks in your CI job, guaranteeing that any code changes made to dbt models don't break these metrics. To do this, add the command `dbt sl validate --select state:modified+` in the CI job. This ensures the validation of modified semantic nodes and their downstream dependencies. [![Semantic validations in CI workflow](/img/docs/dbt-platform/deployment/sl-ci-job.png?v=2 "Semantic validations in CI workflow")](#)Semantic validations in CI workflow ###### Benefits * Testing semantic nodes in a CI job supports deferral and selection of semantic nodes. * It allows you to catch issues early in the development process and deliver high-quality data to your end users. * Semantic validation executes an explain query in the data warehouse for semantic nodes to ensure the generated SQL will execute. * For semantic nodes and models that aren't downstream of modified models, dbt defers to the production models. ##### Set up semantic validations in your CI job To learn how to set this up, refer to the following steps: 1. Click **Orchestration**. 2. Click **Jobs**. 3. Select the relevant job. 4. Click **Settings**. 5. Click **Edit**. 6. Add the `dbt sl validate --select state:modified+` command to the **Commands** field in the **Execution settings** section. The command uses state selection and deferral to run validation on any semantic nodes downstream of model changes. To reduce job times, we recommend only running CI on modified semantic models. 7. Click **Save** to save your changes. There are additional commands and use cases described in the [next section](#use-cases), such as validating all semantic nodes, validating specific semantic nodes, and so on. [![Validate semantic nodes downstream of model changes in your CI job.](/img/docs/dbt-platform/deployment/ci-dbt-sl-validate-downstream.png?v=2 "Validate semantic nodes downstream of model changes in your CI job.")](#)Validate semantic nodes downstream of model changes in your CI job. ##### Use cases Use or combine different selectors or commands to validate semantic nodes in your CI job. Semantic validations in CI support the following use cases:  Semantic nodes downstream of model changes (recommended) To validate semantic nodes that are downstream of a model change, add the two commands in your job **Execution settings** section: ```bash dbt build --select state:modified+ dbt sl validate --select state:modified+ ``` * The first command builds the modified models. * The second command validates the semantic nodes downstream of the modified models. Before running semantic validations, dbt must build the modified models. This process ensures that downstream semantic nodes are validated using the CI schema through the dbt Semantic Layer API. For semantic nodes and models that aren't downstream of modified models, dbt defers to the production models. [![Validate semantic nodes downstream of model changes in your CI job.](/img/docs/dbt-platform/deployment/ci-dbt-sl-validate-downstream.png?v=2 "Validate semantic nodes downstream of model changes in your CI job.")](#)Validate semantic nodes downstream of model changes in your CI job.  Semantic nodes that are modified or affected by downstream modified nodes. To only validate modified semantic nodes, use the following command (with [state selection](https://docs.getdbt.com/reference/node-selection/state-selection.md)): ```bash dbt sl validate --select state:modified+ ``` [![Use state selection to validate modified metric definition models in your CI job.](/img/docs/dbt-platform/deployment/ci-dbt-sl-validate-modified.png?v=2 "Use state selection to validate modified metric definition models in your CI job.")](#)Use state selection to validate modified metric definition models in your CI job. This will only validate semantic nodes. It will use the defer state set configured in your orchestration job, deferring to your production models.  Select specific semantic nodes Use the selector syntax to select the *specific* semantic node(s) you want to validate: ```bash dbt sl validate --select metric:revenue ``` [![Use state selection to validate modified metric definition models in your CI job.](/img/docs/dbt-platform/deployment/ci-dbt-sl-validate-select.png?v=2 "Use state selection to validate modified metric definition models in your CI job.")](#)Use state selection to validate modified metric definition models in your CI job. In this example, the CI job will validate the selected `metric:revenue` semantic node. To select multiple semantic nodes, use the selector syntax: `dbt sl validate --select metric:revenue metric:customers`. If you don't specify a selector, dbt will validate all semantic nodes in your project.  Select all semantic nodes To validate *all* semantic nodes in your project, add the following command to defer to your production schema when generating the warehouse validation queries: ```bash dbt sl validate ``` [![Validate all semantic nodes in your CI job by adding the command: 'dbt sl validate' in your job execution settings.](/img/docs/dbt-platform/deployment/ci-dbt-sl-validate-all.png?v=2 "Validate all semantic nodes in your CI job by adding the command: 'dbt sl validate' in your job execution settings.")](#)Validate all semantic nodes in your CI job by adding the command: 'dbt sl validate' in your job execution settings. #### Troubleshooting Unable to trigger a CI job with GitLab When you connect dbt to a GitLab repository, GitLab automatically registers a webhook in the background, viewable under the repository settings. This webhook is also used to trigger [CI jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md) when you push to the repository. If you're unable to trigger a CI job, this usually indicates that the webhook registration is missing or incorrect. To resolve this issue, navigate to the repository settings in GitLab and view the webhook registrations by navigating to GitLab --> **Settings** --> **Webhooks**. Some things to check: * The webhook registration is enabled in GitLab. * The webhook registration is configured with the correct URL and secret. If you're still experiencing this issue, reach out to the Support team at <support@getdbt.com> and we'll be happy to help!  CI selects or fails on models that are not in my pull request This usually means the comparison manifest does not line up with your branch’s commits. Another job overwrote `manifest.json` with different settings, Production (or another deferred environment) advanced after someone else merged while your pull request stayed open, or the manifest is stale right after a merge. Built-in deferral targets an environment, not a hand-picked job ID. To resolve this, merge or rebase the latest base branch into your pull request so your branch includes recent merges. To refresh the comparison manifest without waiting on a long deploy, use a [merge job](https://docs.getdbt.com/docs/deploy/merge-jobs.md) in the same environment your CI job defers to. For example, one that runs `dbt parse --no-partial-parse` (or [`dbt compile`](https://docs.getdbt.com/reference/commands/compile.md)) immediately after merges.  CI jobs aren't triggering occasionally when opening a PR using the Azure DevOps (ADO) integration dbt won't trigger a CI job run if the latest commit in a pull or merge request has already triggered a run for that job. However, some providers (like GitHub) will enforce the result of the existing run on multiple pull/merge requests. Scenarios where dbt does not trigger a CI job with Azure DevOps: 1. Reusing a branch in a new PR * If you abandon a previous PR (PR 1) that triggered a CI job for the same branch (`feature-123`) merging into `main`, and then open a new PR (PR 2) with the same branch merging into`main` — dbt won't trigger a new CI job for PR 2. 2. Reusing the same commit * If you create a new PR (PR 2) on the same commit (`#4818ceb`) as a previous PR (PR 1) that triggered a CI job — dbt won't trigger a new CI job for PR 2.  Temporary schemas aren't dropping If your temporary schemas aren't dropping after a PR merges or closes, this typically indicates one of these issues: * You have overridden the `generate_schema_name` macro and it isn't using `dbt_cloud_pr_` as the prefix. To resolve this, change your macro so that the temporary PR schema name contains the required prefix. For example: * ✅ Temporary PR schema name contains the prefix `dbt_cloud_pr_` (like `dbt_cloud_pr_123_456_marketing`). * ❌ Temporary PR schema name doesn't contain the prefix `dbt_cloud_pr_` (like `marketing`). A macro is creating a schema but there are no dbt models writing to that schema. dbt doesn't drop temporary schemas that weren't written to as a result of running a dbt model.  Error messages that refer to schemas from previous PRs If you receive a schema-related error message referencing a *previous* PR, this is usually an indicator that you are not using a production job for your deferral and are instead using *self*. If the prior PR has already been merged, the prior PR's schema may have been dropped by the time the CI job for the current PR is kicked off. To fix this issue, select a production job run to defer to instead of self.  Production job runs failing at the 'Clone Git Repository step' dbt can only check out commits that belong to the original repository. dbt *cannot* checkout commits that belong to a fork of that repository. If you receive the following error message at the **Clone Git Repository** step of your job run: ```text Error message: Cloning into '/tmp/jobs/123456/target'... Successfully cloned repository. Checking out to e845be54e6dc72342d5a8f814c8b3316ee220312...> Failed to checkout to specified revision. git checkout e845be54e6dc72342d5a8f814c8b3316ee220312 fatal: reference is not a tree: e845be54e6dc72342d5a8f814c8b3316ee220312 ``` Double-check that your PR isn't trying to merge using a commit that belongs to a fork of the repository attached to your dbt project.  CI job not triggering for Virtual Private dbt users To trigger jobs on dbt using the [API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md), your Git provider needs to connect to your dbt account. If you're on a Virtual Private dbt Enterprise plan using security features like ingress PrivateLink or IP Allowlisting, registering CI hooks may not be available and can cause the job to fail silently.  PR status for CI job stays in 'pending' in Azure DevOps after job run finishes When you start a CI job, the pull request status should show as `pending` while it waits for an update from dbt. Once the CI job finishes, dbt sends the status to Azure DevOps (ADO), and the status will change to either `succeeded` or `failed`. If the status doesn't get updated after the job runs, check if there are any git branch policies in place blocking ADO from receiving these updates. One potential issue is the **Reset conditions** under **Status checks** in the ADO repository branch policy. If you enable the **Reset status whenever there are new changes** checkbox (under **Reset conditions**), it can prevent dbt from updating ADO about your CI job run status. You can find relevant information here: * [Azure DevOps Services Status checks](https://learn.microsoft.com/en-us/azure/devops/repos/git/branch-policies?view=azure-devops\&tabs=browser#status-checks) * [Azure DevOps Services Pull Request Stuck Waiting on Status Update](https://support.hashicorp.com/hc/en-us/articles/18670331556627-Azure-DevOps-Services-Pull-Request-Stuck-Waiting-on-Status-Update-from-Terraform-Cloud-Enterprise-Run) * [Pull request status](https://learn.microsoft.com/en-us/azure/devops/repos/git/pull-request-status?view=azure-devops#pull-request-status) --- ### Contribute a dbt v2 adapter Available in v2 | Local development Work in progress This guide is a work in progress — synthesized from internal v2 adapter team docs, DuckDB/ClickHouse reference implementations, the Exasol community adapter, and the v1 adapter creation guide. **Community contribution scope:** You are responsible for **Phase 1 (foundation)** only — connecting, running macros, and basic materializations. Phase 2 (SQL static analysis) is led by dbt Labs after your Phase 1 PR is merged. #### Step 1: Introduction dbt v2 adapters work very differently from dbt v1 adapters. In v1, each adapter was a standalone Python package that implemented a fragmented Python interface. In v2, adapters live **inside a monorepo written in Rust**, connected to warehouses via ADBC (Arrow Database Connectivity) drivers — and the community contribution model has changed accordingly. This guide walks you through contributing a new v2 adapter to `dbt-labs/dbt` as a community member. The file breakdown in the [reference section](#reference-file-by-file-implementation-guide) shows the ~13 files a complete community v2 adapter touches. Exasol, a community contributed v2 adapter, is the example used throughout this guide. What is dbt v2? dbt v2 is the new Rust-based dbt engine. Adapters in v2 are written in Rust and live inside the `dbt-labs/dbt` monorepo, rather than as standalone Python packages. ##### How adapters are different now In dbt v1, every adapter is: * A **separate Python package** (e.g. `dbt-snowflake`, `dbt-bigquery`) * **Community-owned** and maintained independently * **Dynamically loaded** at runtime via Python's plugin system * Built upon vendor-maintained Python SDK connectors/drivers In dbt v2, adapters are: * **Part of the `dbt-labs/dbt` monorepo**, contributed as PRs * **Written in Rust** (with Jinja SQL macros still used for SQL logic) * Connected to warehouses via **ADBC drivers** — a unified driver interface that abstracts away connection management * Organized by **vertical** (feature area) rather than by warehouse — meaning a bug fix in one crate often benefits all adapters The practical upside: **adding a new adapter requires far less code than it did in v1**, and the cost of development decreases as more adapters are added because shared logic increases. ##### What you'll be building As a community contributor, you're building the **foundation**: connect your warehouse, run dbt macros, and support basic materializations (`dbt run`, `dbt build`). The ~13 files in the reference section define the complete scope. *** #### Step 2: Prerequisites ##### Background knowledge Before starting, you should: * **Have some Rust familiarity** — you do not need to be a Rust expert (an LLM will likely do most of the code writing). You need to be able to read Rust, understand enum match expressions, and make sense of compiler errors. If Rust is new to you, skim [The Rust Book](https://doc.rust-lang.org/book/) Chapter 6 (enums and pattern matching) — those are the concepts you'll encounter most. * **Understand dbt fundamentals** — how [`profiles.yml`](https://docs.getdbt.com/docs/core/connect-data-platform/profiles.yml) works, what [materializations](https://docs.getdbt.com/docs/build/materializations) are, and what [adapter dispatch](https://docs.getdbt.com/reference/dbt-jinja-functions/adapter) macros do. * **Understand your query engine** and how it differs from others. The following questions cover the key considerations for building an adapter: * What character is used for quoting identifiers? * Are three-part names supported, or only two? * Are transactions supported? * What's the most performant way to get all the columns in a schema? * What datatype represents a string? * What connection fields does your warehouse require? * **Have an LLM coding assistant available** (optional) — the adapter development workflow is designed with AI-assisted coding in mind, and this guide calls out where to use it. ##### Your ADBC driver The ADBC driver for your warehouse must already exist before you start dbt Labs does not write drivers. If your warehouse doesn't have an ADBC driver yet, that work comes first — it's a prerequisite you can't skip. **What "having an ADBC driver" actually means** For certain adapters (Snowflake, BigQuery, Databricks, Redshift, DuckDB, ClickHouse, Salesforce, Spark, SQL Server), v2 automatically downloads the correct driver binary from the dbt Labs CDN on first use. Users never have to think about it. Community adapters don't have CDN support. Instead, dbt v2 looks for a shared library by name on the user's system — e.g. `libadbc_driver_exasol.dylib` on macOS, `libadbc_driver_exasol.so` on Linux. If the file isn't present, the connection fails at runtime. This means two things for you as a contributor: 1. **The driver binary must exist somewhere.** It's a compiled shared library (`.dylib`/`.so`/`.dll`) that implements the ADBC C ABI for your warehouse. Exasol's driver is `exarrow-rs`, a separate Rust crate maintained by Exasol — not part of `dbt-core` and not in `apache/arrow-adbc`. Your warehouse's driver will likely live in a similar separate repo. 2. **Your users will need to install it manually.** Unlike first-party adapters, there's no auto-download. Your documentation (Step 6) needs to tell users where to get the driver and how to install it. Check whether a driver already exists for your warehouse: * [dbt-labs/arrow-adbc](https://github.com/dbt-labs/arrow-adbc) — the dbt Labs fork * [apache/arrow-adbc](https://github.com/apache/arrow-adbc) — the upstream community repo * [adbc-drivers](https://github.com/orgs/adbc-drivers/repositories?type=all) — the [ADBC driver foundry](https://adbc-drivers.org/2025/10/29/announcing-adbc-driver-foundry.html), the primary community home for ADBC drivers, with active drivers for Trino, MySQL, BigQuery, Databricks, Snowflake, and more; also includes Rust/Go frameworks (`driverbase-rs`, `template-rs`) for building new drivers from scratch * The warehouse vendor's own GitHub org (many vendors publish ADBC drivers independently) Driver source is flexible — with a security caveat dbt v2 loads drivers by shared library name from the system path (e.g. `libadbc_driver_exasol.dylib` on macOS) — the source repository is flexible, as long as the binary implements the ADBC C ABI. dbt Labs is working on a driver signing and verification mechanism. Until that ships, users are responsible for trusting the driver binary they install. If no driver exists yet, building one is a separate project that comes before the adapter contribution. This is outside the scope of what dbt Labs can help with. [Columnar](https://columnar.tech/) specializes in building ADBC drivers and may be a useful resource if you need help getting a driver built. ##### Are you porting an existing v1 adapter? Building from scratch with no v1 adapter? Skip to the checklist below, then go straight to Step 4. You won't have macro SQL or connection logic to port, but the rest of the guide (crate structure, Rust patterns, AI workflow) still applies. Most community contributors aren't building from scratch — they're porting an adapter they already maintain or use in Python for dbt v1. If that's you, **you have a significant head start**, and this is the most realistic path for the vast majority of contributors. Before writing any Rust, check a few things: **1. Does a v1 adapter already exist for your warehouse?** Check the [trusted adapters](https://docs.getdbt.com/docs/trusted-adapters.md) and [community adapters](https://docs.getdbt.com/docs/community-adapters.md) lists. If one exists, find its GitHub repo — the macro SQL and connection logic are almost directly reusable. **2. Did dbt Labs already add a placeholder for your warehouse?** Some warehouses already appear in v2's `AdapterType` enum but aren't fully implemented yet — the enum variant exists, which means the boilerplate is partially in place. When you add the remaining code, the compiler shows you exactly what's still missing. The warehouses in this state are: | Warehouse | v1 adapter | dbt v2 status | | --------- | ----------------------------------------------------------------- | ------------------------------------------------------------------ | | Athena | [dbt-athena](https://github.com/dbt-athena/dbt-athena) (trusted) | `AdapterType::Athena` exists — needs auth, macros, adapter arms | | Trino | [dbt-trino](https://github.com/starburstdata/dbt-trino) (trusted) | `AdapterType::Trino` exists — needs auth, macros, adapter arms | | Starburst | [dbt-trino](https://github.com/starburstdata/dbt-trino) (trusted) | `AdapterType::Starburst` exists — needs auth, macros, adapter arms | | Dremio | [dbt-dremio](https://github.com/dremio/dbt-dremio) (trusted) | `AdapterType::Dremio` exists — needs auth, macros, adapter arms | | Oracle | [dbt-oracle](https://github.com/oracle/dbt-oracle) (trusted) | `AdapterType::Oracle` exists — needs auth, macros, adapter arms | For warehouses not yet in `AdapterType` at all (MySQL, Hive, Vertica, SQL Server, Teradata, etc.), you start from Step 4.1 by adding the `AdapterType` variant. **What transfers from v1 to v2** | v1 component | Where it goes in v2 | Notes | | ----------------------------------------------------------- | --------------------------------------------- | --------------------------------------------------------------------------------------------------- | | `macros/adapters.sql` | `dbt-loader/.../dbt-<wh>/macros/adapters.sql` | Mostly a direct port — same `<wh>__` dispatch prefix, same macro names, same Jinja patterns | | `macros/catalog.sql` | Same location | The catalog SQL (`list_relations_without_caching`, `get_catalog`) transfers almost verbatim | | Custom materializations and adapter-overrides | Same location | Look out for Jinja that might not yet be supported in v2; that will need to be addressed separately | | Profile fields in `credentials.py` / `profile_template.yml` | `DbConfig` struct in `dbt-schemas` | Each profile field becomes a struct field — optional fields use `Option<T>` | | Connection URI / DSN construction in `connections.py` | `dbt-auth/src/<wh>/mod.rs` | The URI building logic maps cleanly to the auth module pattern | | `BaseRelation.quote_policy` / identifier casing behavior | `Policy::new(...)` in `relation_object.rs` | The 3-part vs. 2-part name structure and quote flags map 1:1 to the v2 `Policy` struct | | Catalog introspection SQL in macros and `adapter.py` | `get_relation.rs` and Jinja macros | The system catalog table names and queries you already know transfer directly | **What doesn't transfer by design** A few components won't transfer 1:1 from v1 to v2 — meaning less code and maintenance for you: * `ConnectionManager` methods (`open`, `cancel`, `get_response`, `execute`) — In v1, these were your responsibility to implement against the Python DB API 2.0 spec. In v2, connection management is owned entirely by the ADBC driver. * Python adapter class hierarchy and execution-wrapping methods — now handled by `match adapter_type()` expressions in the shared `adapter_impl.rs` (Step 4.5 explains how these work). * `setup.py` / package dependencies on `dbt-core` — dbt Labs handles packaging and distribution once your PR is merged. No `setup.py`, no PyPI release, and no version pinning required. **How to use the v1 adapter with AI** The Jinja macros are the biggest asset. Feed your LLM your v1 `macros/adapters.sql` alongside the reference `adapters.sql` (in the file breakdown at the bottom of this guide) and ask it to port the macros into the v2 structure. The SQL logic is the same; you're confirming the dialect is right and the system catalog table names match. For profile config and auth, feed your LLM your v1 `credentials.py` or `profile_template.yml` alongside the reference `DbConfig` struct and auth module (see the file breakdown below). Field names and connection parameters usually map directly. The most common mismatch: v1 `connections.py` has a lot of Python connection-management code that has no equivalent in v2 — the driver handles all of that. Focus only on URI/DSN construction and credential fields — skip anything related to cursors, retries, or connection pooling. ##### Dev machine setup ```bash # Rust rustup # install from https://rustup.rs rustup show # verify # Go (needed for driver builds) go version # verify # Clone the repo git clone https://github.com/dbt-labs/dbt cd dbt # Verify you can build cargo build --bin dbt ``` If you hit Z3 errors: ```bash brew install pkg-config z3 ``` If disk fills during build: ```bash cargo clean # frees old build artifacts — you'll do this often ``` ##### Development workflow The core development loop is the same whether you write code manually or with an AI assistant: 1. Add `AdapterType::MyWarehouse` to the enum (Step 4.1) 2. Run `cargo check -p <crate>` — the compiler lists every match arm missing for your new variant 3. Use the compiler output + the file you're editing + the equivalent reference file (from the breakdown below) to write each arm. If using an AI assistant, paste all three in as context. * *Example:* to fill in the `quote_char` arm, paste the compiler error listing `AdapterType::MyWarehouse` as missing, the `quote_char` match block from `dbt-adapter-core/src/lib.rs`, and the Exasol line (`Exasol => '"'`) as the pattern to follow. 4. Fill in the arm; verify with `cargo check` again 5. Repeat for each crate until error-free Every missing case is a compile error, so the AI always has a precise specification to work from. **What context helps for each arm:** * **The file you're editing** — paste the relevant `match` block or function. For example: the `match self.adapter_type()` block from `adapter_impl.rs` that needs a new arm. * **The equivalent reference file** from the breakdown below. For example: when writing your auth module, paste `crates/dbt-auth/src/exasol/mod.rs` as the pattern. * **The compiler error output.** For example: `error[E0004]: non-exhaustive patterns: AdapterType::MyWarehouse not covered` * **Your warehouse's specifics** — system catalog table names and the connection fields from your `profiles.yml`. **Watch out for:** * **Hallucinated file paths** — AI often invents v2 paths. Use the file breakdown below as ground truth. * **Always verify with the type checker** — run `cargo check -p <crate>` after any AI-generated changes. * **SQL macro patterns from v1** may not apply cleanly in dbt v2. Compare against the reference `adapters.sql` in `crates/dbt-loader/src/dbt_macro_assets/dbt-exasol/macros/adapters.sql` in [dbt-labs/dbt-fusion](https://github.com/dbt-labs/dbt-fusion). *** #### Step 3: Understand the architecture Before writing code, it helps to understand the layers you'll be working in. ##### Why ADBC? A key architectural decision in dbt v2 is the use of **ADBC (Arrow Database Connectivity)** as the unified driver interface. In dbt v1, adapters connected via Python drivers — often wrapping `pyodbc` or proprietary connection mechanisms. Each adapter owned its connection logic entirely. In v2, each warehouse connects through an **ADBC driver** — a pre-compiled binary that handles the wire protocol, authentication handshakes, and connection pooling. Your adapter code never touches any of that. For CDN-supported first-party adapters, dbt v2 downloads this driver automatically on first use. For community adapters, users install it manually — which is why Step 2 covers finding or building a driver. ADBC is column-native end-to-end — if your warehouse supports columnar output (e.g. Arrow IPC or Arrow Flight SQL), data flows through with zero conversion. This means **you do not need to write connection management code** — that lives in the driver. What you write is the warehouse-specific configuration, authentication, relation naming, macro logic, and catalog introspection that sits above the driver. ##### The vertical model Unlike v1 where *each warehouse* had its own monolithic adapter class (`SnowflakeAdapter`, `BigqueryAdapter`), v2 organizes adapter logic by **vertical** — feature areas that *span all warehouses*: * **`dbt-xdbc`** — driver loading, ADBC connections (all warehouses) * **`dbt-auth`** — credential resolution (all warehouses, per-warehouse modules) * **`dbt-adapter`** — a single `ConcreteAdapter` with warehouse-specific behavior driven by `match adapter_type()`, not per-warehouse structs * **`dbt-loader`** — Jinja SQL macros per warehouse When you add a new warehouse, you add a variant to each vertical, not a new top-level package. The compiler enforces this: if you add `AdapterType::MyWarehouse` and any `match adapter_type()` block doesn't handle it, Rust won't compile. For example, if `adapter_impl.rs` has a `match self.adapter_type()` that covers Postgres, Snowflake, Exasol, etc. but not `MyWarehouse`, the compiler reports an error at that exact line — so you always know what's left. This is a meaningful improvement over v1's class inheritance model. In v1, a new adapter subclass would silently inherit parent behavior for any method you didn't explicitly override — which could mask bugs where a warehouse quietly used the wrong default. In v2 there's no inheritance and no implicit behavior. Every function that varies by warehouse requires an explicit match arm, so nothing falls through silently. For example: ```text error[E0004]: non-exhaustive patterns: `AdapterType::MyWarehouse` not covered --> crates/dbt-adapter/src/adapter/adapter_impl.rs:142:18 | | match self.adapter_type() { | ^^^^^^^^^^^^^^^^^^^ pattern `AdapterType::MyWarehouse` not covered ``` ##### Layer stack The diagram below shows how a dbt project request flows through the crates at runtime — from configuration to SQL execution. Your adapter work lives in the middle layers: profile config, credential resolution, driver loading, relation logic, and macros. The bottom layer (SQL execution against the warehouse) is owned by dbt Labs. ```text profiles.yml / dbt_project.yml │ ▼ dbt-schemas ← profile config struct for your warehouse │ ▼ dbt-auth ← credential resolution │ ▼ dbt-xdbc ← ADBC driver loading │ ▼ dbt-adapter ← relation types, metadata adapter, adapter methods │ dbt-loader ← Jinja SQL macros (materializations, catalog, utils) │ ▼ [dbt internal] ← SQL execution against warehouse (dbt Labs) ``` ##### Crate map In Rust, a **crate** is a package — the unit of compilation, roughly equivalent to a "library" or "module" in other languages. The `dbt-labs/dbt` monorepo has multiple crates, each responsible for one vertical slice of functionality across all warehouses. This is a quick-reference map of the six crates you'll touch to build your adapter, in the order you'll work through them in Step 4. | Crate | Location | What you touch | | ------------------ | -------------------------- | ------------------------------------------------------------------------------------------ | | `dbt-adapter-core` | `crates/dbt-adapter-core/` | `AdapterType` enum variant, `quote_char` arm | | `dbt-xdbc` | `crates/dbt-xdbc/` | ADBC driver registration (`Backend` enum, library name) | | `dbt-schemas` | `crates/dbt-schemas/` | `profiles.yml` config struct (`DbConfig` variant) | | `dbt-auth` | `crates/dbt-auth/` | Credential resolution, connection URI construction | | `dbt-adapter` | `crates/dbt-adapter/` | Relation quoting, metadata catalog queries, adapter match arms, column builder, sql\_types | | `dbt-loader` | `crates/dbt-loader/` | Jinja SQL macros (`dbt_macro_assets/dbt-<adapter>/`) | *** #### Step 4: Build a new adapter This step walks you through each crate you need to touch. Work through them in order — each builds on the last. After each sub-step, run the type checker to catch missed match arms and type errors: ```bash cargo check -p <crate-name> ``` Replace `<crate-name>` with the crate you just edited — e.g. `dbt-adapter-core`, `dbt-xdbc`, `dbt-schemas`, `dbt-auth`, `dbt-adapter`, or `dbt-loader`. These match the names in the crate map. ##### 4.1 — Register the adapter type **Crate:** `crates/dbt-adapter-core/` `AdapterType` is the central enum that identifies your warehouse throughout the entire codebase. Add your variant here first. Rust's `match` expressions must handle every possible variant — there's no catch-all default case. So when you add `AdapterType::MyWarehouse`, every `match adapter_type()` block in the codebase that doesn't handle your variant becomes a **compile error**. Run `cargo check -p <crate>` and the compiler hands you a complete list of exactly what still needs to be written. ```rust // crates/dbt-adapter-core/src/lib.rs pub enum AdapterType { Postgres, Snowflake, Bigquery, Databricks, Redshift, Salesforce, Spark, DuckDB, Fabric, ClickHouse, Athena, Starburst, Trino, Datafusion, Dremio, Oracle, Exasol, // ← already added MyWarehouse, // ← add yours here } ``` Also add a `quote_char` arm in the same file. Use double quotes `'"'` for most warehouses; BigQuery and Databricks use a backtick instead: ```rust // crates/dbt-adapter-core/src/lib.rs fn quote_char(&self) -> char { match self { // ... existing adapters ... Exasol => '"', // double-quote (most warehouses) MyWarehouse => '"', // ← add yours here BigQuery | Databricks => '`', } } ``` ##### 4.2 — Register the ADBC driver **Crate:** `crates/dbt-xdbc/` This is where v2 learns how to find and load your warehouse's ADBC driver at runtime. | File | What to do | Required? | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `src/driver.rs` | Add a variant to the `Backend` enum; add the ADBC library name (e.g. `"adbc_driver_exasol"`) and FFI protocol; also define a `LoadStrategy` | **Yes** | | `src/install.rs` | Add CDN download URL and platform strings | No — only if the driver will be distributed via the dbt Labs CDN, which requires separate coordination with dbt Labs. Community adapter drivers are not on the CDN by default. | The `Backend` enum maps to the ADBC shared library name (`lib<name>.so` / `<name>.dll` / `lib<name>.dylib`). You're registering its identity so v2 knows what to load — you're not writing the driver here. For **custom Arrow type mappings** (e.g. DuckDB needed this for `HUGEINT`, `UTINYINT`): only add warehouse-specific type handling if your driver returns types that Arrow's standard schema doesn't cover. Most warehouses don't need this. ##### 4.3 — Add your connection profile **Crate:** `crates/dbt-schemas/` | File | What to do | | ------------------------- | ----------------------------------------------------------------------- | | `src/schemas/profiles.rs` | Add a `DbConfig` variant and a config struct with all connection fields | The config struct should include everything a user would put in their `profiles.yml` for your warehouse. Example from Exasol: ```rust #[derive(Debug, Clone, Serialize, Deserialize, Default, DbtSchema, Merge)] pub struct ExasolDbConfig { pub user: Option<String>, #[serde(skip_serializing_if = "Option::is_none", alias = "pass")] pub password: Option<String>, pub host: Option<String>, pub port: Option<StringOrInteger>, pub database: Option<String>, pub schema: Option<String>, pub encryption: Option<bool>, pub certificate_validation: Option<bool>, pub certificate_fingerprint: Option<String>, pub connection_timeout: Option<StringOrInteger>, pub threads: Option<StringOrInteger>, } ``` After adding the config struct, also add `DbConfig::MyWarehouse(Box<MyWarehouseDbConfig>)` as a new variant to the `DbConfig` enum. This registers your new config type so the rest of the codebase knows it exists. Once you do, the compiler will point you at every place that reads from `DbConfig` and needs a new case for your warehouse — follow those errors to wire it in. ##### 4.4 — Add authentication **Crate:** `crates/dbt-auth/` | File | What to do | Required? | | ------------------------- | -------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | | `src/<warehouse>/mod.rs` | Credential resolution — reads config fields, env vars, key files, tokens; builds the ADBC connection URI and credentials | **Yes** | | `src/<warehouse>/init.rs` | Init SQL generation — SQL that must run when the connection opens (e.g. `USE SCHEMA`, `SET` statements, extension loading) | Optional — only for warehouses that need to run SQL on connection open. Most warehouses don't need this. | | `src/lib.rs` | Register the new module with `mod <warehouse>;` and wire it into the auth dispatch match | **Yes** | The auth module turns a `DbConfig` into a live, authenticated ADBC connection. At minimum you need basic credential handling. More sophisticated auth (OAuth, SSO, key-pair) can be added incrementally. The pattern is: read config fields → construct URI → call `builder.with_parse_uri(uri)`, `builder.with_username(user)`, `builder.with_password(password)`. ##### 4.5 — Build the adapter layer **Crate:** `crates/dbt-adapter/` This is the largest step. You're adding warehouse-specific behavior to the shared adapter layer via `match adapter_type()` arms. Simple vs. complex adapters Most adapters — including Exasol, Athena, Trino, Starburst, Dremio, Oracle, and ClickHouse — add match arms directly to the shared files in `src/relation/`. Only adapters with highly custom relation logic (Snowflake's multi-part names and case rules; BigQuery's project/dataset structure) have their own subdirectory under `src/relation/`. If your warehouse uses standard `schema.table` or `database.schema.table` naming with straightforward quoting, you're likely a simple adapter. Start simple and only add complexity if the compiler forces it. ###### Relation type and quoting note The relation crate is currently being verticalized — `Policy` is moving from `relation_object.rs` to `relation_impl.rs`. Verify file paths against the current repo before using them. **File:** `src/relation/relation_object.rs` Add a match arm that constructs a `Relation` with your quoting policy. The policy controls three things: whether the database is included in fully-qualified names, whether the schema is quoted, and whether the identifier is quoted. For example, Exasol uses a 2-part name (`schema.table`, no database prefix) and quotes both: ```rust // src/relation/relation_object.rs Exasol => Box::new(Relation::new_with_policy( Exasol, RelationPath { database: Some(database).filter(|s| !s.is_empty()), schema: Some(schema), identifier, }, relation_type, Policy::new(false, true, true), // ↑ ↑ ↑ // database schema identifier // disabled quoted quoted custom_quoting, None, false, false, )) as Box<dyn BaseRelation>, ``` Decide up front whether your warehouse uses 2-part or 3-part names, and whether identifiers are case-sensitive. For example, Exasol uppercases unquoted identifiers by default — so all catalog lookup SQL uses `upper()` comparisons. **File:** `src/relation/factory.rs` Add your `AdapterType` to the `create_static_relation` match, wiring it to `RelationStatic` (the generic static relation used for Jinja's `api.Relation`): ```rust Databricks | Spark | Fabric | DuckDB | Exasol | Postgres | Redshift | Salesforce | Bigquery => { let relation_type = RelationStatic { adapter_type, quoting }; StaticBaseRelationObject::new(Arc::new(relation_type)) } ``` ###### Catalog introspection **File:** `src/metadata/get_relation.rs` Add a match arm and a function that queries your warehouse's system catalog to look up a single relation by name: ```rust AdapterType::Exasol => exasol_get_relation( adapter, state, ctx, conn, database, schema, identifier, token, ), ``` ```rust fn exasol_get_relation( adapter: &AdapterImpl, state: &State, ctx: &QueryCtx, conn: &mut dyn Connection, database: &str, schema: &str, identifier: &str, token: CancellationToken, ) -> AdapterResult<Option<Box<dyn BaseRelation>>> { let q_schema = schema.to_uppercase(); let q_ident = identifier.to_uppercase(); let sql = format!( "select 'table' as \"type\" from sys.exa_all_tables \ where table_schema = '{q_schema}' and table_name = '{q_ident}' \ union all \ select 'view' from sys.exa_all_views \ where view_schema = '{q_schema}' and view_name = '{q_ident}'" ); // execute, read result, return relation } ``` Use `information_schema` if your warehouse supports standard SQL, or system catalog tables (`sys.*`, `information_schema.*`) as appropriate. ###### Adapter match arms **File:** `src/adapter/adapter_impl.rs` After adding `AdapterType::MyWarehouse`, the compiler will enumerate every `match self.adapter_type()` block that needs a new arm. Most arms simply delegate to an existing pattern. For example: ```rust // Column builder — delegate to postgres-like builder Exasol => Ok(Self::build_postgres_like(field, type_ops)), // Schema column name for listing relations Exasol => "name", // DATA_TYPE column name in information schema AdapterType::Exasol => "DATA_TYPE", // in src/sql_types.rs ``` For capabilities your adapter doesn't support yet (e.g. `valid_incremental_strategies`), return `unimplemented!()` — that's fine for an initial community adapter contribution. The reference PR has several of these. ###### Column builder **File:** `src/column/column_builder.rs` Add a match arm for how your warehouse's Arrow record batches map to dbt column objects. Most adapters can delegate to `build_postgres_like`: ```rust Exasol => Ok(Self::build_postgres_like(field, type_ops)), ``` Only implement custom logic if your warehouse has unusual type handling. ##### 4.6 — Write your SQL macros **Crate:** `crates/dbt-loader/` Create a new directory at `src/dbt_macro_assets/dbt-<adapter_type>/`. Registration is automatic The loader discovers adapter packages by scanning `src/dbt_macro_assets/`. You don't need to add any Rust code to register your directory — creating it and the `dbt_project.yml` is sufficient. | File | What to do | | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `dbt_macro_assets/dbt-<warehouse>/dbt_project.yml` | Adapter plugin project definition: `name: dbt_<warehouse>`, `macro-paths: ["macros"]` | | `macros/adapters.sql` | dbt v1 adapter macros: `create_schema`, `drop_schema`, `drop_relation`, `rename_relation`, `truncate_relation`, `create_table_as`, `create_view_as`, `list_schemas`, `check_schema_exists`, `information_schema_name`, `current_timestamp`, `get_columns_in_relation`, `list_relations_without_caching` | v2 uses the same `adapter.dispatch()` pattern as v1. Your macros use the `<warehouse>__` prefix to override defaults: ```sql {% macro exasol__create_table_as(temporary, relation, sql) -%} create or replace table {{ relation }} as ( {{ sql }} ) {%- endmacro %} {% macro exasol__create_view_as(relation, sql) -%} create or replace view {{ relation }} as ( {{ sql }} ) {%- endmacro %} {% macro exasol__drop_relation(relation) -%} {% call statement('drop_relation', auto_begin=False) -%} drop {{ relation.type }} if exists {{ relation }} cascade {%- endcall %} {% endmacro %} {% macro exasol__rename_relation(from_relation, to_relation) -%} {% call statement('rename_relation') -%} rename {{ from_relation.type }} {{ from_relation }} to {{ to_relation.identifier }} {%- endcall %} {% endmacro %} ``` Note: `rename_relation` uses only `to_relation.identifier`, not the full relation — Exasol's `RENAME` syntax doesn't take a fully qualified target. For catalog introspection, use your warehouse's system catalog. For example, Exasol queries `sys.*` instead of `information_schema`: ```sql {% macro exasol__list_relations_without_caching(schema_relation) -%} {% call statement('list_relations_without_caching', fetch_result=True) -%} select '{{ schema_relation.database }}' as "database", table_name as "name", table_schema as "schema", 'table' as "type" from sys.exa_all_tables where upper(table_schema) = upper('{{ schema_relation.schema }}') union all select '{{ schema_relation.database }}' as "database", view_name as "name", view_schema as "schema", 'view' as "type" from sys.exa_all_views where upper(view_schema) = upper('{{ schema_relation.schema }}') {%- endcall %} {{ return(load_result('list_relations_without_caching').table) }} {%- endmacro %} {% macro exasol__information_schema_name(database) -%} sys {%- endmacro %} ``` If your warehouse is similar to an existing one (e.g. Postgres-compatible), start by delegating to that dialect's macros and only override where behavior differs: ```sql {% macro mywarehouse__create_table_as(temporary, relation, sql) -%} {{ return(postgres__create_table_as(temporary, relation, sql)) }} {%- endmacro %} ``` *** #### Step 5: Test your adapter ##### Type check after each crate Run the type checker after completing work in each crate to catch missed match arms and type errors: ```bash cargo build -p dbt-adapter-core cargo build -p dbt-xdbc cargo build -p dbt-schemas cargo build -p dbt-auth cargo build -p dbt-adapter cargo build -p dbt-loader ``` ##### End-to-end test Run a real `dbt build` against your warehouse. At minimum, exercise table, view, incremental, and snapshot materializations. A clean `dbt build` on `jaffle-shop-classic` is the standard acceptance bar for a community adapter. ```bash # Build the CLI cargo build --bin dbt # Create a test project (or clone jaffle-shop-classic) ./target/debug/dbt init # Run against your warehouse ./target/debug/dbt build --project-dir <your-project> ``` ##### CI testing CI testing for community adapter PRs is coordinated with the dbt Labs adapters team — the test infrastructure is not publicly distributed. When your PR is ready, reach out in `#adapter-ecosystem` and the adapters team will work with you on warehouse validation. Known gap: CI requires coordination Community contributors cannot run CI independently. dbt Labs' CI pipeline requires warehouse credentials for the target warehouse — which means dbt Labs needs access configs for whatever warehouse you're contributing. The process for this handoff is still being defined. Expect to coordinate closely with the adapters team. If you hit friction, flag it in `#adapter-ecosystem`. *** #### Step 6: Document your adapter Once your adapter is merged and available in a release, document it so users can find and configure it. ##### Write a setup guide Document the `profiles.yml` configuration for your warehouse — what fields are required, what's optional, and example values. Follow the format of existing adapter setup guides on [docs.getdbt.com](http://docs.getdbt.com). Driver installation is critical to document Unlike first-party adapters, your users won't get the driver automatically — dbt v2 won't download it for them. Your setup guide must explain where to get the driver binary and how to install it so dbt v2 can find it at runtime. Without this, users will configure a valid profile and still get a connection error. Include the exact library name dbt v2 looks for (e.g. `libadbc_driver_<yourwarehouse>.dylib`) and where to put it. ##### General documentation guidelines * Assume the reader knows dbt fundamentals but is not an expert on your warehouse inner workings. * Include a complete working `profiles.yml` example. * Document any warehouse-specific quirks (e.g. 2-part vs 3-part naming, identifier case sensitivity). * Link to the warehouse vendor's ADBC driver documentation. *** #### Step 7: Promote your adapter Your PR must be merged first dbt Labs reviews and merges community adapter PRs into `dbt-labs/dbt`. Wait until the PR is merged and the adapter ships in a published release before directing users to it. ##### Community channels Join the dbt Community Slack and find: * **`#adapter-ecosystem`** — the main channel for adapter developers * **`#db-<yourwarehouse>`** — if a channel exists for your warehouse, let users know v2 support is available. Note: v1 adapter users will still be on the Python-based adapter and will need to migrate. * **`#proj-isv-adapters-in-fusion`** — ping the dbt Labs adapters team here once your PR is up for review. ##### Before you announce Align with the adapters team on: which materializations you're targeting in the initial implementation, any known gaps in your ADBC driver, and timeline. This prevents surprises during review and sets accurate expectations for users. *** #### Reference: File-by-file implementation guide A community contributed v2 adapter touches roughly 13 files, all in the public [dbt-labs/dbt](https://github.com/dbt-labs/dbt) repo. The "Exasol example" column shows what it looks like in practice — substitute your warehouse name and system catalog throughout. | Generic path | What it does | Exasol example | | ---------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | | `.changes/unreleased/Features-*.yaml` | Changelog entry | Same for all adapters | | `crates/dbt-adapter-core/src/lib.rs` | Add `AdapterType::<Warehouse>` variant and `quote_char` arm | `AdapterType::Exasol` | | `crates/dbt-xdbc/src/driver.rs` | Add `Backend::<Warehouse>` variant — the ADBC library name and FFI protocol | `Backend::Exasol`, library name `"adbc_driver_exasol"` | | `crates/dbt-schemas/src/schemas/profiles.rs` | `<Warehouse>DbConfig` struct + `DbConfig::<Warehouse>` variant, wired into all config match arms | `ExasolDbConfig` struct + `DbConfig::Exasol` variant | | `crates/dbt-auth/src/<warehouse>/mod.rs` | Auth module — reads config fields, resolves credentials, constructs connection URI | `src/exasol/mod.rs` — builds URI from host, port, user/pass, TLS options | | `crates/dbt-adapter/src/adapter/adapter_impl.rs` | All exhaustive `match adapter_type()` arms — most delegate to existing patterns; use `unimplemented!()` for unimplemented features | Exasol adds ~10 arms, most are 1–2 lines | | `crates/dbt-adapter/src/column/column_builder.rs` | Column builder match arm — maps Arrow record batches to dbt column objects | `Exasol => Ok(Self::build_postgres_like(...))` | | `crates/dbt-adapter/src/metadata/get_relation.rs` | Catalog lookup function — queries system tables to find a relation by name | `exasol_get_relation()` — queries `sys.exa_all_tables` / `sys.exa_all_views` | | `crates/dbt-adapter/src/relation/relation_object.rs` | Match arm — quoting policy and whether database prefix is included in fully-qualified names | `Policy::new(false, true, true)` — database disabled, schema + identifier quoted | | `crates/dbt-adapter/src/relation/factory.rs` | Add warehouse to `RelationStatic` arm in `create_static_relation` | Exasol added alongside Postgres, Redshift, DuckDB, etc. | | `crates/dbt-adapter/src/sql_types.rs` | Metadata column name arms — `DATA_TYPE` and schema listing column names | Exasol uses standard `DATA_TYPE` column name | | `crates/dbt-loader/src/dbt_macro_assets/dbt-<warehouse>/dbt_project.yml` | Macro plugin project definition | `dbt-exasol/dbt_project.yml` | | `crates/dbt-loader/src/dbt_macro_assets/dbt-<warehouse>/macros/adapters.sql` | All required adapter macros | `dbt-exasol/macros/adapters.sql` — uses `sys.*` instead of `information_schema` | #### Reference: Useful commands ```bash # Type check a specific crate cargo build -p <crate-name> # Build the CLI binary cargo build --bin dbt # Run a full dbt build against your warehouse ./target/debug/dbt build --project-dir <your-project> # Free disk space cargo clean ``` --- ### Contribute to dbt adapters The dbt Community helps analytics practitioners share their knowledge, help others, and collectively drive forward the discipline of analytics engineering. There are opportunities here for everyone to contribute, whether you're at the beginning of your analytics engineering journey or you are a seasoned data professional. This section explains how you can contribute to existing adapters or create a new adapter. ##### Contribute to a pre-existing adapter Community-supported plugins are works in progress, and you can contribute by testing and writing code. If you're interested in contributing: * Join both the dedicated channel, [#adapter-ecosystem](https://getdbt.slack.com/archives/C030A0UF5LM), in [dbt Slack](https://community.getdbt.com/) and the channel for your adapter's data store. See the **Slack Channel** link in the [dbt v1 platform](https://docs.getdbt.com/docs/local/profiles.yml.md) pages. * Review open issues in the plugin's source repository. Use the relevant **GitHub repo** link in the [dbt v1 platform](https://docs.getdbt.com/docs/local/profiles.yml.md) pages. ##### Create a new v2 adapter v2 is a Rust-based rewrite of the dbt engine built around a single monorepo. Instead of maintaining a separate Python package per warehouse, all adapters live together, organized by feature area rather than warehouse, inside `dbt-labs/dbt`. A bug fix in authentication or relation logic benefits every adapter at once, and your contribution makes the entire ecosystem stronger. ADBC drivers handle connection management—pre-compiled binaries you register, not write. You build the warehouse-specific logic: credentials, relation naming, SQL macros, and catalog queries across roughly 13 files in total. Contributing a v2 adapter means opening a pull request directly against the monorepo. You'll register your warehouse in the `AdapterType` enum, implement credential and relation types in Rust, write Jinja macros for your SQL dialect, and wire up basic integration tests against a real warehouse. The Rust compiler's match exhaustiveness enforcement acts as your to-do list — every unhandled variant is a compile error until your adapter is complete. For a step-by-step walkthrough, see [Contribute a dbt v2 adapter](https://docs.getdbt.com/guides/adapter-creation-v2.md). --- ### Conversion metrics Conversion metrics let you measure how often one event leads to another for a specific entity within a defined time window. For example, you can track how often a user (entity) who visits your site (base event) makes a purchase (conversion event) within 7 days (time window). To set this up, you’ll specify both the time range and the entity that links/joins the two events. Conversion metrics are different from [ratio metrics](https://docs.getdbt.com/docs/build/ratio.md) because you need to include an entity in the pre-aggregated join. #### Parameters The specification for conversion metrics is as follows: (Applies to dbt v1.12 and later) | Parameter | Description | Required | Type | | ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | -------- | -------------- | | `name` | The name of the metric. | Required | String | | `description` | The description of the metric. | Optional | String | | `type` | The type of metric. Set as `conversion` for conversion metrics. | Required | String | | `label` | The display label for the metric. Accepts plain text, spaces, and quotes. | Optional | String | | `config` | Configuration settings for the metric. | Optional | Dict | | `config.group` | The group the metric belongs to. | Optional | String | | `config.tags` | Tags associated with the metric. | Optional | List | | `config.meta` | Metadata for the metric. | Optional | Dict | | `entity` | The entity for each conversion event. | Required | String | | `calculation` | Method of calculation. Either `conversion_rate` or `conversions`. Defaults to `conversion_rate`. | Optional | String | | `base_metric` | The base metric name or configuration for the conversion event. Can be a string (metric name) or a dict (for additional customization). | Required | String or Dict | | `base_metric.name` | The name of the base metric (when using dict format). | Required | String | | `base_metric.filter` | Filter to apply to the base metric (when using dict format). | Optional | String | | `base_metric.alias` | Alias for the base metric (when using dict format). | Optional | String | | `conversion_metric` | The conversion metric name or configuration. Can be a string (metric name) or a dict (for additional customization). | Required | String or Dict | | `conversion_metric.name` | The name of the conversion metric (when using dict format). | Required | String | | `conversion_metric.filter` | Filter to apply to the conversion metric (when using dict format). | Optional | String | | `conversion_metric.alias` | Alias for the conversion metric (when using dict format). | Optional | String | | `window` | The time window for the conversion event (such as `7 days`, `1 week`, `3 months`). Defaults to infinity. | Optional | String | | `constant_properties` | List of properties to hold constant between base and conversion events. Can be a dimension or entity. | Optional | List | | `constant_properties.base_property` | The dimension or entity of the semantic model linked to the `base_metric`. | Required | String | | `constant_properties.conversion_property` | The dimension or entity of the semantic model linked to the `conversion_metric`. | Required | String | Refer to [additional settings](#additional-settings) to learn how to customize conversion metrics with settings for null values, calculation type, and constant properties. The following code example displays the complete specification for conversion metrics and details how they're applied: (Applies to dbt v1.12 and later) models/file\_name.yml ```yaml models: - name: your_model_name semantic_model: enabled: true .... rest of configs.... metrics: - name: my_conversion_metric description: "Tracks how often a base event leads to a conversion event for an entity" label: "My conversion metric" type: conversion entity: my_primary_entity # Required; the entity the conversion is tracked for calculation: conversion_rate # Optional; conversion_rate | conversions base_metric: my_base_event_metric # Required; metrics defined in another semantic model conversion_metric: my_conversion_event_metric # Required; metrics defined in another semantic model window: 7 days # Optional; defines the time window for conversion constant_properties: # Optional; list of constant properties - base_property: my_dimension_or_entity conversion_property: my_dimension_or_entity ``` #### Conversion metric example The following example will measure conversions from website visits (`VISITS` table) to order completions (`BUYS` table) and calculate a conversion metric for this scenario step by step. Suppose you have two semantic models, `VISITS` and `BUYS`: * The `VISITS` table represents visits to an e-commerce site. * The `BUYS` table represents someone completing an order on that site. The underlying tables look like the following: `VISITS`<br />Contains user visits with `USER_ID` and `REFERRER_ID`. | DS | USER\_ID | REFERRER\_ID | | ---------- | -------- | ------------ | | 2020-01-01 | bob | facebook | | 2020-01-04 | bob | google | | 2020-01-07 | bob | amazon | `BUYS`<br />Records completed orders with `USER_ID` and `REFERRER_ID`. | DS | USER\_ID | REFERRER\_ID | | ---------- | -------- | ------------ | | 2020-01-02 | bob | facebook | | 2020-01-07 | bob | amazon | Next, define a conversion metric as follows: (Applies to dbt v1.12 and later) ```yaml models: - name: your_model_name semantic_model: enabled: true .... rest of configs.... metrics: - name: visit_to_buy_conversion_rate_7d description: "Conversion rate from visiting to transaction in 7 days" type: conversion label: Visit to buy conversion rate (7-day window) entity: user calculation: conversion_rate base_metric: name: visits filter: {{ Dimension('visits__referrer_id') }} = 'facebook' conversion_metric: buys window: 7 days ``` To calculate the conversion, link the `BUYS` event to the nearest `VISITS` event (or closest base event). The following steps explain this process in more detail: ##### Step 1: Join `VISITS` and `BUYS` This step joins the `BUYS` table to the `VISITS` table and gets all combinations of visits-buys events that match the join condition where buys occur within 7 days of the visit (any rows that have the same user and a buy happened at most 7 days after the visit). The SQL generated in these steps looks like the following: ```sql select v.ds, v.user_id, v.referrer_id, b.ds, b.uuid, 1 as buys from visits v inner join ( select *, uuid_string() as uuid from buys -- Adds a uuid column to uniquely identify the different rows ) b on v.user_id = b.user_id and v.ds <= b.ds and v.ds > b.ds - interval '7 days' ``` The dataset returns the following (note that there are two potential conversion events for the first visit): | V.DS | V.USER\_ID | V.REFERRER\_ID | B.DS | UUID | BUYS | | ---------- | ---------- | -------------- | ---------- | ----- | ---- | | 2020-01-01 | bob | facebook | 2020-01-02 | uuid1 | 1 | | 2020-01-01 | bob | facebook | 2020-01-07 | uuid2 | 1 | | 2020-01-04 | bob | google | 2020-01-07 | uuid2 | 1 | | 2020-01-07 | bob | amazon | 2020-01-07 | uuid2 | 1 | ##### Step 2: Refine with window function Instead of returning the raw visit values, use window functions to link conversions to the closest base event. You can partition by the conversion source and get the `first_value` ordered by `visit ds`, descending to get the closest base event from the conversion event: ```sql select first_value(v.ds) over (partition by b.ds, b.user_id, b.uuid order by v.ds desc) as v_ds, first_value(v.user_id) over (partition by b.ds, b.user_id, b.uuid order by v.ds desc) as user_id, first_value(v.referrer_id) over (partition by b.ds, b.user_id, b.uuid order by v.ds desc) as referrer_id, b.ds, b.uuid, 1 as buys from visits v inner join ( select *, uuid_string() as uuid from buys ) b on v.user_id = b.user_id and v.ds <= b.ds and v.ds > b.ds - interval '7 day' ``` The dataset returns the following: | V.DS | V.USER\_ID | V.REFERRER\_ID | B.DS | UUID | BUYS | | ---------- | ---------- | -------------- | ---------- | ----- | ---- | | 2020-01-01 | bob | facebook | 2020-01-02 | uuid1 | 1 | | 2020-01-07 | bob | amazon | 2020-01-07 | uuid2 | 1 | | 2020-01-07 | bob | amazon | 2020-01-07 | uuid2 | 1 | | 2020-01-07 | bob | amazon | 2020-01-07 | uuid2 | 1 | This workflow links the two conversions to the correct visit events. Due to the join, you end up with multiple combinations, leading to fanout results. After applying the window function, duplicates appear. To resolve this and eliminate duplicates, use a distinct select. The UUID also helps identify which conversion is unique. The next steps provide more detail on how to do this. ##### Step 3: Remove duplicates Instead of regular select used in the [Step 2](#step-2-refine-with-window-function), use a distinct select to remove the duplicates: ```sql select distinct first_value(v.ds) over (partition by b.ds, b.user_id, b.uuid order by v.ds desc) as v_ds, first_value(v.user_id) over (partition by b.ds, b.user_id, b.uuid order by v.ds desc) as user_id, first_value(v.referrer_id) over (partition by b.ds, b.user_id, b.uuid order by v.ds desc) as referrer_id, b.ds, b.uuid, 1 as buys from visits v inner join ( select *, uuid_string() as uuid from buys ) b on v.user_id = b.user_id and v.ds <= b.ds and v.ds > b.ds - interval '7 day'; ``` The dataset returns the following: | V.DS | V.USER\_ID | V.REFERRER\_ID | B.DS | UUID | BUYS | | ---------- | ---------- | -------------- | ---------- | ----- | ---- | | 2020-01-01 | bob | facebook | 2020-01-02 | uuid1 | 1 | | 2020-01-07 | bob | amazon | 2020-01-07 | uuid2 | 1 | You now have a dataset where every conversion is connected to a visit event. To proceed: 1. Sum up the total conversions in the "conversions" table. 2. Combine this table with the "opportunities" table, matching them based on group keys. 3. Calculate the conversion rate. ##### Step 4: Aggregate and calculate Now that you’ve tied each conversion event to a visit, you can calculate the aggregated conversions and opportunities (Applies to dbt v1.12 and later) simple metric. Then, you can join them to calculate the actual conversion rate. The SQL to calculate the conversion rate is as follows: ```sql select coalesce(subq_3.metric_time__day, subq_13.metric_time__day) as metric_time__day, cast(max(subq_13.buys) as double) / cast(nullif(max(subq_3.visits), 0) as double) as visit_to_buy_conversion_rate_7d from ( -- base select metric_time__day, sum(visits) as visits from ( select date_trunc('day', first_contact_date) as metric_time__day, 1 as visits from visits ) subq_2 group by metric_time__day ) subq_3 full outer join ( -- conversion select metric_time__day, sum(buys) as buys from ( -- ... -- The output of this subquery is the table produced in Step 3. The SQL is hidden for legibility. -- To see the full SQL output, add --explain to your conversion metric query. ) subq_10 group by metric_time__day ) subq_13 on subq_3.metric_time__day = subq_13.metric_time__day group by metric_time__day ``` ##### Additional settings Use the following additional settings to customize your conversion metrics: * **Null conversion values:** Set null conversions to zero using `fill_nulls_with`. Refer to [Fill null values for metrics](https://docs.getdbt.com/docs/build/fill-nulls-advanced.md) for more info. * **Calculation type:** Choose between showing raw conversions or conversion rate. * **Constant property:** Add conditions for specific scenarios to join conversions on constant properties. ##### Set null conversion events to zero To return zero in the final data set, you can set the value of a null conversion event to zero instead of null. You can add the `fill_nulls_with` parameter to your conversion metric definition like this: (Applies to dbt v1.12 and later) ```yaml metrics: - name: visits type: simple agg: count expr: visit_id fill_nulls_with: 0 # set null conversion values to zero in a simple metric - name: buys type: simple agg: count expr: purchase_id fill_nulls_with: 0 - name: visit_to_buy_conversion_rate_7_day_window description: "Conversion rate from viewing a page to making a purchase" type: conversion label: Visit to buy conversion rate (7 day window) entity: user calculation: conversions base_metric: visits conversion_metric: buys window: 7 days ``` This will return the following results: [![Conversion metric with fill nulls with parameter](/img/docs/dbt-platform/semantic-layer/conversion-metrics-fill-null.png?v=2 "Conversion metric with fill nulls with parameter")](#)Conversion metric with fill nulls with parameter Refer to [Fill null values for metrics](https://docs.getdbt.com/docs/build/fill-nulls-advanced.md) for more info. ##### Set calculation type parameter Use the conversion calculation parameter to either show the raw number of conversions or the conversion rate. The default value is the conversion rate. You can change the default to display the number of conversions by setting the `calculation: conversion` parameter: (Applies to dbt v1.12 and later) ```yaml metrics: - name: visit_to_buy_conversions_1_week_window description: "Visit to buy conversions" type: conversion label: Visit to buy conversions (1 week window) entity: user calculation: conversions base_metric: visits conversion_metric: buys window: 1 week fill_nulls_with: 0 ``` ##### Set constant property *Refer to [Amplitude's blog posts on constant properties](https://amplitude.com/blog/holding-constant) to learn about this concept.* You can add a constant property to a conversion metric to count only those conversions where a specific dimension or entity matches in both the base and conversion events. For example, if you're at an e-commerce company and want to answer the following question: * *How often did visitors convert from `View Item Details` to `Complete Purchase` with the same product in each step?* <br /> * This question is tricky to answer because users could have completed these two conversion milestones across many products. For example, they may have viewed a pair of shoes, then a T-shirt, and eventually checked out with a bow tie. This would still count as a conversion, even though the conversion event only happened for the bow tie. Back to the initial questions, you want to see how many customers viewed an item detail page and then completed a purchase for the *same* product. In this case, you want to set `product_id` as the constant property. You can specify this in the configs as follows: (Applies to dbt v1.12 and later) ```yaml metrics: - name: view_item_detail_to_purchase_with_same_item description: "Conversion rate for users who viewed the item detail page and purchased the item" type: conversion label: View item detail > Purchase entity: user calculation: conversions base_metric: view_item_detail conversion_metric: purchase window: 1 week constant_properties: - base_property: product conversion_property: product ``` You will add an additional condition to the join to make sure the constant property is the same across conversions. ```sql select distinct first_value(v.ds) over (partition by buy_source.ds, buy_source.user_id, buy_source.session_id order by v.ds desc rows between unbounded preceding and unbounded following) as ds, first_value(v.user_id) over (partition by buy_source.ds, buy_source.user_id, buy_source.session_id order by v.ds desc rows between unbounded preceding and unbounded following) as user_id, first_value(v.referrer_id) over (partition by buy_source.ds, buy_source.user_id, buy_source.session_id order by v.ds desc rows between unbounded preceding and unbounded following) as referrer_id, buy_source.uuid, 1 as buys from {{ source_schema }}.fct_view_item_details v inner join ( select *, {{ generate_random_uuid() }} as uuid from {{ source_schema }}.fct_purchases ) buy_source on v.user_id = buy_source.user_id and v.ds <= buy_source.ds and v.ds > buy_source.ds - interval '7 day' and buy_source.product_id = v.product_id --Joining on the constant property product_id ``` #### Related docs * [Fill null values for metrics](https://docs.getdbt.com/docs/build/fill-nulls-advanced.md) --- ### Cost Insights dbt platform | Enterprise, Enterprise+ Cost Insights shows estimated costs and compute time for your dbt projects and models directly in the dbt platform, so you can measure and share the impact of optimizations like [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) and [state-aware orchestration](https://docs.getdbt.com/docs/deploy/state-aware-about.md). State-aware orchestration is now dbt State [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) works with all engines and environments: dbt v1, dbt platform, and dbt v2 If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) and [state-aware orchestration](https://docs.getdbt.com/docs/deploy/state-aware-about.md) make your dbt workflows more efficient by reusing models and tests instead of running full rebuilds. When either is enabled, Cost Insights helps you demonstrate the resulting cost reductions and efficiency gains. These cost and cost reduction estimates are based on a retroactive analysis of runs after you enable dbt State or state-aware orchestration. They reflect actual historical usage, *not* forecasts of future costs or cost reductions. With Cost Insights, you can see: * **How much your dbt models cost to run**: See the compute cost and times for each model and job in your warehouse's native units. * **The cost reductions from using dbt State or state-aware orchestration**: Understand the cost reduction when dbt State or state-aware orchestration reuses unchanged models. * **Cost trends over time**: Track your warehouse spend and optimization impact across your dbt projects. * **Asset type filtering**: On Cost Insights charts (**Cost**, **Usage**, **Query run time**, **Builds**), use the **Assets** dropdown menu to filter data by **Models**, **Tests**, or **All**. Each tab keeps its own selection. * **Per-job cost breakdown**: In the Cost Insights table view, use the **All** and **Jobs** buttons to switch between an aggregated view and a per-job cost breakdown. The Cost Insights section is available in different dbt platform areas and lets you view your cost data and the impact of dbt State and state-aware orchestration optimizations across various dimensions: * [Project dashboard](https://docs.getdbt.com/docs/explore/explore-cost-data.md#project-dashboard) * [Catalog on Model page](https://docs.getdbt.com/docs/explore/explore-cost-data.md#model-performance-in-catalog) * [Job details page](https://docs.getdbt.com/docs/explore/explore-cost-data.md#job-details) [![Cost Insights in the project dashboard](/img/docs/dbt-platform/cost-insights/cost-insights-project.png?v=2 "Cost Insights in the project dashboard")](#)Cost Insights in the project dashboard [![Cost Insights in Catalog](/img/docs/dbt-platform/cost-insights/cost-insights-model.png?v=2 "Cost Insights in Catalog")](#)Cost Insights in Catalog [![Cost Insights in job details](/img/docs/dbt-platform/cost-insights/cost-insights-job.png?v=2 "Cost Insights in job details")](#)Cost Insights in job details #### Prerequisities To view cost data, ensure you have: * One of the roles listed in [Assign required permissions](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md#assign-required-permissions). * A supported data warehouse: * Snowflake * BigQuery * Databricks * Amazon Redshift [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") For setup instructions, see [Set up Cost Insights](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md). #### Understanding cost and reduction estimates note Cost estimates are intended for visibility and optimization, not billing reconciliation. dbt calculates the cost of running your dbt models using your data warehouse's usage metadata and billing context. dbt computes costs daily using up to the *last seven days of available data*. ##### Warehouse-specific logic The following sections explain how costs are calculated for each supported warehouse. Expand each section to view the details.  Snowflake dbt computes Snowflake query costs using Snowflake's query attribution data and your credit price (`price_per_credit`). dbt pulls the `price_per_credit` value directly from Snowflake when available; otherwise, dbt uses the configured or default value in the dbt platform. For more information about configuring or viewing these values, see [Configure Cost Insights settings](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md#configure-cost-insights-settings-optional). Formula: ```text credits_per_query * price_per_credit ``` Where: * `credits_per_query` — Cloud services, compute, and query acceleration credits attributed to the query. * For standard and Generation 2 warehouses, dbt sources this value from [`QUERY_ATTRIBUTION_HISTORY`](https://docs.snowflake.com/en/sql-reference/account-usage/query_attribution_history). * For [Adaptive Warehouses](https://docs.snowflake.com/en/user-guide/warehouses-adaptive), dbt sources this value from [`QUERY_METERING_HISTORY`](https://docs.snowflake.com/en/sql-reference/account-usage/query_metering_history) — refer to [Configure platform metadata credentials](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md#snowflake) for required permissions. * `price_per_credit` — Your Snowflake credit price (from Snowflake system tables when available, otherwise from your configured input or the default rate). Snowflake attribution limitation Snowflake doesn't attribute cost to queries that run in roughly 100 milliseconds or less. As a result, Cost Insights totals may be *lower* than the compute spend shown in your Snowflake billing dashboards. For more information, see the [Snowflake documentation](https://docs.snowflake.com/en/sql-reference/account-usage/query_attribution_history#usage-notes).  BigQuery BigQuery does not expose per-query cost directly in system tables. Instead, dbt estimates cost by combining *query usage* with a *pricing input* (either from your configuration or the default rate). * **On-demand pricing** The cost is determined by how much data each query processes. The usage shows the amount of that data billed for the query. Formula: ```text data_processed_per_query * price_per_tib ``` Where: * `data_processed_per_query` - Total data billed for the query (normalized to TiB). dbt sources this value from `information_schema.jobs.total_bytes_billed`. For more information, see the [BigQuery documentation](https://docs.cloud.google.com/bigquery/docs/information-schema-jobs). * `price_per_tib` - BigQuery on-demand price per TiB (from your configuration or the default rate). * **Capacity pricing (reservations)** The cost is determined by how long each query runs on reserved compute. The usage shows the amount of that reserved compute time consumed by a query. Formula: ```text compute_time_per_query * price_per_slot_hour ``` Where: * `compute_time_per_query` - Total slot time used by the query (in hours). dbt sources this value from `information_schema.jobs.total_slot_ms`. For more information, see the [BigQuery documentation](https://docs.cloud.google.com/bigquery/docs/information-schema-jobs). * `price_per_slot_hour` - BigQuery capacity price per slot-hour (from your configuration or the default rate) * **Cached queries** Queries served from cache do not consume compute and are counted as $0.  Databricks Databricks does not directly attribute usage to individual queries. Instead, dbt estimates per-query cost by proportionally allocating Databricks Units (DBUs) based on how long each query ran during a billing period. * Queries that run longer receive a larger share of usage. * Usage is converted to dollars using your list price. Formula: ```text usage_per_query * cost_per_dbu ``` Where: * `usage_per_query` - DBUs attributed to the query. * `cost_per_dbu` - Dollar cost per DBU for the relevant stock-keeping unit. For information about the pricing system table, see the [Databricks documentation](https://docs.databricks.com/aws/en/admin/system-tables/pricing). Databricks reports usage in billing windows. These windows are periods of time where a compute resource consumed a known number of DBUs. Queries have their own start and end times. For information about the billing usage system table, see the [Databricks documentation](https://docs.databricks.com/aws/en/admin/system-tables/billing). To attribute usage to queries: 1. dbt identifies which billing windows each query overlaps. 2. dbt calculates how long the query ran during each window. 3. dbt allocates DBUs *proportionally* based on the query’s share of total execution time on that compute resource during the same window. Conceptually: ```text DBUs_in_window * (query_runtime / total_query_runtime_in_window) ``` dbt sums this across all overlapping windows to get `usage_per_query`.  Amazon Redshift[Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Cost Insights supports both Amazon Redshift Serverless and provisioned cluster deployments. dbt detects your deployment type automatically when testing the connection. note On Redshift, dbt attributes query costs using the comments it automatically injects into each query. If another process removes or replaces these comments, dbt can't tie the query back to its model, and its cost won't be attributed. Make sure nothing in your Redshift environment removes or replaces dbt's query comments. * **Redshift Serverless** dbt estimates cost by identifying which Redshift Processing Unit (RPU) billing periods overlap with each query's execution time and attributing the proportional share of RPU-hours to the query. Formula: ```text rpu_hours_per_query * rpu_price_per_hour ``` Where: * `rpu_hours_per_query` - RPU-hours attributed to the query based on its proportional overlap with each billing period. dbt sources billing period data from `SYS_SERVERLESS_USAGE`. For more information, see the [Amazon Redshift documentation](https://docs.aws.amazon.com/redshift/latest/mgmt/serverless-billing.html). * `rpu_price_per_hour` - Your RPU price per hour (from your configured value in [Cost Insights settings](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md#configure-cost-insights-settings-optional)). * **Redshift Provisioned** dbt estimates cost based on how long each query ran and how many nodes your cluster has. Formula: ```text elapsed_time_hours * node_count * node_price_per_hour ``` Where: * `elapsed_time_hours` - Query execution time in hours. dbt sources this from `SYS_QUERY_HISTORY`. * `node_count` - Number of nodes in your cluster. dbt sources this from `STV_SLICES`. * `node_price_per_hour` - Your price per node per hour (from your configured value in [Cost Insights settings](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md#configure-cost-insights-settings-optional)). **Additional considerations:** * **Data retention**: Redshift system tables (`SYS_QUERY_HISTORY`, `SYS_SERVERLESS_USAGE`) retain only seven days of history. Cost data for Redshift may cover a shorter window than other warehouses. dbt calculates costs for whatever data is available within that window. * **Pricing required**: There are no default price values for Redshift. Costs will appear as $0 until you configure `rpu_price_per_hour` (serverless) or `node_price_per_hour` (provisioned) in [Cost Insights settings](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md#configure-cost-insights-settings-optional). Concurrent query costs Redshift attributes cost to each query as if it ran alone on the cluster. If queries run concurrently, the sum of individual query costs may exceed your actual bill. ##### Cost reduction calculation dbt calculates cost reductions by comparing actual costs to what costs would have been *without model reuse*. To do this, dbt uses data from the last seven days (where available) and performs the following steps: 1. Calculates the average cost per model build. 2. Counts how many times a model was reused instead of rebuilt. 3. Multiplies the reused model count by the average cost per build to determine total cost reduction. Formula: ```text average_cost_per_build * reuse_count ``` dbt calculates reductions per model and per deployment environment (production and staging), based on recent historical runs. Additional notes: * dbt calculates estimated costs and savings daily. * Pricing inputs come from warehouse system tables (where available), connection-level configuration, or default list prices. ###### Example The following example shows how dbt calculates cost reductions. Looking back seven days, assuming a model runs on two distinct days: | Day | Total cost | Total executions | | --------- | ---------- | ---------------- | | Day 1 | $5 | 5 | | Day 2 | $10 | 10 | | **Total** | **$15** | **15** | <br /> The average cost per execution: $15 ÷ 15 runs = $1 per run If the model was *reused* eight times instead of rebuilt during this same period, the estimated cost reduction is: $1 average cost per run \* 8 reuses = $8 #### Considerations Keep the following in mind when using Cost Insights: **Data collection and refresh** * Cost Insights uses your platform metadata credentials to access warehouse system tables. No separate credentials are needed beyond the platform metadata setup. * You need sufficient [permissions](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md#configure-platform-metadata-credentials) to query warehouse metadata tables. * Cost data is calculated *once per day* by a scheduled job that runs at approximately 17:00 UTC. * The data collection job processes completed calendar days only. It does not include the current day because warehouse usage data may still be incomplete. * Jobs that ran yesterday (or earlier) will have cost data available after the next daily refresh. * Jobs that ran today will not have cost data until the following day’s refresh, regardless of what time they ran. * If you don’t see cost data for a recent job, make sure at least one full calendar day has passed since it ran. The **Updated** badge in the **Cost Insights** section shows when the last refresh occurred. **Cost accuracy** * dbt calculates costs using warehouse-reported usage data and applies default credit or compute costs based on standard warehouse pricing. * If you have custom pricing agreements with your warehouse provider, override the default values in your account settings to ensure accurate cost reporting. For more information, see [Set up Cost Insights](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md#configure-cost-insights-settings-optional). * Update your cost variables whenever your warehouse pricing contracts change to maintain accurate tracking. * Changes to cost variables only apply to future calculations — historical cost data remains unchanged. **Optimization data** * Optimization and usage reduction data is available once dbt State or state-aware orchestration is enabled and begins reusing models across runs. * For accounts already using dbt State or state-aware orchestration, run at least one full model build within the last 10 days before enabling Cost Insights to establish a baseline for cost reduction calculations. If you don't see cost reduction data, run a full build to establish the baseline. * Cost Insights currently calculates estimated reductions in warehouse compute usage at the model level and will expand to include tests and seeds in the future. **Exporting data** * You can export cost data as a CSV file for further analysis and reporting. For more information, see [Explore cost data](https://docs.getdbt.com/docs/explore/explore-cost-data.md). #### Related FAQs Why might my actual warehouse costs differ from displayed costs? Cost Insights shows estimates based on warehouse-reported usage and your configured pricing variables. These estimates are based on a retroactive analysis of historical runs and reflect actual usage, *not* forecasts of future costs. Adjustments and differences may occur if: * Your warehouse has custom pricing that differs from the default compute credit unit. * There are discounts or credits applied at the billing level that aren't reflected in usage tables. * Costs include other charges beyond compute. Costs Insights in the dbt platform is designed to be directionally accurate, showing you dbt-specific components rather than matching your billing exactly. How often is cost data refreshed? Cost data refreshes daily and reflects the previous day's usage. This means there is a lag of up to one day between when a job runs and when its cost data appears in Cost Insights. How do I troubleshoot if cost data isn't appearing? If cost data isn't appearing in Cost Insights, check the following: * Verify that platform metadata credentials are configured in your account settings and that the credential test is passing. For more information, see [Set up Cost Insights](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md#configure-platform-metadata-credentials). * Ensure you have one of the required permissions to view cost data. For more information, see [Assign required permissions](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md#assign-required-permissions). * Confirm that at least one job is running in a production environment. Cost data only appears after jobs have executed. * Cost data refreshes daily and reflects the previous day's usage, which means there is a lag of up to one day between when a job runs and when its cost data appears. If you just ran a job, wait until the next day to see the data. * After enabling Cost Insights, dbt looks back 10 days to build baselines for cost reduction calculations. If you don't see cost reduction data, ensure you have sufficient job history within the last 10 days. Does the Cost Insights feature incur warehouse costs? dbt issues lightweight, read-only queries against your warehouse to retrieve metadata and to power features such as Cost Insights. dbt scopes and filters these queries to minimize impact, and most customers see negligible costs (typically on the order of cents). How does increasing job frequency affect cost reduction estimates? Cost reduction metrics reflect how dbt optimizes compute costs by reusing existing results instead of running the same model again. When you increase your job run frequency (for example, because performance improvements make it easier to schedule jobs more often), dbt has more opportunities to reuse models. As reuse increases, dbt optimizes more compute, which means your reported cost reductions may also increase. This metric shows the efficiency impact of reuse within your current workload. It reflects the compute costs that dbt reduces by reusing models instead of rebuilding them, rather than showing your total warehouse spend reduction. What happened to state-aware orchestration? On June 1, 2026, dbt Labs and Fivetran announced **[dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md)**[Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") as a new and improved version of state-aware orchestration. A key feature is [`lag_tolerance`](https://docs.getdbt.com/reference/resource-configs/lag-tolerance.md), which controls how much time must pass since the last upstream data change before a node is eligible for a rebuild. dbt State improves upon state-aware orchestration in a few key ways: * **Works everywhere** — dbt State works with dbt v1, dbt v2, and dbt platform, as well as external orchestrators, across both development and deployment environments. * **Smarter data freshness tracking** — dbt State tracks data freshness across the DAG and automatically propagates it through models materialized as views. Unlike state-aware orchestration's `build_after` config which compares against the model's last successful execution, dbt State's `lag_tolerance` compares against the freshness of the underlying data. * **Advanced change detection** — dbt State can detect and ignore file modifications that don't change actual transformation logic, such as adding a comment or cleaning up whitespace. If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. For details on billing after the trial ends, refer to [dbt State usage and pricing](https://docs.getdbt.com/docs/platform/billing.md#dbt-state-usage). While dbt State is in preview, there is no required migration timeline — dbt Labs will communicate a timeline when dbt State reaches general availability. To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). --- ### Creating metrics After building [semantic models](https://docs.getdbt.com/docs/build/semantic-models.md), it's time to start adding metrics. This page explains the different supported metric types you can add to your dbt project. (Applies to dbt v1.12 and later) You can define metrics in two locations within your `models` YAML files: * **Within semantic models** (under `models` -> `semantic_model` -> `metrics`) for metrics that use dimensions from a single semantic model. Recommended for simple metrics. * **At the top level** (under a separate `metrics` key) for advanced metrics (cumulative, ratio, derived, conversion) that reference metrics from different semantic models. Simple metrics can't be defined at the top level. Define metrics in YAML files within your dbt project, and not in a `config` block on a model. #### Parameters The keys for metrics parameters are: (Applies to dbt v1.12 and later) | Parameter | Description | Required | Type | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------ | | `name` | Provide the reference name for the metric. This name must be a unique metric name and can consist of lowercase letters, numbers, and underscores. | Required | String | | `description` | Describe your metric. | Optional | String | | `type` | Define the type of metric, which can be `conversion`, `cumulative`, `derived`, `ratio`, or `simple`. | Required | String | | `label` | Defines the display value in downstream tools. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | String | | `config` | Use the [`config`](https://docs.getdbt.com/reference/resource-properties/config.md) property to specify configurations for your metric. Supports [`meta`](https://docs.getdbt.com/reference/resource-configs/meta.md), [`group`](https://docs.getdbt.com/reference/resource-configs/group.md), [`tags`](https://docs.getdbt.com/reference/resource-configs/tags.md), and [`enabled`](https://docs.getdbt.com/reference/resource-configs/enabled.md) configurations. | Optional | Dict | | `filter` | You can optionally add a [filter](#filters) string to any metric type, applying filters to dimensions, entities, time dimensions, or other metrics during metric computation. Consider it as your WHERE clause. | Optional | String | ##### Type-specific parameters Each metric type has additional specific parameters: * **Simple metrics**: `agg` (required), `expr`, `percentile`, `percentile_type`, `non_additive_dimension`, `agg_time_dimension`, `join_to_timespine`, `fill_nulls_with` * **Cumulative metrics**: `input_metric` (required), `window`, `grain_to_date`, `period_agg` * **Derived metrics**: `expr` (required), `input_metrics` (required) * **Ratio metrics**: `numerator` (required), `denominator` (required) * **Conversion metrics**: `entity` (required), `calculation` (required), `base_metric` (required), `conversion_metric` (required), `window`, `constant_properties` Refer to the following sections about each metric type for detailed information on type-specific parameters. ##### Example Here's a complete example of the metrics spec configuration: (Applies to dbt v1.12 and later) models/file\_name.yml ```yaml models: - name: my_model semantic_model: enabled: true name: my_semantic_model # Define columns, dimensions, entities... columns: - name: my_column # column config... # Metrics defined within the semantic model metrics: - name: my_simple_metric description: A metric description type: simple label: My Simple Metric agg: sum expr: revenue_column # Optional, defaults to metric name config: meta: owner: "@analytics_team" # Advanced metrics that reference metrics from different semantic models metrics: - name: my_advanced_cumulative_metric type: cumulative window: 1 week input_metric: name: my_metric_name_from_another_semantic_model filter: my_filter alias: my_metric_name_a_week_ago_in_another_semantic_model ``` 📹 Learn about the dbt Semantic Layer with on-demand video courses! Explore our [dbt Semantic Layer on-demand course](https://learn.getdbt.com/courses/semantic-layer) to learn how to define and query metrics in your dbt project. Additionally, dive into mini-courses for querying the dbt Semantic Layer in your favorite tools: [Tableau](https://courses.getdbt.com/courses/tableau-querying-the-semantic-layer), [Excel](https://learn.getdbt.com/courses/querying-the-semantic-layer-with-excel), [Hex](https://courses.getdbt.com/courses/hex-querying-the-semantic-layer), and [Mode](https://courses.getdbt.com/courses/mode-querying-the-semantic-layer). #### Default granularity for metrics (Applies to dbt v1.12 and later) Set the native grain on the time dimension column and choose the rollup grain at query time. * Define the data’s grain on the time dimension column with granularity (for example, hour, day, month). * Metrics inherit the model’s default aggregation time dimension (set at the model level), and you can optionally override it per metric with `agg_time_dimension`. * When querying, pick the rollup grain (day/week/month/year) in your BI tool or with the [Semantic Layer API](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md). You can only roll up to grains coarser than or equal to the column’s native grain. ##### Example * `order_time` is stored at hourly grain. * The `orders` metric uses that time dimension; you roll it up to month at query time (not in the metric spec). * A second metric shows how to point at a different time dimension using `agg_time_dimension`. models/file\_name.yml ```yaml models: - name: orders semantic_model: enabled: true # Model-level default aggregation time dimension (optional) agg_time_dimension: order_time columns: - name: order_time granularity: hour # Native grain of the column dimension: type: time - name: created_at granularity: day dimension: type: time metrics: - name: orders type: simple agg: count expr: 1 # No time_granularity here. Roll up to month at query time. - name: orders_by_created_day type: simple agg: count expr: 1 agg_time_dimension: created_at # Use an alternate time dimension for this metric ``` #### Conversion metrics [Conversion metrics](https://docs.getdbt.com/docs/build/conversion.md) help you track when a base event and a subsequent conversion event occur for an entity within a set time period. (Applies to dbt v1.12 and later) models/file\_name.yml ```yaml models: - name: my_model semantic_model: enabled: true # Define columns, dimensions, entities... metrics: - name: my_conversion_metric description: Conversion metric description type: conversion label: My conversion metric entity: my_primary_entity # Required calculation: conversion_rate # conversion_rate | conversions base_metric: simple_metric # Required, must refer to metric name or metric dict conversion_metric: my_simple_metric_that_uses_the_other_time_dimension # Required, must refer to metric name or metric dict window: 1 week # Optional time window constant_properties: # Optional - base_property: dimension_or_entity_name # Dimension or entity from base metric's semantic model conversion_property: dimension_or_entity_name # Dimension or entity from conversion metric's semantic model # For conversion metrics referencing metrics from different semantic models metrics: - name: my_advanced_conversion_metric config: group: example_group tags: - example_tag - another_tag meta: owner: "@docs-team" type: conversion entity: my_primary_entity # required for conversion metrics calculation: conversion_rate # conversion_rate | conversions # required for conversion metrics base_metric: my_simple_metric # str | metric dict (if filters are needed) conversion_metric: my_other_metric_from_another_semantic_model # str | metric dict (if filters are needed) window: 1 week # optional, as above constant_properties: # optional - DIMENSION OR ENTITY - base_property: my_primary_entity # Required. A reference to a dimension/entity of the semantic model linked to the base_metric conversion_property: another_semantic_model_categorical_dimension ``` #### Cumulative metrics (Applies to dbt v1.12 and later) [Cumulative metrics](https://docs.getdbt.com/docs/build/cumulative.md) aggregate a simple metric over a given period. If no period is specified, the window will accumulate the simple metric over all of the recorded time period. Note that you will need to create the [time spine model](https://docs.getdbt.com/docs/build/metricflow-time-spine.md) before you add cumulative metrics. models/file\_name.yml ```yaml models: - name: my_model semantic_model: enabled: true # Define columns, dimensions, entities... metrics: - name: active_users type: simple agg: count_distinct expr: user_id - name: wau_rolling_7 type: cumulative label: Weekly active users description: Rolling 7 day window of active users input_metric: active_users # Required - must refer to metric name or dict window: 7 days # Optional - omit for infinite window (all time) join_to_timespine: true # Optional fill_nulls_with: 0 # Optional # For cumulative metrics using metrics from different semantic models metrics: - name: my_advanced_cumulative_metric type: cumulative label: Advanced cumulative metric input_metric: name: metric_from_another_model filter: "{{ Dimension('entity__dimension') }} > 0" alias: filtered_metric window: 1 week # Optional: can specify window OR grain_to_date, not both # grain_to_date: month # Alternative to window # period_agg: first # Optional: first | last | average ``` #### Derived metrics [Derived metrics](https://docs.getdbt.com/docs/build/derived.md) allow you to perform calculations using other metrics. For example, you can calculate `gross_profit` by subtracting a `cost` metric from a `revenue` metric, or calculate growth by comparing a metric to its value from a previous time period. (Applies to dbt v1.12 and later) models/file\_name.yml ```yaml models: - name: my_model semantic_model: enabled: true # Define columns, dimensions, entities... metrics: - name: order_total type: simple agg: sum expr: order_amount - name: order_cost type: simple agg: sum expr: cost_amount - name: order_gross_profit description: Gross profit from each order type: derived label: Order gross profit expr: revenue - cost # Required for derived metrics input_metrics: # Required for derived metrics - name: order_total alias: revenue - name: order_cost alias: cost # For derived metrics using metrics from different semantic models metrics: - name: my_advanced_derived_metric type: derived label: Advanced derived metric expr: metric_a - metric_b_offset + metric_c # Required input_metrics: # Required - list of metric names or dicts - name: metric_a filter: "{{ Dimension('entity__dimension') }} > 10" - name: metric_b alias: metric_b_offset offset_window: 1 week # Only allowed for derived metrics - name: metric_c offset_to_grain: month # Only allowed for derived metrics ``` #### Ratio metrics [Ratio metrics](https://docs.getdbt.com/docs/build/ratio.md) involve a numerator metric and a denominator metric. A `filter` string can be applied to both the numerator and denominator or separately to the numerator or denominator. (Applies to dbt v1.12 and later) models/file\_name.yml ```yaml models: - name: my_model semantic_model: enabled: true # Define columns, dimensions, entities... metrics: - name: cancellations type: simple agg: sum - name: transaction_amount description: The amount of transactions label: Transaction Amount type: simple agg: sum - name: cancellation_rate type: ratio label: Cancellation rate numerator: cancellations # Can be string or dict denominator: transaction_amount # Can be string or dict filter: | # Optional - applies to both numerator and denominator "{{ Dimension('customer__country') }} = 'MX'" - name: enterprise_cancellation_rate type: ratio label: Enterprise cancellation rate numerator: name: cancellations filter: "{{ Dimension('company__tier') }} = 'enterprise'" # Optional alias: enterprise_cancellations # Optional denominator: transaction_amount filter: | # Optional - applies to both {{ Dimension('customer__country') }} = 'MX' # For ratio metrics using metrics from different semantic models metrics: - name: my_advanced_ratio_metric type: ratio label: Advanced ratio metric numerator: name: metric_from_model_a filter: "{{ Dimension('entity__dimension') }} > 10" alias: filtered_numerator denominator: name: metric_from_model_b filter: "{{ Dimension('entity__dimension') }} < 100" ``` #### Simple metrics (Applies to dbt v1.12 and later) [Simple metrics](https://docs.getdbt.com/docs/build/simple.md) point directly to a single column expression within a semantic model. You can think of a simple metric as the foundational building block for other metrics. It performs an aggregation (like `sum`, `count`, or `average`, and so on) on a single field in your model. models/file\_name.yml ```yaml models: - name: my_model semantic_model: enabled: true # Define columns, dimensions, entities... metrics: - name: cancellations description: The number of cancellations type: simple label: Cancellations agg: sum # Required: sum | max | min | average | median | count_distinct | percentile | count | sum_boolean expr: cancellations_usd # Optional, defaults to metric name join_to_timespine: true # Optional fill_nulls_with: 0 # Optional filter: | # Optional {{ Dimension('order__value') }} > 100 and "{{ Dimension('user__acquisition') }} is not null # Example with percentile aggregation - name: order_percentile type: simple label: Order 95th Percentile agg: percentile expr: order_value percentile: 95.0 # Required if agg is percentile percentile_type: discrete # Required: discrete | continuous # Example with non-additive dimension - name: account_balance type: simple label: Account balance agg: sum expr: balance_amount non_additive_dimension: name: snapshot_date # Must be a dimension window_agg: max # min | max group_by: # Optional - list of entity names - account_id # Example with config - name: revenue type: simple label: Total revenue agg: sum expr: revenue_amount config: meta: owner: "@finance_team" team: "Revenue analytics" ``` #### Filters Configure a filter using Jinja templating and the following syntax to reference entities, dimensions, time dimensions, or metrics in filters. Refer to [Metrics as dimensions](https://docs.getdbt.com/docs/build/ref-metrics-in-filters.md) for details on how to use metrics as dimensions with metric filters: models/metrics/file\_name.yml ```yaml filter: | {{ Entity('entity_name') }} filter: | {{ Dimension('primary_entity__dimension_name') }} filter: | {{ TimeDimension('time_dimension', 'granularity') }} filter: | {{ Metric('metric_name', group_by=['entity_name']) }} ``` For example, if you want to filter for the order date dimension grouped by month, use the following syntax: ```yaml filter: | {{ TimeDimension('order_date', 'month') }} ``` #### Related docs * [Semantic models](https://docs.getdbt.com/docs/build/semantic-models.md) * [Fill null values for metrics](https://docs.getdbt.com/docs/build/fill-nulls-advanced.md) * [Metrics as dimensions with metric filters](https://docs.getdbt.com/docs/build/ref-metrics-in-filters.md) --- ### Cross-platform Mesh using Iceberg catalogs Beta ### Cross-platform Mesh using Iceberg catalogs [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") If a model is configured with `catalog_name`, dbt uses the current project's [catalog definition](https://docs.getdbt.com/docs/build/iceberg/about-catalogs.md) (in `catalogs.yml`), for the current active adapter, to resolve the top-level namespace of that model. This means that you can materialize a dbt model in your `databricks_project` to an Iceberg table in Unity catalog, and select from it in another model in your `snowflake_project`. You can even materialize that model back to Unity catalog, and then use it for other models in `databricks_project`. The requirement is an automated service for linking or syncing metadata across your Iceberg catalog and connected data platforms. These include [Snowflake catalog-linked databases](https://docs.snowflake.com/en/sql-reference/sql/create-database-catalog-linked), [Databricks catalog federation](https://docs.databricks.com/aws/en/query-federation/catalog-federation), [AWS Glue catalog federation](https://docs.aws.amazon.com/lake-formation/latest/dg/catalog-federation.html), and [GCP Lakehouse runtime catalog](https://docs.cloud.google.com/lakehouse/docs/about-lakehouse-catalogs). Supported cross-platform combinations include: * Snowflake ↔ Databricks using Unity catalog ([Snowflake docs](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration-rest-unity), [tutorial](https://docs.snowflake.com/en/user-guide/tutorials/tables-iceberg-set-up-bidirectional-access-to-unity-catalog)) * Snowflake ↔ Athena using Glue catalog ([Snowflake docs](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration-rest-glue)) * Snowflake ↔ BigQuery using BigLake catalog ([Snowflake docs](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration-rest-biglake)) * Snowflake ↔ DuckDB using Horizon catalog ([example](https://github.com/dataders/dbt_aws_cloud_cost)) * Databricks ↔ DuckDB using Unity catalog ([example](https://github.com/dataders/dbt_aws_cloud_cost)) #### Example: Snowflake ↔ Databricks using Unity catalog Let's imagine two "mesh" projects, [`jaffle_finance`](https://github.com/dbt-labs/jaffle-shop-mesh-finance) and [`jaffle_marketing`](https://github.com/dbt-labs/jaffle-shop-mesh-marketing), with a cross-project dependency `jaffle_finance -> jaffle_marketing`. So long as the `jaffle_finance` project writes its public models to an Iceberg catalog that the `jaffle_marketing` project can read from, and both projects configure that Iceberg catalog in `catalogs.yml` by the same `name` — these projects can now run on **different data platforms.** ```yaml # jaffle_finance/dbt_project.yml flags: use_catalogs_v2: true models: jaffle_finance: marts: access: public catalog_name: finance_db ``` ```yaml # jaffle_finance/catalogs.yml catalogs: - name: finance_db type: unity config: databricks: # where jaffle_finance project will write its public models to catalog_database: finance_db ``` ```yaml # jaffle_marketing/dbt_project.yml flags: use_catalogs_v2: true ``` ```yaml # jaffle_marketing/catalogs.yml catalogs: - name: finance_db type: unity config: snowflake: # catalog-linked database pointing to same Unity catalog # where jaffle_marketing project will read jaffle_finance public models from catalog_database: snowflake_cld__finance_db ``` note If you create the [Snowflake catalog-linked database](https://docs.snowflake.com/en/sql-reference/sql/create-database-catalog-linked) with `CATALOG_CASE_SENSITIVITY = CASE_INSENSITIVE`, then you don't need to quote all column and identifier names that Snowflake is reading from or writing to Unity. What's going on here? * The `jaffle_finance` project, running on Databricks, materializes "mart" models into the `finance_db` database in Unity catalog. * This is actually an Iceberg catalog, which we've made accessible to Snowflake through a catalog-linked database. In Snowflake, the catalog-linked database is named `snowflake_cld__finance_db`. * When running the `jaffle_marketing` project, dbt sees a cross-project reference to a public model in the `jaffle_finance` project: ```sql -- marketing/models/marts/roi_by_channel.sql with monthly_revenue as ( select * from {{ ref('jaffle_finance', 'monthly_revenue') }} ), ... ``` ![Flow chart: The marketing project's ROI by channel model depends on the finance project's monthly revenue model](https://github.com/user-attachments/assets/e0a07911-b9d5-4422-9a81-6287e1946ca2) This works for [both ways of resolving cross-project references](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md): 1. `package` dependencies (supported in dbt v1 + dbt platform) 2. `project` dependencies (dbt platform Enterprise; refer to the [advantages of this approach](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#advantages)) **For `package` dependencies:** ```yaml # dependencies.yml packages: - git: https://github.com/dbt-labs/jaffle-shop-mesh-finance ``` The upstream package model `jaffle_finance.monthly_revenue` is configured with `catalog_name: finance_db`. dbt uses the currently running (root) project's `catalogs.yml` to resolve its three-part relation name. In this project (`jaffle_marketing`) + this platform (Snowflake), that catalog is configured with `database: snowflake_cld__finance_db`. Therefore, dbt resolves the reference to: **For `project` dependencies:** ```yaml # dependencies.yml projects: - name: jaffle_finance ``` Previously, if we only used the upstream model's `database` config, then dbt would resolve this to `finance_db.jaffle_finance.monthly_revenue` — which doesn't exist in Snowflake. Instead, dbt now **matches up the `catalog_name`** for the referenced model with the entries in `catalogs.yml` for the currently running (root) project. dbt sees that `jaffle_finance.monthly_revenue` has `catalog_name: finance_db`, which for this project (`jaffle_marketing`) + this platform (Snowflake) is configured with `database: snowflake_cld__finance_db`. Therefore, dbt resolves the reference to: ```sql -- marketing/target/compiled/models/marts/roi_by_channel.sql with monthly_revenue as ( select * from snowflake_cld__finance_db."jaffle_finance"."monthly_revenue" ), ... ``` *And it just works!* Behind the scenes, Snowflake is syncing Databricks' `finance_db` ↔ Snowflake's `snowflake_cld__finance_db`, with eventual consistency. This means that `finance_db.jaffle_finance.monthly_revenue` in Databricks and `snowflake_cld__finance_db."jaffle_finance"."monthly_revenue"` in Snowflake are pointers to *the exact same Iceberg table* in the Databricks-managed Unity catalog. ##### Alternative approach: catalog federation Writes to external Iceberg catalogs are generally slower than writes to managed Iceberg tables, and they can also run into reliability issues at scale. (See our [blog](https://docs.getdbt.com/blog/catalog-linked-databases) and [benchmark](https://github.com/dbt-labs/snow-dbx-iceberg-benchmark).) Instead of having Snowflake write directly to Unity catalog through a catalog-linked database, you can have Snowflake write to its (managed) Horizon catalog, and use [Databricks catalog federation](https://docs.databricks.com/aws/en/query-federation/catalog-federation) to synchronize the Iceberg metadata for subsequent reads. [![Multi-catalog pattern for Databricks + Snowflake cross-platform Mesh](/img/docs/building-a-dbt-project/snow-dbx-xplat-mesh.png?v=2 "Multi-catalog pattern for Databricks + Snowflake cross-platform Mesh")](#)Multi-catalog pattern for Databricks + Snowflake cross-platform Mesh catalogs.yml ```yaml catalogs: - name: finance_unity type: unity config: databricks: # catalog where jaffle_finance project will write its public models to catalog_database: finance_db snowflake: # name of catalog-linked database # where jaffle_marketing will read jaffle_finance's public models catalog_database: snowflake_cld__finance_db - name: marketing_horizon type: horizon config: snowflake: # database where jaffle_marketing will write its public models to catalog_database: horizon_db databricks: # name of federated catalog in Unity # where jaffle_finance will read jaffle_marketing's public models catalog_database: dbx_federation__horizon_db ``` --- ### Cumulative metrics (Applies to dbt v1.12 and later) Cumulative metrics aggregate values from other metrics across a defined accumulation period. If you don’t specify a period, the metric accumulates values over the entire available time range. Use cumulative metrics when you want to calculate rolling or period-to-date values, such as weekly active users or month-to-date revenue. * You must create a [time spine model](https://docs.getdbt.com/docs/build/metricflow-time-spine.md) before you define cumulative metrics so that MetricFlow can join time-based aggregations to the time spine. * If a cumulative metric depends on metrics or dimensions defined in a different semantic model, set cumulative metrics under the top level `metrics` key. #### Parameters (Applies to dbt v1.12 and later) | Parameter | Description | Required | Type | | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------- | | `name` | The name of the metric. | Required | String | | `description` | The description of the metric. | Optional | String | | `type` | The type of the metric (cumulative, derived, ratio, or simple). | Required | String | | `label` | Optional string that defines the display value in downstream tools. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | String | | `input_metric` | The name of the metric being referenced. Supports the following nested parameters: `name`, `filter`, and `alias`. | Required | Dict | | `input_metric.name` | The name of the metric being referenced. | Required | String | | `input_metric.filter` | The [filter](https://docs.getdbt.com/docs/build/metrics-overview.md#filters) to apply to the metric. | Optional | String | | `input_metric.alias` | The alias to apply to the metric. | Optional | String | | `join_to_timespine` | Boolean indicating if the aggregated metric should be joined to the time spine table to fill in missing dates. Default is `false`. | Optional | Boolean | | `window` | Specifies the accumulation window, such as `1 month`, `7 days`, or `1 year`. Cannot be used with `grain_to_date`. | Optional | String | | `grain_to_date` | Sets the accumulation grain, such as `hour`, `day`, `week`, `month`, `year`, restarting accumulation at the beginning of each specified grain period. For example, selecting `month` will aggregate the `month to date` aggregation of the metric. Cannot be used with `window`. | Optional | String | | `period_agg` | Defines how to re-aggregate the cumulative metric when querying with a non-default granularity: `first`, `last`, or `average`. Defaults to `first` if `period_agg` isn't specified. | Optional | String | ##### Complete specification The following displays the complete specification for cumulative metrics, along with an example: (Applies to dbt v1.12 and later) ```yaml metrics: - name: my_advanced_cumulative_metric # required description: my_description # optional label: my_label # optional type: cumulative # required -- cumulative | ratio | derived | conversion # if cumulative, optionally can supply window or grain_to_date, but not both window: 1 week # (interval) # grain_to_date: hour # hour | day | week | month | year | ... input_metric: # required, must refer to metric name or metric dict name: my_metric_name_from_another_semantic_model filter: "{{ Dimension('entity__dimension_name') }} > 10" alias: my_metric_name_a_week_ago_in_another_semantic_model ``` #### Cumulative metrics example Cumulative metrics measure data over a given window and consider the window infinite when no window parameter is passed, accumulating the data over all time. The following example shows how to define cumulative metrics in a YAML file: (Applies to dbt v1.12 and later) * `cumulative_order_total`: Calculates the cumulative order total over all time. Uses `input_metric` to specify the simple metric `order_total` to be aggregated. * `cumulative_order_total_l1m`: Calculates the trailing 1-month cumulative order total. Uses `window` parameter to specify a window of 1 month. * `cumulative_order_total_mtd`: Calculates the month-to-date cumulative order total. Uses `grain_to_date` parameter to specify a grain of `month`. models/marts/fct\_orders.yml ```yaml metrics: - name: cumulative_order_total label: "Cumulative order total (All-Time)" description: "The cumulative value of all orders" type: cumulative input_metric: order_total - name: cumulative_order_total_l1m label: "Cumulative order total (L1M)" description: "Trailing 1-month cumulative order total" type: cumulative window: 1 month input_metric: order_total - name: cumulative_order_total_mtd label: "Cumulative order total (MTD)" description: "The month-to-date value of all orders" type: cumulative grain_to_date: month input_metric: order_total ``` ##### Granularity options (Applies to dbt v1.12 and later) Use the `period_agg` parameter with `first`, `last`, and `average` functions to aggregate cumulative metrics over the requested period. This is because granularity options for cumulative metrics are different than the options for other metric types. * For other metrics, we use the `date_trunc` function to implement granularity. * However, cumulative metrics are non-additive (values can't be added up), so we can't use the `date_trunc` function to change their time grain granularity. * By default, we take the first value of the period. You can change this by specifying a different function using the `period_agg` parameter. In the following example, we define a cumulative metric, `cumulative_revenue`, that calculates the cumulative revenue for all orders: models/marts/fct\_orders.yml ```yaml metrics: - name: cumulative_revenue description: "The cumulative revenue for all orders." label: "Cumulative revenue (all-time)" type: cumulative input_metric: revenue period_agg: first # Optional. Defaults to first. Accepted values: first|last|average ``` In this example, `period_agg` is set to `first`, which chooses the first value for the selected granularity window. To query `cumulative_revenue` by week, use the following query syntax: * `dbt sl query --metrics cumulative_revenue --group-by metric_time__week`  Expand toggle to view how the SQL compiles Note the use of the `window` function to select the `first` value. For `last` and `average`, we would replace the `first_value()` function in the generated SQL with `last_value()` and `average` respectively. ```sql -- re-aggregate metric via the group by select metric_time__week, metric_time__quarter, revenue_all_time from ( -- window function for metric re-aggregation select metric_time__week, metric_time__quarter, first_value(revenue_all_time) over ( partition by metric_time__week, metric_time__quarter order by metric_time__day rows between unbounded preceding and unbounded following ) as revenue_all_time from ( -- join self over time range -- pass only elements: ['txn_revenue', 'metric_time__week', 'metric_time__quarter', 'metric_time__day'] -- aggregate measures -- compute metrics via expressions select subq_11.metric_time__day as metric_time__day, subq_11.metric_time__week as metric_time__week, subq_11.metric_time__quarter as metric_time__quarter, sum(revenue_src_28000.revenue) as revenue_all_time from ( -- time spine select ds as metric_time__day, date_trunc('week', ds) as metric_time__week, date_trunc('quarter', ds) as metric_time__quarter from mf_time_spine subq_12 group by ds, date_trunc('week', ds), date_trunc('quarter', ds) ) subq_11 inner join fct_revenue revenue_src_28000 on ( date_trunc('day', revenue_src_28000.created_at) <= subq_11.metric_time__day ) group by subq_11.metric_time__day, subq_11.metric_time__week, subq_11.metric_time__quarter ) subq_16 ) subq_17 group by metric_time__week, metric_time__quarter, revenue_all_time ``` ##### Window options This section details examples of when to specify and not to specify window options. (Applies to dbt v1.12 and later) * When a period is specified, MetricFlow applies a sliding window to the underlying simple metric, such as tracking weekly active users with a 7-day window. * Without specifying a period, cumulative metrics accumulate values over all time, useful for running totals like current revenue and active subscriptions.  Example of window specified If a window option is specified, MetricFlow applies a sliding window to the underlying metric. Suppose the underlying metrics, `customers`, is configured to count the unique customers making orders at the Jaffle shop: models/marts/customers.yml ```yaml models: - name: customers semantic_model: enabled: true ... metrics: - name: customers description: "Unique customers making orders at the Jaffle shop" label: "Customers" type: simple agg: count_distinct expr: customer_id ``` We can write a cumulative metric `weekly_customers` as such: models/marts/customers.yml ```yaml metrics: - name: weekly_customers description: "Weekly active customers with a 7-day sliding window" label: "Weekly Active Customers" type: cumulative input_metric: customers window: 7 days period_agg: first # When using non-default granularity with cumulative metrics, re-aggregation is required. period_agg: first selects the first value in each granularity window during re-aggregation. ``` From the sample YAML example, note the following: * `type`: Specify cumulative to indicate the type of metric. * `input_metric`: Specify the metric to be aggregated (in this case, `customers`). * `window`: Specify the accumulation window (in this case, 7 days). * `period_agg`: Specify the re-aggregation function (in this case, `first`). * `grain_to_date`: Specify the grain to date (in this case, `week`). For example, in the `weekly_customers` cumulative metric, MetricFlow takes a sliding 7-day window of relevant customers and applies a count distinct function. If you remove `window`, the metric will accumulate over all time.  Example of window not specified Suppose you (a subscription-based company for the sake of this example) have an event-based log table with the following columns: * `date`: a date column * `user_id`: (integer) an ID specified for each user that is responsible for the event * `subscription_plan`: (integer) a column that indicates a particular subscription plan associated with the user. * `subscription_revenue`: (integer) a column that indicates the value associated with the subscription plan. * `event_type`: (integer) a column that populates with +1 to indicate an added subscription, or -1 to indicate a deleted subscription. * `revenue`: (integer) a column that multiplies `event_type` and `subscription_revenue` to depict the amount of revenue added or lost for a specific date. Using cumulative metrics without specifying a window, you can calculate running totals for metrics like the count of active subscriptions and revenue at any point in time. The following YAML file shows creating a cumulative metrics to obtain current revenue and the total number of active subscriptions as a cumulative sum using the `revenue` and `subscription_count` metrics: models/marts/fct\_orders.yml ```yaml metrics: - name: current_revenue description: "Current revenue" label: "Current Revenue" type: cumulative input_metric: revenue - name: active_subscriptions description: "Count of active subscriptions" label: "Active Subscriptions" type: cumulative input_metric: subscription_count ``` ##### Grain to date You can choose to specify a grain to date in your cumulative metric configuration to accumulate a metric from the start of a grain (such as week, month, or year). When using a window, such as a month, MetricFlow will go back one full calendar month. However, grain to date will always start accumulating from the beginning of the grain, regardless of the latest date of data. (Applies to dbt v1.12 and later) For example, let's consider an underlying simple metric `order_total` defined within a semantic model: ```yaml models: - name: fct_orders semantic_model: enabled: true # Simple metrics defined within semantic model metrics: - name: order_total description: "Sum of order total amounts" type: simple agg: sum expr: order_total ``` We can compare the difference between a 1-month window and a monthly grain to date. * The cumulative metric in a window approach applies a sliding window of 1 month * The grain to date by month resets at the beginning of each month. (Applies to dbt v1.12 and later) models/marts/fct\_orders.yml ```yaml # Top-level metrics key for cumulative metrics metrics: - name: cumulative_order_total_mtd label: "Cumulative order total (MTD)" description: "The month-to-date value of all orders" type: cumulative grain_to_date: month input_metric: order_total ``` Cumulative metric with grain to date: (Applies to dbt v1.12 and later) models/marts/fct\_orders.yml ```yaml # Top-level metrics key for cumulative metrics metrics: - name: orders_last_month_to_date label: "Orders month to date" description: "Month-to-date count of orders" type: cumulative grain_to_date: month input_metric: order_count ```  Expand toggle to view how the SQL compiles ```sql with staging as ( select subq_3.date_day as metric_time__day, date_trunc('week', subq_3.date_day) as metric_time__week, sum(subq_1.order_count) as orders_last_month_to_date from dbt_jstein.metricflow_time_spine subq_3 inner join ( select date_trunc('day', ordered_at) as metric_time__day, 1 as order_count from analytics.dbt_jstein.orders orders_src_10000 ) subq_1 on ( subq_1.metric_time__day <= subq_3.date_day ) and ( subq_1.metric_time__day >= date_trunc('month', subq_3.date_day) ) group by subq_3.date_day, date_trunc('week', subq_3.date_day) ) select * from ( select metric_time__week, first_value(orders_last_month_to_date) over (partition by date_trunc('week', metric_time__day) order by metric_time__day) as cumulative_revenue from staging ) group by metric_time__week, cumulative_revenue order by metric_time__week 1 ``` #### SQL implementation example To calculate the cumulative value of the metric over a given window we do a time range join to a timespine table using the primary time dimension as the join key. We use the accumulation window in the join to decide whether a record should be included on a particular day. The following SQL code produced from an example cumulative metric is provided for reference: To implement cumulative metrics, refer to the SQL code example: ```sql select count(distinct distinct_users) as weekly_active_users, metric_time from ( select subq_3.distinct_users as distinct_users, subq_3.metric_time as metric_time from ( select subq_2.distinct_users as distinct_users, subq_1.metric_time as metric_time from ( select metric_time from transform_prod_schema.mf_time_spine subq_1356 where ( metric_time >= cast('2000-01-01' as timestamp) ) and ( metric_time <= cast('2040-12-31' as timestamp) ) ) subq_1 inner join ( select distinct_users as distinct_users, date_trunc('day', ds) as metric_time from demo_schema.transactions transactions_src_426 where ( (date_trunc('day', ds)) >= cast('1999-12-26' as timestamp) ) AND ( (date_trunc('day', ds)) <= cast('2040-12-31' as timestamp) ) ) subq_2 on ( subq_2.metric_time <= subq_1.metric_time ) and ( subq_2.metric_time > dateadd(day, -7, subq_1.metric_time) ) ) subq_3 ) group by metric_time, limit 100; ``` #### Limitations If you specify a `window` in your cumulative metric definition, you must include `metric_time` as a dimension in the SQL query. This is because the accumulation window is based on metric time. For example, ```sql select count(distinct subq_3.distinct_users) as weekly_active_users, subq_3.metric_time from ( select subq_2.distinct_users as distinct_users, subq_1.metric_time as metric_time group by subq_3.metric_time ``` #### Related docs * [Fill null values for simple, derived, or ratio metrics](https://docs.getdbt.com/docs/build/fill-nulls-advanced.md) --- ### Custom aliases #### Overview When dbt runs a model, it will generally create a relation (either a table or a view ) in the database, except in the case of an [ephemeral model](https://docs.getdbt.com/docs/build/materializations.md), when it will create a CTE for use in another model. By default, dbt uses the model's filename as the identifier for the relation or CTE it creates. This identifier can be overridden using the [`alias`](https://docs.getdbt.com/reference/resource-configs/alias.md) model configuration. ##### Why alias model names? The names of schemas and tables are effectively the "user interface" of your data warehouse. Well-named schemas and tables can help provide clarity and direction for consumers of this data. In combination with [custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md), model aliasing is a powerful mechanism for designing your warehouse. The file naming scheme that you use to organize your models may also interfere with your data platform's requirements for identifiers. For example, you might wish to namespace your files using a period (`.`), but your data platform's SQL dialect may interpret periods to indicate a separation between schema names and table names in identifiers, or it may forbid periods from being used at all in CTE identifiers. In cases like these, model aliasing can allow you to retain flexibility in the way you name your model files without violating your data platform's identifier requirements. ##### Usage The `alias` config can be used to change the name of a model's identifier in the database. The following table shows examples of database identifiers for models both with and without a supplied `alias`, and with different materializations. | Model | Config | Relation Type | Database Identifier | | ---------------- | ----------------------------------------------------------- | ------------- | -------------------------------- | | ga\_sessions.sql | {{ config(materialization='view') }} | view | "analytics"."ga\_sessions" | | ga\_sessions.sql | {{ config(materialization='view', alias='sessions') }} | view | "analytics"."sessions" | | ga\_sessions.sql | {{ config(materialization='ephemeral') }} | CTE | "\_\_dbt\_\_cte\_\_ga\_sessions" | | ga\_sessions.sql | {{ config(materialization='ephemeral', alias='sessions') }} | CTE | "\_\_dbt\_\_cte\_\_sessions" | To configure an alias for a model, supply a value for the model's `alias` configuration parameter. For example: models/google\_analytics/ga\_sessions.sql ```sql -- This model will be created in the database with the identifier `sessions` -- Note that in this example, `alias` is used along with a custom schema {{ config(alias='sessions', schema='google_analytics') }} select * from ... ``` Or in a `schema.yml` file. models/google\_analytics/schema.yml ```yaml models: - name: ga_sessions config: alias: sessions ``` When referencing the `ga_sessions` model above from a different model, use the `ref()` function with the model's *filename* as usual. For example: models/combined\_sessions.sql ```sql -- Use the model's filename in ref's, regardless of any aliasing configs select * from {{ ref('ga_sessions') }} union all select * from {{ ref('snowplow_sessions') }} ``` ##### generate\_alias\_name The alias generated for a model is controlled by a macro called `generate_alias_name`. This macro can be overridden in a dbt project to change how dbt aliases models. This macro works similarly to the [generate\_schema\_name](https://docs.getdbt.com/docs/build/custom-schemas.md#advanced-custom-schema-configuration) macro. To override dbt's alias name generation, create a macro named `generate_alias_name` in your own dbt project. The `generate_alias_name` macro accepts two arguments: 1. The custom alias supplied in the model config 2. The node that a custom alias is being generated for The default implementation of `generate_alias_name` simply uses the supplied `alias` config (if present) as the model alias, otherwise falling back to the model name. This implementation looks like this: get\_custom\_alias.sql ```jinja2 {% macro generate_alias_name(custom_alias_name=none, node=none) -%} {%- if custom_alias_name -%} {{ custom_alias_name | trim }} {%- elif node.version -%} {{ return(node.name ~ "_v" ~ (node.version | replace(".", "_"))) }} {%- else -%} {{ node.name }} {%- endif -%} {%- endmacro %} ``` 💡 Use Jinja's whitespace control to tidy your macros! When you're modifying macros in your project, you might notice extra white space in your code in the `target/compiled` folder. You can remove unwanted spaces and lines with Jinja's [whitespace control](https://docs.getdbt.com/faqs/Jinja/jinja-whitespace.md) by using a minus sign. For example, use `{{- ... -}}` or `{%- ... %}` around your macro definitions (such as `{%- macro generate_schema_name(...) -%} ... {%- endmacro -%}`). (Applies to dbt v1.12 and later) ##### generate\_latest\_version\_pointer\_alias When the [`latest_version_pointer`](https://docs.getdbt.com/reference/resource-configs/latest_version_pointer.md) config is enabled, dbt uses the `generate_latest_version_pointer_alias` macro to determine the name of the pointer view it creates for the latest version of a versioned model. This macro follows the same pattern as [`generate_alias_name`](#generate_alias_name). The default implementation uses the model's base name (for example, `dim_customers`) unless a custom alias is set using `latest_version_pointer.alias`: macros/generate\_latest\_version\_pointer\_alias.sql ```jinja2 {% macro generate_latest_version_pointer_alias(custom_alias_name=none, node=none) -%} {{ return(adapter.dispatch('generate_latest_version_pointer_alias', 'dbt')(custom_alias_name, node)) }} {%- endmacro %} {% macro default__generate_latest_version_pointer_alias(custom_alias_name=none, node=none) -%} {%- if custom_alias_name -%} {{ custom_alias_name | trim }} {%- else -%} {{ node.name }} {%- endif -%} {%- endmacro %} ``` To override the default, create a macro named `generate_latest_version_pointer_alias` in your project. For example, to use a `_latest` suffix instead of the base name: macros/get\_latest\_version\_pointer\_alias.sql ```jinja2 {% macro generate_latest_version_pointer_alias(custom_alias_name=none, node=none) -%} {%- if custom_alias_name -%} {{ custom_alias_name | trim }} {%- else -%} {{ node.name ~ "_latest" }} {%- endif -%} {%- endmacro %} ``` ##### Dispatch macro - SQL alias management for databases and dbt packages See docs on macro `dispatch`: ["Managing different global overrides across packages"](https://docs.getdbt.com/reference/dbt-jinja-functions/dispatch.md#managing-different-global-overrides-across-packages) ##### Caveats ###### Ambiguous database identifiers Using aliases, it's possible to accidentally create models with ambiguous identifiers. Given the following two models, dbt would attempt to create two views with *exactly* the same names in the database (ie. `sessions`): models/snowplow\_sessions.sql ```sql {{ config(alias='sessions') }} select * from ... ``` models/sessions.sql ```sql select * from ... ``` Whichever one of these models runs second would "win", and generally, the output of dbt would not be what you would expect. To avoid this failure mode, dbt will check if your model names and aliases are ambiguous in nature. If they are, you will be presented with an error message like this: ```text $ dbt compile Encountered an error: Compilation Error dbt found two resources with the database representation "analytics.sessions". dbt cannot create two resources with identical database representations. To fix this, change the "schema" or "alias" configuration of one of these resources: - model.my_project.snowplow_sessions (models/snowplow_sessions.sql) - model.my_project.sessions (models/sessions.sql) ``` If these models should indeed have the same database identifier, you can work around this error by configuring a [custom schema](https://docs.getdbt.com/docs/build/custom-schemas.md) for one of the models. ###### Model versions **Related documentation:** * [Model versions](https://docs.getdbt.com/docs/mesh/govern/model-versions.md) * [`versions`](https://docs.getdbt.com/reference/resource-properties/versions.md#alias) By default, dbt will create versioned models with the alias `<model_name>_v<v>`, where `<v>` is that version's unique identifier. You can customize this behavior just like for non-versioned models by configuring a custom `alias` or re-implementing the `generate_alias_name` macro. #### Related docs * [Customize dbt models database, schema, and alias](https://docs.getdbt.com/guides/customize-schema-alias.md?step=1) to learn how to customize dbt models database, schema, and alias * [Custom schema](https://docs.getdbt.com/docs/build/custom-schemas.md) to learn how to customize dbt schema * [Custom database](https://docs.getdbt.com/docs/build/custom-databases.md) to learn how to customize dbt database --- ### Custom databases A word on naming Different warehouses have different names for *logical databases*. The information in this document covers "databases" on Snowflake, Redshift, and Postgres; "projects" on BigQuery; and "catalogs" on Databricks Unity Catalog. The values `project` and `database` are interchangeable in BigQuery project configurations. #### Configuring custom databases The logical database that dbt models are built into can be configured using the `database` model configuration. If this configuration is not supplied to a model, then dbt will use the database configured in the active target from your `profiles.yml` file. If the `database` configuration *is* supplied for a model, then dbt will build the model into the configured database. The `database` configuration can be supplied for groups of models in the `dbt_project.yml` file, or for individual models in model SQL files. ##### Configuring database overrides in `dbt_project.yml`: This config changes all models in the `jaffle_shop` project to be built into a database called `jaffle_shop`. dbt\_project.yml ```yaml name: jaffle_shop models: jaffle_shop: +database: jaffle_shop # For BigQuery users: # project: jaffle_shop ``` ##### Configuring database overrides in a model file This config changes a specific model to be built into a database called `jaffle_shop`. models/my\_model.sql ```sql {{ config(database="jaffle_shop") }} select * from ... ``` ##### generate\_database\_name The database name generated for a model is controlled by a macro called `generate_database_name`. This macro can be overridden in a dbt project to change how dbt generates model database names. This macro works similarly to the [generate\_schema\_name](https://docs.getdbt.com/docs/build/custom-schemas.md#advanced-custom-schema-configuration) macro. To override dbt's database name generation, create a macro named `generate_database_name` in your own dbt project. The `generate_database_name` macro accepts two arguments: 1. The custom database supplied in the model config 2. The node that a custom database is being generated for The default implementation of `generate_database_name` simply uses the supplied `database` config if one is present, otherwise the database configured in the active `target` is used. This implementation looks like this: get\_custom\_database.sql ```jinja2 {% macro generate_database_name(custom_database_name=none, node=none) -%} {%- set default_database = target.database -%} {%- if custom_database_name is none -%} {{ default_database }} {%- else -%} {{ custom_database_name | trim }} {%- endif -%} {%- endmacro %} ``` 💡 Use Jinja's whitespace control to tidy your macros! When you're modifying macros in your project, you might notice extra white space in your code in the `target/compiled` folder. You can remove unwanted spaces and lines with Jinja's [whitespace control](https://docs.getdbt.com/faqs/Jinja/jinja-whitespace.md) by using a minus sign. For example, use `{{- ... -}}` or `{%- ... %}` around your macro definitions (such as `{%- macro generate_schema_name(...) -%} ... {%- endmacro -%}`). ##### Managing different behaviors across packages See docs on macro `dispatch`: ["Managing different global overrides across packages"](https://docs.getdbt.com/reference/dbt-jinja-functions/dispatch.md) #### Considerations ##### BigQuery When dbt opens a BigQuery connection, it will do so using the `project_id` defined in your active `profiles.yml` target. This `project_id` will be billed for the queries that are executed in the dbt run, even if some models are configured to be built in other projects. #### Related docs * [Customize dbt models database, schema, and alias](https://docs.getdbt.com/guides/customize-schema-alias.md?step=1) to learn how to customize dbt models database, schema, and alias * [Custom schema](https://docs.getdbt.com/docs/build/custom-schemas.md) to learn how to customize dbt model schema * [Custom aliases](https://docs.getdbt.com/docs/build/custom-aliases.md) to learn how to customize dbt model alias name --- ### Custom schemas By default, all dbt models are built in the schema specified in your [environment](https://docs.getdbt.com/docs/dbt-platform-environments.md) (dbt platform) or [profile's target](https://docs.getdbt.com/docs/local/dbt-environments.md) (dbt v1). This default schema is called your *target schema*. For projects with many models, it's common to organize them across multiple schemas. For example, you might want to: * Group models based on the business unit using the model, creating schemas such as `core`, `marketing`, `finance` and `support`. * Hide intermediate models in a `staging` schema, and only present models that should be queried by an end user in an `analytics` schema. To do this, specify a custom schema. dbt generates the schema name for a model by appending the custom schema to the target schema. For example, `<target_schema>_<custom_schema>`. | Target schema | Custom schema | Resulting schema | | ------------------------ | ------------- | ----------------------------------- | | analytics\_prod | None | analytics\_prod | | alice\_dev | None | alice\_dev | | dbt\_cloud\_pr\_123\_456 | None | dbt\_cloud\_pr\_123\_456 | | analytics\_prod | marketing | analytics\_prod\_marketing | | alice\_dev | marketing | alice\_dev\_marketing | | dbt\_cloud\_pr\_123\_456 | marketing | dbt\_cloud\_pr\_123\_456\_marketing | #### How do I use custom schemas? To specify a custom schema for a model, use the `schema` configuration key. As with any configuration, you can do one of the following: * apply this configuration to a specific model by using a config block within a model * apply it to a subdirectory of models by specifying it in your `dbt_project.yml` file orders.sql ```sql {{ config(schema='marketing') }} select ... ``` dbt\_project.yml ```yaml # models in `models/marketing/ will be built in the "*_marketing" schema models: my_project: marketing: +schema: marketing ``` #### Understanding custom schemas When first using custom schemas, it's a common misunderstanding to assume that a model *only* uses the new `schema` configuration; for example, a model that has the configuration `schema: marketing` would be built in the `marketing` schema. However, dbt puts it in a schema like `<target_schema>_marketing`. There's a good reason for this deviation. Each dbt user has their own target schema for development (refer to [Managing Environments](#managing-environments)). If dbt ignored the target schema and only used the model's custom schema, every dbt user would create models in the same schema and would overwrite each other's work. By combining the target schema and the custom schema, dbt ensures that objects it creates in your data warehouse don't collide with one another. If you prefer to use different logic for generating a schema name, you can change the way dbt generates a schema name (see below). ##### How does dbt generate a model's schema name? dbt uses a default macro called `generate_schema_name` to determine the name of the schema that a model should be built in. The following code represents the default macro's logic: ```sql {% macro generate_schema_name(custom_schema_name, node) -%} {%- set default_schema = target.schema -%} {%- if custom_schema_name is none -%} {{ default_schema }} {%- else -%} {{ default_schema }}_{{ custom_schema_name | trim }} {%- endif -%} {%- endmacro %} ``` <br /> 💡 Use Jinja's whitespace control to tidy your macros! When you're modifying macros in your project, you might notice extra white space in your code in the `target/compiled` folder. You can remove unwanted spaces and lines with Jinja's [whitespace control](https://docs.getdbt.com/faqs/Jinja/jinja-whitespace.md) by using a minus sign. For example, use `{{- ... -}}` or `{%- ... %}` around your macro definitions (such as `{%- macro generate_schema_name(...) -%} ... {%- endmacro -%}`). #### Changing the way dbt generates a schema name If your dbt project has a custom macro called `generate_schema_name`, dbt will use it instead of the default macro. This allows you to customize the name generation according to your needs. To customize this macro, copy the example code in the section [How does dbt generate a model's schema name](#how-does-dbt-generate-a-models-schema-name) into a file named `macros/generate_schema_name.sql` and make changes as necessary. Be careful. dbt will ignore any custom `generate_schema_name` macros included in installed packages.  Warning: Don't replace \`default\_schema\` in the macro If you're modifying how dbt generates schema names, don't just replace `{{ default_schema }}_{{ custom_schema_name | trim }}` with `{{ custom_schema_name | trim }}` in the `generate_schema_name` macro. If you remove `{{ default_schema }}`, it causes developers to override each other's models if they create their own custom schemas. This can also cause issues during development and continuous integration (CI). ❌ The following code block is an example of what your code *should not* look like: ```sql {% macro generate_schema_name(custom_schema_name, node) -%} {%- set default_schema = target.schema -%} {%- if custom_schema_name is none -%} {{ default_schema }} {%- else -%} # The following is incorrect as it omits {{ default_schema }} before {{ custom_schema_name | trim }}. {{ custom_schema_name | trim }} {%- endif -%} {%- endmacro %} ``` ##### generate\_schema\_name arguments | Argument | Description | Example | | -------------------- | -------------------------------------------------------------------------------------------- | ---------------------------------------------------- | | custom\_schema\_name | The configured value of `schema` in the specified node, or `none` if a value is not supplied | `marketing` | | node | The `node` that is currently being processed by dbt | `{"name": "my_model", "resource_type": "model",...}` | ##### Jinja context available in generate\_schema\_name If you choose to write custom logic to generate a schema name, it's worth noting that not all variables and methods are available to you when defining this logic. In other words: the `generate_schema_name` macro is compiled with a limited Jinja context. The following context methods *are* available in the `generate_schema_name` macro: | Jinja context | Type | Available | | --------------------------------------------------------------------------------- | -------- | ------------------ | | [target](https://docs.getdbt.com/reference/dbt-jinja-functions/target.md) | Variable | ✅ | | [env\_var](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md) | Variable | ✅ | | [var](https://docs.getdbt.com/reference/dbt-jinja-functions/var.md) | Variable | Limited, see below | | [exceptions](https://docs.getdbt.com/reference/dbt-jinja-functions/exceptions.md) | Macro | ✅ | | [log](https://docs.getdbt.com/reference/dbt-jinja-functions/log.md) | Macro | ✅ | | Other macros in your project | Macro | ✅ | | Other macros in your packages | Macro | ✅ | ##### Which vars are available in generate\_schema\_name? Globally-scoped variables and variables defined on the command line with [--vars](https://docs.getdbt.com/docs/build/project-variables.md) are accessible in the `generate_schema_name` context. ##### Managing different behaviors across packages See docs on macro `dispatch`: ["Managing different global overrides across packages"](https://docs.getdbt.com/reference/dbt-jinja-functions/dispatch.md) #### A built-in alternative pattern for generating schema names A common customization is to use the custom schema in production when provided, with the target schema serving only as a fallback if no custom schema is specified. In other environments, such as development and CI, custom schema configurations are ignored, defaulting to the target schema instead. Production Environment (`target.name == 'prod'`) | Target schema | Custom schema | Resulting schema | | --------------- | ------------- | ---------------- | | analytics\_prod | None | analytics\_prod | | analytics\_prod | marketing | marketing | Development/CI Environment (`target.name != 'prod'`) | Target schema | Custom schema | Resulting schema | | ------------------------ | ------------- | ------------------------ | | alice\_dev | None | alice\_dev | | alice\_dev | marketing | alice\_dev | | dbt\_cloud\_pr\_123\_456 | None | dbt\_cloud\_pr\_123\_456 | | dbt\_cloud\_pr\_123\_456 | marketing | dbt\_cloud\_pr\_123\_456 | Similar to the regular macro, this approach guarantees that schemas from different environments will not collide. dbt ships with a macro for this use case — called `generate_schema_name_for_env` — which is disabled by default. To enable it, add a custom `generate_schema_name` macro to your project that contains the following code: macros/generate\_schema\_name.sql ```sql -- put this in macros/generate_schema_name.sql {% macro generate_schema_name(custom_schema_name, node) -%} {{ generate_schema_name_for_env(custom_schema_name, node) }} {%- endmacro %} ``` When using this macro, you'll need to set the target name in your production job to `prod`. #### Managing environments In the `generate_schema_name` macro examples shown in the [built-in alternative pattern](#a-built-in-alternative-pattern-for-generating-schema-names) section, the `target.name` context variable is used to change the schema name that dbt generates for models. If the `generate_schema_name` macro in your project uses the `target.name` context variable, you must ensure that your different dbt environments are configured accordingly. While you can use any naming scheme you'd like, we typically recommend: * **dev** — Your local development environment; configured in a `profiles.yml` file on your computer. * **ci** — A [continuous integration](https://docs.getdbt.com/docs/platform/git/connect-github.md) environment running on pull requests in GitHub, GitLab, and so on. * **prod** — The production deployment of your dbt project, like in dbt, Airflow, or [similar](https://docs.getdbt.com/docs/deploy/deployments.md). If your schema names are being generated incorrectly, double-check your target name in the relevant environment. For more information, consult the [managing environments in dbt](https://docs.getdbt.com/docs/local/dbt-environments.md) guide. #### Related docs * [Customize dbt models database, schema, and alias](https://docs.getdbt.com/guides/customize-schema-alias.md?step=1) to learn how to customize dbt models database, schema, and alias * [Custom database](https://docs.getdbt.com/docs/build/custom-databases.md) to learn how to customize dbt model database * [Custom aliases](https://docs.getdbt.com/docs/build/custom-aliases.md) to learn how to customize dbt model alias name --- ### Custom target names #### dbt Scheduler You can define a custom target name for any dbt job to correspond to settings in your dbt project. This is helpful if you have logic in your dbt project that behaves differently depending on the specified target, for example: ```sql select * from a_big_table -- limit the amount of data queried in dev {% if target.name != 'prod' %} where created_at > date_trunc('month', current_date) {% endif %} ``` To set a custom target name for a job in dbt, configure the **Target Name** field for your job in the Job Settings page. [![Overriding the target name to 'prod'](/img/docs/dbt-platform/using-dbt-platform/jobs-settings-target-name.png?v=2 "Overriding the target name to 'prod'")](#)Overriding the target name to 'prod' #### dbt Studio IDE When developing in dbt, you can set a custom target name in your user credentials. Click your account name above the profile icon in the left panel, select **Account settings**, then go to **Credentials**. Choose the project to update the target name. [![Overriding the target name to 'dev'](/img/docs/dbt-platform/using-dbt-platform/development-credentials.png?v=2 "Overriding the target name to 'dev'")](#)Overriding the target name to 'dev' --- ### Data health signals Preview ### Data health signals [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt platform | Starter, Enterprise, Enterprise+ Data health signals offer a quick, at-a-glance view of data health when browsing your resources in Catalog. They keep you informed on the status of your resource's health using the indicators **Healthy**, **Caution**, **Degraded**, or **Unknown**. Note, we don’t calculate data health for non-dbt resources. * Supported resources are [models](https://docs.getdbt.com/docs/build/models.md), [sources](https://docs.getdbt.com/docs/build/sources.md), and [exposures](https://docs.getdbt.com/docs/build/exposures.md). * For accurate health data, ensure the resource is up-to-date and had a recent job run. * Each data health signal reflects key data health components, such as test success status, missing resource descriptions, missing tests, absence of builds in 30-day windows, [and more](#data-health-signal-criteria). [![View data health signals for your models.](/img/docs/collaborate/dbt-explorer/data-health-signal.png?v=2 "View data health signals for your models.")](#)View data health signals for your models. #### Access data health signals Access data health signals in the following places: * In the [search function](https://docs.getdbt.com/docs/explore/explore-projects.md#search-resources) or under **Models**, **Sources**, or **Exposures** in the **Resource** tab. * For sources, the data health signal also indicates the [source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) status. * In the **Health** column on [each resource's details page](https://docs.getdbt.com/docs/explore/explore-projects.md#view-resource-details). Hover over or click the signal to view detailed information. * In the **Health** column of public models tables. * In the [DAG lineage graph](https://docs.getdbt.com/docs/explore/explore-projects.md#project-lineage). Click any node to open the node details panel where you can view it and its details. * In [Data health tiles](https://docs.getdbt.com/docs/explore/data-tile.md) through an embeddable iFrame and visible in your BI dashboard. [![Access data health signals in multiple places in dbt Catalog.](/img/docs/collaborate/dbt-explorer/data-health-signal.gif?v=2 "Access data health signals in multiple places in dbt Catalog.")](#)Access data health signals in multiple places in dbt Catalog. #### Data health signal criteria Each resource has a health state that is determined by specific set of criteria. Select the following tabs to view the criteria for that resource type. ##### Models The health state of a model is determined by the following criteria: | **Health state** | **Criteria** | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ✅ **Healthy** | All of the following must be true:<br /><br />- Built successfully in the last run<br />- Built in the last 30 days<br />- Model has tests configured<br />- All tests passed<br />- All upstream [sources are fresh](https://docs.getdbt.com/docs/build/sources.md#source-data-freshness) or freshness is not applicable (set to `null`)<br />- Has a description | | 🟡 **Caution** | One of the following must be true:<br /><br />- Not built in the last 30 days<br />- Tests are not configured<br />- Tests return warnings<br />- One or more upstream sources are stale:<br />    - Has a freshness check configured<br />    - Freshness check ran in the past 30 days<br />    - Freshness check returned a warning<br />- Missing a description | | 🔴 **Degraded** | One of the following must be true:<br /><br />- Model failed to build<br />- Model has failing tests<br />- One or more upstream sources are stale:<br />    - Freshness check hasn’t run in the past 30 days<br />    - Freshness check returned an error | | ⚪ **Unknown** | - Unable to determine health of resource; no job runs have processed the resource. | ##### Sources The health state of a source is determined by the following criteria: | **Health state** | **Criteria** | | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ✅ Healthy | All of the following must be true:<br /><br />- Freshness check configured<br />- Freshness check passed<br />- Freshness check ran in the past 30 days<br />- Has a description | | 🟡 Caution | One of the following must be true:<br /><br />- Freshness check returned a warning<br />- Freshness check not configured<br />- Freshness check not run in the past 30 days<br />- Missing a description | | 🔴 Degraded | - Freshness check returned an error | | ⚪ Unknown | Unable to determine health of resource; no job runs have processed the resource. | ##### Exposures The health state of an exposure is determined by the following criteria: | **Health state** | **Criteria** | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ✅ Healthy | All of the following must be true:<br /><br />- Underlying sources are fresh<br />- Underlying models built successfully<br />- Underlying models’ tests passing<br /> | | 🟡 Caution | One of the following must be true:<br /><br />- At least one underlying source’s freshness checks returned a warning<br />- At least one underlying model was skipped<br />- At least one underlying model’s tests returned a warning<br /> | | 🔴 Degraded | One of the following must be true:<br /><br />- At least one underlying source’s freshness checks returned an error<br />- At least one underlying model did not build successfully<br />- At least one model’s tests returned an error | --- ### Data health tile dbt platform | Enterprise, Enterprise+ With data health tiles, stakeholders will get an at-a-glance confirmation on whether the data they’re looking at is stale or degraded. It allows teams to immediately go back into Catalog to see more details and investigate issues. The data health tile: * Distills [data health signals](https://docs.getdbt.com/docs/explore/data-health-signals.md) for data consumers. * Deep links you into Catalog where you can further dive into upstream data issues. * Provides richer information and makes it easier to debug. * Revamps the existing, [job-based tiles](#job-based-data-health). Data health tiles rely on [exposures](https://docs.getdbt.com/docs/build/exposures.md) to surface data health signals in your dashboards. An exposure defines how specific outputs — like dashboards or reports — depend on your data models. Exposures in dbt can be configured in two ways: * Manual — Defined [manually](https://docs.getdbt.com/docs/build/exposures.md#declaring-an-exposure) and explicitly in your project’s YAML files. * Automatic — Pulled automatically for supported dbt integrations. dbt automatically [creates and visualizes downstream exposures](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures.md), removing the need for manual YAML definitions. These downstream exposures are stored in dbt’s metadata system, appear in [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md), and behave like manual exposures, however they don’t exist in YAML files. [![Example of passing Data health tile in your dashboard.](/img/docs/collaborate/dbt-explorer/data-tile-pass.png?v=2 "Example of passing Data health tile in your dashboard.")](#)Example of passing Data health tile in your dashboard. [![Embed data health tiles in your dashboards to distill data health signals for data consumers.](/img/docs/collaborate/dbt-explorer/data-tiles.png?v=2 "Embed data health tiles in your dashboards to distill data health signals for data consumers.")](#)Embed data health tiles in your dashboards to distill data health signals for data consumers. #### Prerequisites * You must have a dbt account on an [Enterprise-tier plan](https://www.getdbt.com/pricing/). * You must be an account admin to set up [service tokens](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md#permissions-for-service-account-tokens). * You must have [develop permissions](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md). * You have [exposures](https://docs.getdbt.com/docs/build/exposures.md) defined in your project: * If using manual exposures, they must be explicitly defined in your YAML files. * If using automatic downstream exposures, ensure your BI tool is [configured](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md) with dbt. * You have [source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) enabled in the job that generates this exposure. * The exposure used for the data health tile must have the [`type` property](https://docs.getdbt.com/docs/build/exposures.md#available-properties) set to `dashboard`. Otherwise, you won't be able to view the **Embed data health tile in your dashboard** dropdown in Catalog. #### View exposure in dbt Catalog First, be sure to enable [source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) in the job that generates this exposure. 1. Navigate to Catalog by clicking on the **Catalog** link in the navigation. 2. In the main **Overview** page, go to the left navigation. 3. Click on **Exposures** to view the [exposures](https://docs.getdbt.com/docs/build/exposures.md) list. 4. Select a dashboard exposure and go to the **General** tab to view the data health information. 5. In this tab, you’ll see: * The exposure name at the top of the page. * The resource type (for example, **Exposure**) and an **Updated at** timestamp. * A **Data health** section that shows when the **Last check completed** (relative time since the check ran). * A freshness summary (for example, **Data is fresh**) and a breakdown of upstream **Sources** (fresh, delayed, stale, skipped, outdated, unconfigured). * A quality summary (for example, **Data quality passed**) and a breakdown of upstream **Models** and **Tests**. 6. You can click the **Open in Dashboard** button in the upper right to immediately view this in your analytics tool. [![View an exposure in dbt Catalog.](/img/docs/collaborate/dbt-explorer/data-tile-exposures.png?v=2 "View an exposure in dbt Catalog.")](#)View an exposure in dbt Catalog. #### Embed in your dashboard Once you’ve navigated to the exposure in Catalog, you’ll need to set up your data health tile and [service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md). You can embed data health tile to any analytics tool that supports URL or iFrame embedding. Follow these steps to set up your data health tile: 1. Go to **Account settings** in dbt. 2. Select **API tokens** in the left sidebar and then **Service tokens**. 3. Click on **Create service token** and give it a name. 4. Select the [**Metadata Only**](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) permission. This token will be used to embed the tile in your dashboard in the later steps. [![Set up your dashboard status tile and service token to embed a data health tile](/img/docs/collaborate/dbt-explorer/data-tile-setup.png?v=2 "Set up your dashboard status tile and service token to embed a data health tile")](#)Set up your dashboard status tile and service token to embed a data health tile 5. Copy the **Metadata Only** token and save it in a secure location. You'll need it token in the next steps. 6. Navigate back to Catalog and select an exposure. tip The exposure used for the data health tile must have the [`type` property](https://docs.getdbt.com/docs/build/exposures.md#available-properties) set to `dashboard`. Otherwise, you won't be able to view the **Embed data health tile in your dashboard** dropdown in Catalog. 7. Below the **Data health** section, expand on the toggle for instructions on how to embed the exposure tile (if you're an account admin with develop permissions). 8. In the expanded toggle, you'll see a text field where you can paste your **Metadata Only token**. [![Expand the toggle to embed data health tile into your dashboard.](/img/docs/collaborate/dbt-explorer/data-tile-example.png?v=2 "Expand the toggle to embed data health tile into your dashboard.")](#)Expand the toggle to embed data health tile into your dashboard. 9. Once you’ve pasted your token, you can select either **URL** or **iFrame** depending on which you need to add to your dashboard. If your analytics tool supports iFrames, you can embed the dashboard tile within it. #### Examples The following examples show how to embed the data health tile in Omni, PowerBI, Tableau, and Sigma. ##### Omni example Follow these steps to embed the data health tile in [Omni](https://omni.co/): [![Embed data health tile in Omni](/img/docs/collaborate/dbt-explorer/omni-example.png?v=2 "Embed data health tile in Omni")](#)Embed data health tile in Omni 1. Create a dashboard in Omni. 2. Copy the iFrame snippet available in Catalog's **Data health** section, under the **Embed data health into your dashboard** toggle. 3. Add a new Text or Markdown [element](https://docs.omni.co/visualize-present/dashboards/text-markdown) in your Dashboard with the code from step 2, it should be in the following format: ```html/text <iframe src='https://YOUR_ACCOUNT_PREFIX.metadata.REGION.dbt.com/exposure-tile?uniqueId=exposure.EXPOSURE_NAME&environmentType=production&environmentId=ENV_ID_NUMBER&token=<YOUR_METADATA_TOKEN>' title='Exposure Status Tile'></iframe> ``` 4. Save the tile and your Omni dashboard should now have a dbt platform hosted data health tile that is automatically updated based on the state of your dbt environment. ##### PowerBI example You can embed the data health tile iFrame in PowerBI using PowerBI Pro Online, Fabric PowerBI, or PowerBI Desktop. [![Embed data health tile iFrame in PowerBI](/img/docs/collaborate/dbt-explorer/power-bi.png?v=2 "Embed data health tile iFrame in PowerBI")](#)Embed data health tile iFrame in PowerBI Follow these steps to embed the data health tile in PowerBI: 1. Create a dashboard in PowerBI and connect to your database to pull in the data. 2. Create a new PowerBI measure by right-clicking on your **Data**, **More options**, and then **New measure**. [![Create a new PowerBI measure.](/img/docs/collaborate/dbt-explorer/power-bi-measure.png?v=2 "Create a new PowerBI measure.")](#)Create a new PowerBI measure. 3. Navigate to Catalog, select the exposure, and expand the [**Embed data health into your dashboard**](https://docs.getdbt.com/docs/explore/data-tile.md#embed-in-your-dashboard) toggle. 4. Go to the **iFrame** tab and copy the iFrame code. Make sure the Metadata Only token is already set up. 5. In PowerBI, paste the iFrame code you copied into your measure calculation window. The iFrame code should look like this: ```html/text <iframe src='https://1234.metadata.ACCESS_URL/exposure-tile?uniqueId=exposure.EXPOSURE_NAME&environmentType=staging&environmentId=123456789&token=YOUR_METADATA_TOKEN' title='Exposure status tile' height='400'></iframe> ``` [![In the 'Measure tools' tab, replace your values with the iFrame code.](/img/docs/collaborate/dbt-explorer/power-bi-measure-tools.png?v=2 "In the 'Measure tools' tab, replace your values with the iFrame code.")](#)In the 'Measure tools' tab, replace your values with the iFrame code. 6. PowerBI desktop doesn't support HTML rendering by default, so you need to install an HTML component from the PowerBI Visuals Store. 7. To do this, go to **Build visuals** and then **Get more visuals**. 8. Login with your PowerBI account. 9. There are several third-party HTML visuals. The one tested for this guide is [HTML content](https://appsource.microsoft.com/en-us/product/power-bi-visuals/WA200001930?tab=Overview). Install it, but please keep in mind it's a third-party plugin not created or supported by dbt Labs. 10. Drag the metric with the iFrame code into the HTML content widget in PowerBI. This should now display your data health tile. [![Drag the metric with the iFrame code into the HTML content widget in PowerBI. This should now display your data health tile.](/img/docs/collaborate/dbt-explorer/power-bi-final.png?v=2 "Drag the metric with the iFrame code into the HTML content widget in PowerBI. This should now display your data health tile.")](#)Drag the metric with the iFrame code into the HTML content widget in PowerBI. This should now display your data health tile. *Refer to [this tutorial](https://www.youtube.com/watch?v=SUm9Hnq8Th8) for additional information on embedding a website into your Power BI report.* ##### Tableau example Follow these steps to embed the data health tile in Tableau: [![Embed data health tile iFrame in Tableau](/img/docs/collaborate/dbt-explorer/tableau-example.png?v=2 "Embed data health tile iFrame in Tableau")](#)Embed data health tile iFrame in Tableau 1. Create a dashboard in Tableau and connect to your database to pull in the data. 2. Copy the URL or iFrame snippet available in Catalog's **Data health** section, under the **Embed data health into your dashboard** toggle. 3. Insert a **Web Page** object. 4. Paste the copied URL and click **Ok**. The URL you copy from Catalog looks similar to the following but may include additional Tableau-specific parameters (such as `biResourceId` and `autoBiProvider=tableau`) that are required to resolve the exposure: ```text/html https://metadata.ACCESS_URL/exposure-tile?uniqueId=exposure.EXPOSURE_NAME&environmentType=production&environmentId=220370&token=<YOUR_METADATA_TOKEN> ``` 5. You should now see the data health tile embedded in your Tableau dashboard. ##### Sigma example Follow these steps to embed the data health tile in Sigma: [![Embed data health tile in Sigma](/img/docs/collaborate/dbt-explorer/sigma-example.jpg?v=2 "Embed data health tile in Sigma")](#)Embed data health tile in Sigma 1. Create a dashboard in Sigma and connect to your database to pull in the data. 2. Ensure you've copied the URL or iFrame snippet available in Catalog's **Data health** section, under the **Embed data health into your dashboard** toggle. 3. Add a new embedded UI element in your Sigma Workbook in the following format: ```html/text https://metadata.ACCESS_URL/exposure-tile?uniqueId=exposure.EXPOSURE_NAME&environmentType=production&environmentId=ENV_ID_NUMBER&token=<YOUR_METADATA_TOKEN> ``` *Note, replace the placeholders with your actual values.* 4. You should now see the data health tile embedded in your Sigma dashboard. #### Job-based data health Legacy The default experience is the [environment-based data health tile](#view-exposure-in-dbt-explorer) with Catalog. This section is for legacy job-based data health tiles. If you're using the revamped environment-based exposure tile, refer to the previous section. Expand the following to learn more about the legacy job-based data health tile.  Job-based data health In dbt, the [Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md) can power dashboard status tiles, which are job-based. A dashboard status tile is placed on a dashboard (specifically: anywhere you can embed an iFrame) to give insight into the quality and freshness of the data feeding into that dashboard. This is done in dbt [exposures](https://docs.getdbt.com/docs/build/exposures.md). ###### Functionality The dashboard status tile looks like this: [![](/img/docs/dbt-platform/using-dbt-platform/dashboard-status-tiles/passing-tile.jpeg?v=2)](#) The data freshness check fails if any sources feeding into the exposure are stale. The data quality check fails if any dbt tests fail. A failure state could look like this: [![](/img/docs/dbt-platform/using-dbt-platform/dashboard-status-tiles/failing-tile.jpeg?v=2)](#) Clicking into **see details** from the Dashboard Status Tile takes you to a landing page where you can learn more about the specific sources, models, and tests feeding into this exposure. ###### Setup First, be sure to enable [source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) in the job that generates this exposure. In order to set up your dashboard status tile, here is what you need: 1. **Metadata Only token.** You can learn how to set up a Metadata-Only token [here](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md). 2. **Exposure name.** You can learn more about how to set up exposures [here](https://docs.getdbt.com/docs/build/exposures.md). 3. **Job iD.** Remember that you can select your job ID directly from the URL when looking at the relevant job in dbt. You can insert these three fields into the following iFrame, and then embed it **anywhere that you can embed an iFrame**: ```html/text <iframe src='https://metadata.YOUR_ACCESS_URL/exposure-tile?name=<exposure_name>&jobId=<job_id>&token=<metadata_only_token>' title='Exposure Status Tile'></iframe> ``` Replace `YOUR_ACCESS_URL` with your region and plan's Access URL dbt is hosted in multiple regions in the world and each region has a different access URL. Replace `YOUR_ACCESS_URL` with the appropriate [Access URL](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for your region and plan. For example, if your account is hosted in the EMEA region, you would use the following iFrame code: ```html/text <iframe src='https://metadata.emea.dbt.com/exposure-tile?name=<exposure_name>&jobId=<job_id>&token=<metadata_only_token>' title='Exposure Status Tile'></iframe> ``` ###### Embedding with BI tools The dashboard status tile should work anywhere you can embed an iFrame. But below are some tactical tips on how to integrate with common BI tools. ##### Mode ###### Mode Mode allows you to directly [edit the HTML](https://mode.com/help/articles/report-layout-and-presentation/#html-editor) of any given report, where you can embed the iFrame. Note that Mode has also built its own [integration](https://mode.com/get-dbt/) with the dbt Discovery API! ##### Looker ###### Looker Looker does not allow you to directly embed HTML and instead requires creating a [custom visualization](https://docs.looker.com/admin-options/platform/visualizations). One way to do this for admins is to: * Add a [new visualization](https://fishtown.looker.com/admin/visualizations) on the visualization page for Looker admins. You can use [this URL](https://metadata.cloud.getdbt.com/static/looker-viz.js) to configure a Looker visualization powered by the iFrame. It will look like this: [![Configure a Looker visualization powered by the iFrame](/img/docs/dbt-platform/using-dbt-platform/dashboard-status-tiles/looker-visualization.jpeg?v=2 "Configure a Looker visualization powered by the iFrame")](#)Configure a Looker visualization powered by the iFrame * Once you have set up your custom visualization, you can use it on any dashboard! You can configure it with the exposure name, job ID, and token relevant to that dashboard. [![](</img/docs/dbt-platform/using-dbt-platform/dashboard-status-tiles/custom-looker.jpeg ?v=2>)](#) ##### Tableau ###### Tableau Tableau does not require you to embed an iFrame. You only need to use a Web Page object on your Tableau Dashboard and a URL in the following format: ```html/text https://metadata.YOUR_ACCESS_URL/exposure-tile?name=<exposure_name>&jobId=<job_id>&token=<metadata_only_token> ``` Replace `YOUR_ACCESS_URL` with your region and plan's Access URL dbt is hosted in multiple regions in the world and each region has a different access URL. Replace `YOUR_ACCESS_URL` with the appropriate [Access URL](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for your region and plan. For example, if your account is hosted in the North American region, you would use the following code: ```html/text https://metadata.cloud.getdbt.com/exposure-tile?name=<exposure_name>&jobId=<job_id>&token=<metadata_only_token> ``` [![Configure Tableau by using a Web page object.](/img/docs/dbt-platform/using-dbt-platform/dashboard-status-tiles/tableau-object.png?v=2 "Configure Tableau by using a Web page object.")](#)Configure Tableau by using a Web page object. ##### Sigma ###### Sigma Sigma does not require you to embed an iFrame. Add a new embedded UI element in your Sigma Workbook in the following format: ```html/text https://metadata.YOUR_ACCESS_URL/exposure-tile?name=<exposure_name>&jobId=<job_id>&token=<metadata_only_token> ``` Replace `YOUR_ACCESS_URL` with your region and plan's Access URL dbt is hosted in multiple regions in the world and each region has a different access URL. Replace `YOUR_ACCESS_URL` with the appropriate [Access URL](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for your region and plan. For example, if your account is hosted in the APAC region, you would use the following code: ```html/text https://metadata.au.dbt.com/exposure-tile?name=<exposure_name>&jobId=<job_id>&token=<metadata_only_token> ``` [![Configure Sigma by using an embedded UI element.](/img/docs/dbt-platform/using-dbt-platform/dashboard-status-tiles/sigma-embed.gif?v=2 "Configure Sigma by using an embedded UI element.")](#)Configure Sigma by using an embedded UI element. --- ### Databricks and Apache Iceberg dbt supports materializing Iceberg tables in Unity Catalog in two ways: * **Simplest:** The model config `table_format = 'iceberg'` instructs dbt to materialize this model as an Iceberg table in Unity Catalog * **Extensible:** Define an Iceberg catalog in `catalogs.yml` and configure this model with `catalog_name` dbt supports creating Iceberg tables for two Databricks materializations: * [Table](https://docs.getdbt.com/docs/build/materializations.md#table) * [Incremental](https://docs.getdbt.com/docs/build/materializations.md#incremental) #### Databricks Iceberg support Databricks is built on [Delta Lake](https://docs.databricks.com/aws/en/delta/) and stores data in the [Delta table](https://docs.databricks.com/aws/en/introduction/delta-comparison#delta-tables-default-data-table-architecture) format. Databricks supports two methods for creating Iceberg tables in its data catalog, [Unity Catalog](https://docs.databricks.com/aws/en/data-governance/unity-catalog/): * Creating [Unity Catalog managed Iceberg tables](https://docs.databricks.com/aws/en/tables/managed). Databricks Runtime 16.4 LTS and later support this feature. * Enabling [Iceberg reads](https://docs.databricks.com/aws/en/delta/uniform) on Delta tables. These tables still use the Delta file format, but generate both Delta and Iceberg-compatible metadata. Databricks Runtime 14.3 LTS and later support this feature. dbt supports both creating managed Iceberg tables and Iceberg-enabled Delta tables (formerly [UniForm](https://www.databricks.com/blog/delta-uniform-universal-format-lakehouse-interoperability)). (Applies to dbt v2.0 and later) The [`use_uniform`](#choose-between-managed-iceberg-and-uniform) config determines whether dbt creates a managed Iceberg table or a Delta table. Because `use_uniform` defaults to `false`, setting `table_format: 'iceberg'` creates a managed Iceberg table. External Iceberg compute engines can read from and write to these Iceberg tables using Unity Catalog's [Iceberg REST API endpoint](https://docs.databricks.com/aws/en/external-access/iceberg). However, Databricks only has limited support for reading from external Iceberg catalogs (and externally managed Iceberg tables) through [Databricks catalog federation](https://docs.databricks.com/aws/en/query-federation/catalog-federation) (configured outside of dbt). dbt doesn't yet support enabling [Iceberg v3](https://docs.databricks.com/aws/en/iceberg/iceberg-v3) on managed Iceberg tables. (Applies to dbt v2.0 and later) ##### Choose between managed Iceberg and UniForm Set `table_format: 'iceberg'` on a model to create a Unity Catalog managed Iceberg table. You don't need a `catalogs.yml` file or a `catalog_name` config. Two model configs control which kind of Iceberg table dbt creates: | Config | Type | Required | Description | Default | | -------------- | ------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | | `table_format` | String | Yes, to create an Iceberg table | Set to `iceberg` to materialize the model as an Iceberg table. | `default` | | `use_uniform` | Boolean | No | When `false`, dbt creates a Unity Catalog managed Iceberg table (`create table ... using iceberg`). When `true`, dbt creates a Delta table with Iceberg reads enabled (`create table ... using delta tblproperties (...)`). | `false` | This model creates a managed Iceberg table: models/my\_iceberg\_model.sql ```sql {{ config( materialized='table', table_format='iceberg' ) }} select * from {{ ref('raw_orders') }} ``` To create an Iceberg-enabled Delta table instead, set `use_uniform` to `true`: models/my\_uniform\_model.sql ```sql {{ config( materialized='table', table_format='iceberg', use_uniform=true ) }} select * from {{ ref('raw_orders') }} ``` ##### External tables dbt also supports creating externally-managed Iceberg tables using the model configuration [`location_root`](https://docs.getdbt.com/reference/resource-configs/databricks-configs.md#configuring-tables). Databricks' DDL for creating tables requires a fully qualified `location`. dbt defines this parameter on the user's behalf to streamline usage and enforce basic isolation of table data: * When you set a `location_root` string, dbt generates a `location` string of the form: `{{ location_root }}/{{ model_name }}`. * If you set the configuration option `include_full_name_in_path` to `true`, dbt generates a `location` string of the form `{{ location_root }}/{{ database_name}}/{{ schema_name }}/{{ model_name }}`. In dbt v2, you may set `location_root` within the catalog definition in `catalogs.yml`, under `config.databricks` (in the new catalog spec) or `adapter_properties` (in the old catalog spec). ##### Catalogs Configure catalogs in order to: * Define multiple configurations for Databricks-managed Iceberg tables within one catalog * Support cross-platform Mesh Notes: * Every Databricks catalog may optionally configure `table_format`. By default, this is set to `iceberg` for `catalog_type=unity`, and `default` for `hive_metastore`. * On Databricks, `catalog_name` takes precedence over the `catalog` config when determining the model's top-level namespace. #### Configure catalog integration for Iceberg tables 1. Create a `catalogs.yml` at the top level of your dbt project. An example of Unity Catalog as the catalog: ##### New spec (beta) catalogs.yml ```yaml catalogs: - name: unity_catalog type: unity table_format: iceberg # optional config: databricks: # optional location_root: s3://cloud-storage-uri ``` ##### Old spec catalogs.yml ```yaml catalogs: - name: unity_catalog active_write_integration: unity_catalog_integration write_integrations: - name: unity_catalog_integration table_format: iceberg catalog_type: unity file_format: delta adapter_properties: location_root: s3://cloud-storage-uri ``` 2. Add the `catalog_name` config parameter in either a config block (inside the .sql model file), properties YAML file (model folder), or your project YAML file (`dbt_project.yml`). <br /> <br /> An example of `iceberg_model.sql`: ```yaml {{ config( materialized = 'table', catalog_name = 'unity_catalog' ) }} select * from {{ ref('jaffle_shop_customers') }} ``` 3. Execute the dbt model with a `dbt run -s iceberg_model`. --- ### dbt AI FAQs Answers to common questions about dbt AI features, including dbt Wizard and dbt Copilot. dbt Wizard is an AI-powered assistant fully integrated into your dbt experience that handles the tedious tasks, speeds up workflows, and ensures consistency, helping you deliver exceptional data products faster. dbt Labs is committed to protecting your privacy and data. This page provides information about how dbt Wizard handles your data. For more information, check out the [dbt Labs AI development principles](https://www.getdbt.com/legal/ai-principles) page. #### Overview  What is dbt Wizard? dbt Wizard is the latest and recommended agentic experience for governed data development in dbt, available in both the dbt platform and the terminal CLI. It helps teams ship trusted dbt changes faster and with less risk by understanding project context, routing to the right dbt tools, validating changes, and supporting review before changes are persisted. Use dbt Wizard to investigate lineage and downstream impact, generate or refactor SQL from natural-language prompts, create [documentation](https://docs.getdbt.com/docs/build/documentation.md), [data tests](https://docs.getdbt.com/docs/build/data-tests.md), [metrics](https://docs.getdbt.com/docs/build/metrics-overview.md), and [semantic models](https://docs.getdbt.com/docs/build/semantic-models.md), and validate changes with warehouse awareness. In the dbt platform, dbt Wizard is available in the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-studio-ai.md) and the [dbt Wizard home tab](https://docs.getdbt.com/docs/platform/wizard-home.md). dbt Copilot is available in [Canvas](https://docs.getdbt.com/docs/platform/use-canvas.md) and [Insights](https://docs.getdbt.com/docs/explore/dbt-insights.md). In the CLI, you can use dbt Wizard from your terminal for local development workflows.  Where can I find dbt Wizard? dbt Wizard is available in the dbt platform and as a terminal CLI. * In the platform, you can use dbt Wizard in the [Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) for governed data development in dbt. * In the CLI, use the [dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-cli.md) for local development and automation. To use dbt Wizard in the platform, you need any dbt [platform account](https://www.getdbt.com/contact). All dbt platform plans have access to dbt Wizard in Studio IDE and the [home tab](https://docs.getdbt.com/docs/platform/wizard-home.md). AI features are enabled by default. Admins can [turn them off or back on anytime](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md).  What are the benefits of using dbt Wizard? dbt Wizard helps teams ship trusted dbt changes faster and with less risk. Use it to: * Ask project-aware questions and investigate lineage, dependencies, and downstream impact. * Generate or refactor SQL from natural-language prompts. * Generate documentation, tests, metrics, and semantic models. * Validate changes with warehouse awareness before review. * Review proposed file changes as diffs before they are persisted. dbt Wizard is built into dbt experiences with dbt governance, privacy, and security controls.  Is dbt Wizard the same as dbt Copilot? No, dbt Wizard and dbt Copilot are separate products. dbt Wizard is the latest and next generation of agentic product available in the dbt platform and as a CLI. It uses your project context to help you develop governed dbt changes faster. Think of it like a smart AI agent that has a map of your project. Instead of having to read through each file and understand the context, it can answer questions and help you develop *and* validate your changes faster. dbt Copilot features for those who have access to them, include quick-action buttons in Studio IDE, the Copilot pane in Insights and Canvas.  Can I use my existing dbt Copilot action allotment with dbt Wizard? No, dbt Copilot actions apply only to dbt Copilot usage. Refer to [dbt Wizard billing and AI access FAQs](https://docs.getdbt.com/docs/dbt-ai/wizard-billing-faqs.md). #### Availability  Who has access to dbt Wizard? **In the dbt platform**: When enabled by an admin, dbt Wizard is available to users with a dbt [developer license](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md) on any [dbt platform account](https://www.getdbt.com/contact). **In the CLI**: dbt Wizard CLI uses dbt Labs-managed models (OpenAI, Anthropic, or open weight models), billed through dbt's spend-limit billing. You can also bring your own API key or credentials for a supported provider using [BYOK](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md): OpenAI, Anthropic, Azure AI Foundry, AWS Bedrock, Google Gemini, or Snowflake Cortex (preview). BYOK is a good backup option if you want to manage AI costs directly — token costs are billed directly by whichever provider you choose. Install and configure the CLI on your local machine. Refer to [Use dbt Wizard locally](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) for more information.  Is dbt Wizard available for all deployment types? Yes, dbt Wizard is deployed everywhere, including [multi-tenant and single-tenant deployments](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). #### How it works  What data/code is used to train the AI model supporting dbt Wizard? dbt Wizard is supported by dbt Labs-managed models (OpenAI, Anthropic, or open weight models), or by several third-party pre-trained AI models at your discretion (BYOK OpenAI, BYOK Anthropic, BYOK Azure AI Foundry, and so on). When using managed OpenAI, our agreement with OpenAI prohibits OpenAI from retaining your data persistently. Refer to our [dbt Labs AI principles page](https://www.getdbt.com/legal/ai-principles) for more information.  Which AI model providers does dbt Wizard use? In the dbt platform, dbt Wizard uses a managed OpenAI model by default. dbt Labs also offers managed open weight models. On any plan, you can also [bring your own provider keys](https://docs.getdbt.com/docs/platform/wizard-byok-platform.md) for OpenAI, Anthropic, or Azure AI Foundry. The [dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-cli.md) supports the same dbt Labs-managed models, or OpenAI, Anthropic, Azure AI Foundry, AWS Bedrock, Google Gemini, and Snowflake Cortex (preview) in bring-your-own-key mode. Refer to [Configure BYOK](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) and [Supported AI providers](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/overview.md#supported-ai-providers) for more information. For how model choice affects cost, which models draw from your consumption pool, and how BYOK billing works, refer to [dbt Wizard billing and access FAQs](https://docs.getdbt.com/docs/dbt-ai/wizard-billing-faqs.md). Refer to the [Model Provider Rate table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) and [Service Consumption Table](https://www.getdbt.com/legal/service-consumption-table) for more information.  Do we support BYOK (bring your own key) at the project level? In dbt platform, the dbt Wizard BYOK option is currently an account-only configuration. However, there may be a future where we make this configurable on a project-level. BYOK is a good option if you want to manage AI costs directly, rather than using a dbt Labs-managed model. dbt Wizard CLI supports BYOK locally for OpenAI, Anthropic, Azure AI Foundry, AWS Bedrock, Google Gemini, and Snowflake Cortex (preview). #### Privacy and data This section covers dbt Wizard in the dbt platform. For what the CLI collects and how to opt out, refer to [dbt Wizard CLI data use and telemetry](https://docs.getdbt.com/docs/dbt-ai/wizard-telemetry.md). [![How data flows when dbt Wizard runs in the dbt platform, and what is and isn't shared with the AI provider.](/img/docs/dbt-platform/wizard-architecture-platform.png?v=2 "How data flows when dbt Wizard runs in the dbt platform, and what is and isn't shared with the AI provider.")](#)How data flows when dbt Wizard runs in the dbt platform, and what is and isn't shared with the AI provider. Your prompt, project metadata, and any query results you approve are sent to the AI provider over TLS. Warehouse credentials are never shared, queries run only after you approve them, and dbt Labs-managed providers can't retain your data or train on it.  Does dbt Wizard access my warehouse data? dbt Wizard can run dbt commands and queries on your behalf, and every query needs your explicit permission first. When a query runs, dbt Wizard sends those results — which may include row-level data — to the AI provider so it can respond in your session. For dbt-managed AI providers, we have zero data retention (ZDR) agreements in place that prevents the provider from retaining or using this data for training. If you bring your own AI provider (BYOK), that provider's terms will govern retention and training. Always review AI output for accuracy. Refer to [Model Provider Rate table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) and [Service Consumption Table](https://www.getdbt.com/legal/service-consumption-table) for more information.  Does dbt Wizard store or use personal data? dbt Wizard stores your conversation history — including your prompts, responses, and any query results returned during your session — so you can revisit past chats. Conversation history is retained for 90 days; feedback you submit on a dbt Wizard conversation is retained for 400 days. You can delete your conversation history or feedback at any time in the product. You control the information you submit to dbt Wizard. dbt Labs does not use your prompts, conversation history, command results, or feedback to train AI models.  Is my data used by dbt Labs to train AI models? No. dbt Labs does not use customer content processed by dbt Wizard — including warehouse query results, prompts, or conversation history — for AI model training. A zero data retention (ZDR) policy is also in place with AI providers, which prevents training on the provider side as well.  Does dbt Labs share my personal data with third parties? dbt Labs only shares client personal information as needed to perform the services, under client instructions, or for legal, tax, or compliance reasons.  Can dbt Wizard data be deleted upon client written request? Yes. dbt Wizard conversation history is retained for 90 days by default, feedback you submit on a dbt Wizard conversation is retained for 400 days, and you can delete this information in the product at any time. To the extent a client identifies personal or sensitive information uploaded to dbt Labs systems, that data can be deleted within 30 days of written request.  Does dbt Labs own the output generated by dbt Wizard? dbt Labs will not dispute your ownership of any output (e.g., code or artifacts) that are unique to your company generated when you use dbt Wizard. Your code will not be used to train AI models for the benefit of dbt Labs or other third parties, including other dbt Labs customers.  Does dbt Labs have terms in place? dbt Wizard is governed by our [Terms of Use](https://www.getdbt.com/terms-of-use). In the event clients prefer additional terms, clients may enter into the presigned AI & Beta Addendum (the dbt Labs signature will be dated as of the date the client signs). Contact your account manager for more information. Clients who signed with terms after January 2024 don't need additional terms prior to enabling dbt Wizard. Longer term clients have also protected their data through confidentiality and data deletion obligations. In the event clients prefer additional terms, clients may enter into the presigned AI & Beta Addendum (the dbt Labs signature will be dated as of the date the client signs). #### Considerations  What are the considerations for using dbt Wizard? * dbt Wizard is not available in the dbt API. Future releases are planned that may bring dbt Wizard to even more parts of the dbt application. #### dbt Wizard allowlisting URLs  Allowlisting URLs dbt Wizard doesn't specifically block AI-related URLs. However, if your organization use endpoint protection platforms, firewalls, or network proxies (such as Zscaler), you may encounter the following issues with dbt Wizard: * Block unknown or AI-related domains. * Break TLS/SSL traffic to inspect it. * Disallow specific ports or services. We recommend the following URLs to be allowlisted: **For dbt Wizard in the IDE**: * `/api/ide/accounts/${accountId}/develop/${developId}/ai/generate_generic_tests/...` * `/api/ide/accounts/${accountId}/develop/${developId}/ai/generate_documentation/...` * `/api/ide/accounts/${accountId}/develop/${developId}/ai/generate_semantic_model/...` * `/api/ide/accounts/${accountId}/develop/${developId}/ai/generate_inline` * `/api/ide/accounts/${accountId}/develop/${developId}/ai/generate_metrics/...` * `/api/ide/accounts/${accountId}/develop/${developId}/ai/track_response` **For dbt Copilot in Canvas**: * `/api/private/visual-editor/v1/ai/llm-generate` * `/api/private/visual-editor/v1/ai/track-response` * `/api/private/visual-editor/v1/files/${fileId}/llm-generate-dag-through-chat` --- ### dbt Catalog FAQs dbt platform [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) is dbt’s new knowledge base and lineage visualization experience. It offers an interactive and high-level view of your company’s entire data estate, where you can dive deep into the context you need to understand and improve lineage so your teams can trust the data they’re using to make decisions. #### Overview  How does dbt Catalog help with data quality? Catalog makes it easy and intuitive to understand your entire lineage — from data source to the reporting layer — so you can troubleshoot, improve, and optimize your pipelines. With built-in features like project recommendations and model performance analysis, you can be sure you have appropriate test and documentation coverage across your estate and quickly spot and remediate slow-running models. With column-level lineage, you can quickly identify the potential downstream impacts of table changes or work backwards to quickly understand the root cause of an incident. Catalog gives teams the insights they need to improve data quality proactively, ensuring pipelines stay performant and data trust remains solid.  How is dbt Catalog priced? Catalog is generally available to all regions and deployment types on all dbt [Enterprise-tier and Starter plans](https://www.getdbt.com/). Certain features within Catalog, such as project recommendations, multi-project lineage, column-level lineage, and more are only available on the Enterprise and Enterprise+ plans. Catalog can be accessed by users with developer and read-only seats.  What happened to dbt Docs? Catalog is the default documentation experience for dbt customers. dbt Docs is still available but doesn't offer the same speed, metadata, or visibility as Catalog and will become a legacy feature. #### How dbt Catalog works  Can I use dbt Catalog on-premises or with my self-hosted dbt deployment? No. Catalog and all of its features are only available as a dbt platform user experience. Catalog reflects the metadata from your dbt platform project(s) and their runs.  How does dbt Catalog support dbt environments? Catalog supports a production or staging [deployment environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md) for each project you want to explore. It defaults to the latest production or staging state of a project. Users can only assign one production and one staging environment per dbt project. Support for development (dbt platform CLI and Studio IDE) environments is coming soon.  How do I get started in Catalog? How does it update? Simply select **Catalog** from the dbt top navigation bar. Catalog automatically updates after each dbt run in the given project’s environment (production, by default). The dbt commands you run within the environment will generate and update the metadata in Catalog, so make sure to run the correct combination of commands within the jobs of the environment; for more details, refer to [Generate metadata](https://docs.getdbt.com/docs/explore/explore-projects.md#generate-metadata).  Is it possible to export dbt lineage to an external system or catalog? Yes. The lineage that powers Catalog is also available through the Discovery API.  How does dbt Catalog integrate with third-party tools to show end-to-end lineage? Catalog reflects all the lineage defined within the dbt project. Our vision for Catalog is to incorporate additional metadata from external tools like data loaders (sources) and BI/analytics tools (exposures) integrated with dbt, all seamlessly incorporated into the lineage of the dbt project.  Why did previously visible data in dbt Catalog disappear? Catalog automatically deletes stale metadata after 3 months if no jobs were run to refresh it. To avoid this, make sure you schedule jobs to run more frequently than 3 months with the necessary commands. #### Key features  Does dbt Catalog support multi-project discovery (dbt Mesh)? Yes. Refer to [Explore multiple projects](https://docs.getdbt.com/docs/explore/explore-multiple-projects.md) to learn more.  What kind of search capabilities does dbt Catalog support? Resource search capabilities include using keywords, partial strings (fuzzy search), and set operators like `OR`. Meanwhile, lineage search supports using dbt selectors. For details, refer to [Keyword search](https://docs.getdbt.com/docs/explore/explore-projects.md#search-resources).  Can I view model execution information for a job that is currently being run? dbt updates the performance and metrics after a job run. However, **Model performance** charts only display data for *completed* UTC days. This means that runs from the current UTC day won't appear in the charts until the UTC day changes (midnight UTC). For example, if you're in US Pacific time, you won't see the current day's runs reflected until 4:00 PM PT.  Can I analyze the number of successful model runs within a month? A chart of models built by month is available in thedbt dashboard.  Can model or column descriptions be edited within dbt? Yes. Today, you can edit descriptions in the Studio IDE or dbt platform CLI by changing the YAML files within the dbt project. In the future, Catalog will support more ways of editing descriptions.  Where do recommendations come from? Can they be customized? Recommendations largely mirror the best practice rules from the `dbt_project_evaluator` package. At this time, recommendations can’t be customized. In the future, Catalog will likely support recommendation customization capabilities (for example, in project code). #### Column-level lineage  What are the best use cases for column-level lineage in dbt Catalog? Column-level lineage in Catalog can be used to improve many data development workflows, including: * **Audit** — Visualize how data moves through and is used in your dbt project * **Root cause** — Improve time to detect and resolve data quality issues, tracking back to the source * **Impact analysis** — Trace transformations and usage to avoid introducing issues for consumers * **Efficiency** — Prune unnecessary columns to reduce costs and data team overhead  Does the column-level lineage remain functional even if column names vary between models? Yes. Column-level lineage can handle name changes across instances of the column in the dbt project.  Can multiple projects leverage the same column definition? No. Cross-project column lineage is supported in the sense of viewing how a public model is used across projects, but not on a column-level.  Can column descriptions be propagated down in downstream lineage automatically? Yes, a reused column, labeled as passthrough or rename, inherits its description from source and upstream model columns. In other words, source and upstream model columns propagate their descriptions downstream whenever they are not transformed, meaning you don’t need to manually define the description. Refer to [Inherited column descriptions](https://docs.getdbt.com/docs/explore/column-level-lineage.md#inherited-column-descriptions) for more info.  Is column-level lineage also available in the development tab? Not currently, but we plan to incorporate column-level awareness across features in dbt in the future. #### Availability, access, and permissions  How can non-developers interact with dbt Catalog? Read-only users can consume metadata in Catalog. More bespoke experiences and exploration avenues for analysts and less-technical contributors will be provided in the future.  Does dbt Catalog require a specific dbt plan? Catalog is available on dbt Starter and all Enterprise plans. Certain features within Catalog are only available on Enterprise and Enterprise+ plans, including Model Performance, project recommendations, multi-project lineage, column-level lineage, and more. Refer to the [availability by plan](https://docs.getdbt.com/docs/explore/explore-projects.md#availability-by-plan) table for a complete list.  Will self-hosted dbt users be able to leverage any of these new dbt Catalog features? No. Catalog is a dbt-only product experience.  Is it possible to access dbt Catalog using a read-only license? Yes, users with read-only access can use the Catalog. Specific feature availability within Catalog will depend on your dbt plan.  Is there an easy way to share useful dbt Catalog content with people outside of dbt? The ability to embed and share views is being evaluated as a potential future capability.   Is dbt Catalog accessible from other areas inside dbt? Yes, you can [access Catalog from various dbt features](https://docs.getdbt.com/docs/explore/access-from-dbt-platform.md), ensuring you have a seamless experience navigating between resources and lineage in your project. While the primary way to access Catalog is through the **Catalog** link in the navigation, you can also access it from the [Studio IDE](https://docs.getdbt.com/docs/explore/access-from-dbt-platform.md#studio-ide), [the lineage tab in jobs](https://docs.getdbt.com/docs/explore/access-from-dbt-platform.md#lineage-tab-in-jobs), and the [model timing tab in jobs](https://docs.getdbt.com/docs/explore/access-from-dbt-platform.md#model-timing-tab-in-jobs). --- ### dbt Copilot dbt platform | Starter, Enterprise, Enterprise+ dbt Copilot helps you generate SQL, documentation, tests, and semantic models in the dbt platform. dbt Wizard is the recommended agent for dbt work dbt Wizard is the recommended AI agent for governed data development in dbt. It handles the full development lifecycle — investigation, building, validation, and shipping — grounded in your dbt project's lineage, tests, contracts, and metric definitions. Refer to [dbt AI FAQs](https://docs.getdbt.com/docs/dbt-ai/dbt-ai-faqs.md#is-dbt-wizard-the-same-as-dbt-copilot), [Billing](https://docs.getdbt.com/docs/dbt-ai/wizard-billing-faqs.md), and [dbt's Terms of Use](https://www.getdbt.com/terms-of-use) for more information. *The earlier version of dbt Copilot in the Studio IDE is available only to a limited set of accounts. dbt Wizard is available to all accounts and is the recommended way to develop with AI in the Studio IDE — it covers everything dbt Copilot's quick actions did, plus multi-step changes with built-in validation.* #### Where to access dbt Copilot dbt Copilot is available across the following experiences in the dbt platform. Refer to the links for more info on how to use each experience. * [dbt Copilot in Canvas](https://docs.getdbt.com/docs/platform/build-canvas-copilot.md): Build visual models using natural language prompts in Canvas * [dbt Copilot in Insights](https://docs.getdbt.com/docs/dbt-ai/analyst-agent.md): Chat with your data and get answers powered by the dbt Semantic Layer in Insights #### Action limits by plan dbt Copilot usage is metered in actions — one completed AI request counts as one action. Each plan includes a monthly action allotment per license: | Plan | Actions per month | | ----------- | ----------------- | | Developer | ❌ | | Starter | 100 | | Enterprise | 5,000 | | Enterprise+ | 10,000 | Enterprise and Enterprise+ limits don't apply if you [bring your own key (BYOK)](https://docs.getdbt.com/docs/platform/wizard-byok-platform.md), since your AI provider bills that usage directly. Legacy Enterprise-tier plans enrolled before May 1, 2025 have a 1,000 action limit. Refer to [dbt AI usage](https://docs.getdbt.com/docs/platform/billing/dbt-ai-usage.md) for what counts as an action, what happens when you hit the limit, and how to check your usage. #### Configure AI provider for dbt Copilot You need dbt admin permissions to change account settings and configure providers. dbt Copilot is configured separately from dbt Wizard — for dbt Wizard providers, refer to [Manage AI features](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md#configure-ai-provider). dbt Copilot supports fewer providers than dbt Wizard, including bring your own key (BYOK) on any plan: * dbt Labs-managed OpenAI API key * BYOK OpenAI API key * BYOK Azure OpenAI API key Snowflake Cortex, AWS Bedrock, Azure AI Foundry, and Anthropic aren't supported for dbt Copilot. dbt Copilot is available for inline assistance in Studio IDE, Canvas, and Insights. Configure it separately from dbt Wizard if your team uses these inline AI experiences. **To configure a provider:** 1. Click your account name and select **Account settings** in the side menu. 2. Under **Settings**, click **Copilot**. 3. Under **AI providers**, click **Edit** to configure the AI integration. 4. Select your **Key management** option from the dropdown, then follow the steps for your provider below. ##### dbt Labs OpenAI 1. Select the toggle for **dbt Labs** to use dbt Labs' managed OpenAI key. 2. Click **Save**. [![Example of the dbt Labs integration page](/img/docs/dbt-platform/account-integration-dbtlabs.png?v=2 "Example of the dbt Labs integration page")](#)Example of the dbt Labs integration page ##### OpenAI 1. Select the toggle for **OpenAI** to use your own OpenAI key. 2. Enter the API key. 3. Click **Save**. [![Example of the OpenAI integration page](/img/docs/dbt-platform/account-integration-openai.png?v=2 "Example of the OpenAI integration page")](#)Example of the OpenAI integration page Data residency limitation OpenAI projects with [data residency controls](https://platform.openai.com/docs/guides/your-data#data-residency-controls) enabled and configured for the United States (project region set to US) don't currently support BYOK. These projects can only use the API key in the dbt platform configuration. Specifying custom endpoints required for data residency isn’t yet supported, and we’re evaluating a solution for this. To use BYOK, ensure your OpenAI project doesn’t have data residency controls enabled. Projects without project region settings will use the standard OpenAI endpoint (`https://api.openai.com`) and support BYOK. ##### Azure OpenAI To learn about deploying your own OpenAI model on Azure, refer to [Deploy models on Azure OpenAI](https://learn.microsoft.com/en-us/azure/ai-studio/how-to/deploy-models-openai). Configure credentials for your Azure OpenAI deployment in dbt as follows: 1. Locate your Azure OpenAI configuration in your Azure Deployment details page. 2. Enter your Azure OpenAI API key. 3. Enter the **Endpoint**, **API Version**, and **Deployment / Model Name**. 4. Click **Save**. Use the full Azure Target URI For the **Endpoint** field, enter the full Azure Target URI from Azure — not just the base endpoint. Entering only the base endpoint, for example `https://<resource>.openai.azure.com`, prevents credential validation and blocks setup. Supported formats include: * `https://<resource>.openai.azure.com/openai/deployments/<deployment>/chat/completions?api-version=<version>` * `https://<resource>.openai.azure.com/openai/responses?api-version=<version>` [![Example of Azure OpenAI integration section](/img/docs/dbt-platform/account-integration-azure-manual.png?v=2 "Example of Azure OpenAI integration section")](#)Example of Azure OpenAI integration section * For BYOK, enable the latest text generation models as well as the `text-embedding-3-small` model. * Ensure your project doesn't have data residency controls enabled. #### Considerations * dbt Copilot is a separate experience from dbt Wizard. For agentic, full-lifecycle AI development, use [dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-overview.md). * AI features are enabled by default. Admins can [turn them off or back on anytime](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md). * Certain features are only available on Enterprise and Enterprise+ plans. Refer to [Billing](https://docs.getdbt.com/docs/platform/billing.md) for details. * dbt Copilot doesn't yet support generating semantic models with the latest YAML spec. --- ### dbt Copilot in Insights Beta ### dbt Copilot in Insights [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt platform | Enterprise, Enterprise+ dbt Copilot in Insights lets you chat with your data and get accurate answers powered by the [dbt Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md). Unlike generic AI chat interfaces, dbt Wizard in Insights provides consistent, explainable results with transparent SQL, lineage, and data policies. dbt Wizard is the recommended agent for dbt work dbt Wizard is the recommended AI agent for governed data development in dbt. It handles the full development lifecycle — investigation, building, validation, and shipping — grounded in your dbt project's lineage, tests, contracts, and metric definitions. Refer to [dbt AI FAQs](https://docs.getdbt.com/docs/dbt-ai/dbt-ai-faqs.md#is-dbt-wizard-the-same-as-dbt-copilot), [Billing](https://docs.getdbt.com/docs/dbt-ai/wizard-billing-faqs.md), and [dbt's Terms of Use](https://www.getdbt.com/terms-of-use) for more information. #### Prerequisites * Enable beta features under **Account settings** > **Personal profile** > **Experimental features**. See [Preview new dbt platform features](https://docs.getdbt.com/docs/dbt-versions/experimental-features.md) for steps. * Have access to [dbt Insights](https://docs.getdbt.com/docs/explore/dbt-insights.md) and meet those prerequisites. * Be on a dbt platform [Enterprise-tier](https://www.getdbt.com/pricing) plan — [book a demo](https://www.getdbt.com/contact) to learn more about Insights. * Available on all [tenant](https://docs.getdbt.com/docs/platform/about-platform/tenancy.md) configurations. * Have a dbt [developer license](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md) with access to Insights. * Configured [user credentials](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md#get-started-with-the-studio-ide). #### Using dbt Copilot in Insights Use dbt Copilot to analyze your data and get contextualized results in real time by asking natural language questions to the [Insights](https://docs.getdbt.com/docs/explore/dbt-insights.md) dbt Wizard in Insights agent. 1. Click the **dbt Copilot** icon in the Query console sidebar menu. 2. In the dropdown menu above the dbt Copilot prompt box, select **Agent**. 3. In the dbt Copilot prompt box, enter your question. 4. Click **↑** to submit your question. The agent then translates natural language questions into structured queries, executes queries against governed dbt models and metrics, and returns results with references, assumptions, and possible next steps. The agent can loop through these steps multiple times if it hasn't reached a complete answer, allowing for complex, multi-step analysis.⁠ dbt Insights automatically executes the SQL query suggested by dbt Copilot in Insights, and you can preview the SQL results in the **Data** tab. 5. Confirm the results or continue asking the agent for more insights about your data. Your conversation with the agent remains even if you switch tabs within dbt Insights. However, they disappear when you navigate out of Insights or when you close your browser. [![Using dbt Copilot in Insights](/img/docs/dbt-insights/insights-copilot-agent.png?v=2 "Using dbt Copilot in Insights")](#)Using dbt Copilot in Insights --- ### dbt licensing (Applies to dbt v2.0 and later) dbt v2 has the following distributions today, all free to install and run. | Distribution | Package | Use it when | | ------------ | ---------- | ------------------------------------------------------------------------------------ | | dbt v2 | `dbt` | The recommended v2 experience. | | dbt OSS | `dbt-core` | Your organization has a strict requirement to use the Apache 2.0 open-source runtime | If you have a older project that isn’t ready to move to v2, continue using v1.x for compatibility. For new or upgraded projects, we recommend [upgrading to v2](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md?version=2.0#dbt-v2). #### Which one should I use? For most people: dbt v2. It has more [capabilities](https://docs.getdbt.com/docs/dbt/dbt-availability.md?version=2.0#what-you-get-with-fusion) out of the box than the open source v2 — including a built-in high-performance SQL linter — even if you never create a dbt account. We recommend everyone to just [install dbt](https://docs.getdbt.com/docs/local/install-dbt.md) and get dbt v2 by default. Typically you'd choose the open source installtion directly only if you're in one of two specific situations: 1. Your organization's license policy requires a strict open-source distribution 2. You're building something custom on top of the OSS code itself. Already running dbt v1? You don't have to move to v2 — it's still fully supported. Over time, new capabilities will land in v2 only, so most people will eventually want to [upgrade](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md?version=2.0#dbt-v2). To check which distribution you're using, run `dbt --version` in the command line. #### What changed, and what didn't **Changed:** * v2 is available through two distributions: dbt v2 and dbt OSS. * dbt OSS, the Apache 2.0 open-source distribution for v2, is powered by the shared Rust engine code and is now available in `dbt-core`. * dbt v2 builds on dbt OSS and extends it with additional proprietary capabilities under the dbt Product Licensing Agreement. **Unchanged:** * dbt v1 is still fully available and still Apache 2.0. * dbt v2 is still completely free to use, with some features unlocked by a free login or a paid dbt platform account — not required for any distribution. * Contributing to dbt is still open to everyone. #### Licensing details [dbt v1](https://github.com/dbt-labs/dbt) is released under the [Apache 2.0 license](http://www.apache.org/licenses/LICENSE-2.0). dbt v2 is proprietary to dbt Labs, made available under the [dbt Product Licensing Agreement](https://www.getdbt.com/dbt-fusion-engine-license-agreement). For the full breakdown of what's permitted under each license — source visibility, contributions, modifications, self-hosting, and redistribution — see the [dbt Licensing FAQ](https://www.getdbt.com/licenses-faq). dbt platform is a separate hosted product governed by its own [terms of service](https://www.getdbt.com/terms-of-use). Also not to be confused with dbt platform [licenses](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md). --- ### dbt platform v1 Compatible track - changelog dbt platform | Free Select the **v1 Compatible** and **v1 Extended** release tracks if you need a less-frequent release cadence, the ability to test new dbt releases before they go live in production, and/or ongoing compatibility with the latest open source releases of dbt v1. Each monthly **v1 Compatible** release includes functionality matching up-to-date open source versions of dbt v1 and adapters at the time of release. For more information, see [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). #### August 2026 Release date: August 18, 2026 This compatible release includes functionality from the following versions of dbt v1 OSS: ```text dbt-core==1.12.0 # shared interfaces dbt-adapters==1.24.5 dbt-common==1.38.0 dbt-extractor==0.6.0 dbt-semantic-interfaces==0.9.0 dbt-sl-sdk[sync]==0.13.4 dbt-state==2.42.0 # adapters dbt-athena==1.11.0 dbt-bigquery==1.12.0 dbt-databricks==1.12.4 dbt-fabric==1.9.4 dbt-postgres==1.11.0 dbt-redshift==1.11.0 dbt-snowflake==1.12.0 dbt-spark==1.10.3 dbt-synapse==1.8.5 dbt-teradata==1.11.0 dbt-trino==1.10.3 ``` Changelogs: * [dbt-core 1.12.0](https://github.com/dbt-labs/dbt/blob/1.12.latest/CHANGELOG.md) * [dbt-adapters 1.24.5](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/CHANGELOG.md) * [dbt-common 1.38.0](https://github.com/dbt-labs/dbt-common/blob/main/CHANGELOG.md) * [dbt-state 2.42.0](https://github.com/dbt-labs/dbt-state) * [dbt-athena 1.11.0](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-athena/CHANGELOG.md) * [dbt-bigquery 1.12.0](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-bigquery/CHANGELOG.md) * [dbt-databricks 1.12.4](https://github.com/databricks/dbt-databricks/blob/main/CHANGELOG.md) * [dbt-fabric 1.9.4](https://github.com/microsoft/dbt-fabric/releases/tag/v1.9.4) * [dbt-postgres 1.11.0](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-postgres/CHANGELOG.md) * [dbt-redshift 1.11.0](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-redshift/CHANGELOG.md) * [dbt-snowflake 1.12.0](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-snowflake/CHANGELOG.md) * [dbt-spark 1.10.3](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-spark/CHANGELOG.md) * [dbt-synapse 1.8.5](https://github.com/microsoft/dbt-synapse/blob/v1.8.latest/CHANGELOG.md) * [dbt-teradata 1.11.0](https://github.com/Teradata/dbt-teradata/releases/tag/v1.11.0) * [dbt-trino 1.10.3](https://github.com/starburstdata/dbt-trino/blob/master/CHANGELOG.md) #### July 2026 Release date: July 21, 2026 ##### dbt platform ##### Fixes * Prune stale cross-project dependencies from injected external nodes before linking. A publication can list a transitive public ancestor that has since been deleted upstream; injecting it as a dangling `depends_on` edge previously failed compile with `GraphDependencyNotFoundError`. The dead edge is now dropped with a warning and restored once the producing project's publication is regenerated. ##### Dependencies * Pin `pyathena` to v3.35 and older This compatible release includes functionality from the following versions of dbt v1 OSS: ```text dbt-core==1.11.12 # shared interfaces dbt-adapters==1.22.10 dbt-common==1.37.5 dbt-extractor==0.6.0 dbt-semantic-interfaces==0.9.0 dbt-sl-sdk[sync]==0.13.4 # adapters dbt-athena==1.10.2 dbt-bigquery==1.11.1 dbt-databricks==1.12.2 dbt-fabric==1.9.4 dbt-postgres==1.10.2 dbt-redshift==1.10.2 dbt-snowflake==1.11.6 dbt-spark==1.10.3 dbt-synapse==1.8.5 dbt-teradata==1.10.3 dbt-trino==1.10.2 ``` Changelogs: * [dbt-core 1.11.12](https://github.com/dbt-labs/dbt/blob/1.11.latest/CHANGELOG.md) * [dbt-adapters 1.22.10](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/CHANGELOG.md) * [dbt-common 1.37.5](https://github.com/dbt-labs/dbt-common/blob/main/CHANGELOG.md) * [dbt-athena 1.10.2](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-athena/CHANGELOG.md) * [dbt-bigquery 1.11.1](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-bigquery/CHANGELOG.md) * [dbt-databricks 1.12.2](https://github.com/databricks/dbt-databricks/blob/main/CHANGELOG.md) * [dbt-fabric 1.9.4](https://github.com/microsoft/dbt-fabric/releases/tag/v1.9.4) * [dbt-postgres 1.10.2](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-postgres/CHANGELOG.md) * [dbt-redshift 1.10.2](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-redshift/CHANGELOG.md) * [dbt-snowflake 1.11.6](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-snowflake/CHANGELOG.md) * [dbt-spark 1.10.3](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-spark/CHANGELOG.md) * [dbt-synapse 1.8.5](https://github.com/microsoft/dbt-synapse/blob/v1.8.latest/CHANGELOG.md) * [dbt-teradata 1.10.3](https://github.com/Teradata/dbt-teradata/releases/tag/v1.10.3) * [dbt-trino 1.10.2](https://github.com/starburstdata/dbt-trino/blob/master/CHANGELOG.md) #### June 2026 Release date: June 15, 2026 ##### dbt cloud-based platform This compatible release includes functionality from the following versions of dbt v1 OSS: ```text dbt-core==1.11.11 # shared interfaces dbt-adapters==1.22.10 dbt-common==1.37.5 dbt-extractor==0.6.0 dbt-semantic-interfaces==0.9.0 dbt-sl-sdk[sync]==0.13.4 # adapters dbt-athena==1.10.1 dbt-bigquery==1.11.1 dbt-databricks==1.12.1 dbt-fabric==1.9.4 dbt-postgres==1.10.0 dbt-redshift==1.10.1 dbt-snowflake==1.11.5 dbt-spark==1.10.1 dbt-synapse==1.8.5 dbt-teradata==1.10.2 dbt-trino==1.10.2 ``` Changelogs: * [dbt-core 1.11.11](https://github.com/dbt-labs/dbt/blob/1.11.latest/CHANGELOG.md) * [dbt-adapters 1.22.10](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/CHANGELOG.md) * [dbt-common 1.37.5](https://github.com/dbt-labs/dbt-common/blob/main/CHANGELOG.md) * [dbt-athena 1.10.1](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-athena/CHANGELOG.md) * [dbt-bigquery 1.11.1](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-bigquery/CHANGELOG.md) * [dbt-databricks 1.12.1](https://github.com/databricks/dbt-databricks/blob/main/CHANGELOG.md) * [dbt-fabric 1.9.4](https://github.com/microsoft/dbt-fabric/releases/tag/v1.9.4) * [dbt-postgres 1.10.0](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-postgres/CHANGELOG.md) * [dbt-redshift 1.10.1](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-redshift/CHANGELOG.md) * [dbt-snowflake 1.11.5](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-snowflake/CHANGELOG.md) * [dbt-spark 1.10.1](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-spark/CHANGELOG.md) * [dbt-synapse 1.8.5](https://github.com/microsoft/dbt-synapse/blob/v1.8.latest/CHANGELOG.md) * [dbt-teradata 1.10.2](https://github.com/Teradata/dbt-teradata/releases/tag/v1.10.2) * [dbt-trino 1.10.2](https://github.com/starburstdata/dbt-trino/blob/master/CHANGELOG.md) #### May 2026 The compatible release scheduled for May 2026 will be skipped in order to stabilize `dbt-core 1.12.0` release across the dbt platform. Compatible releases will resume in June 2026. #### April 2026 Release date: April 21, 2026 ##### dbt cloud-based platform This compatible release includes functionality from the following versions of dbt v1 OSS: ```text dbt-core==1.11.8 # shared interfaces dbt-adapters==1.22.10 dbt-common==1.37.3 dbt-extractor==0.6.0 dbt-semantic-interfaces==0.9.0 dbt-sl-sdk[sync]==0.13.2 # adapters dbt-athena==1.10.0 dbt-bigquery==1.11.1 dbt-databricks==1.11.7 dbt-fabric==1.9.4 dbt-postgres==1.10.0 dbt-redshift==1.10.1 dbt-snowflake==1.11.4 dbt-spark==1.10.1 dbt-synapse==1.8.4 dbt-teradata==1.10.2 dbt-trino==1.10.1 ``` Changelogs: * [dbt-core 1.11.8](https://github.com/dbt-labs/dbt/blob/1.11.latest/CHANGELOG.md) * [dbt-adapters 1.22.10](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/CHANGELOG.md) * [dbt-common 1.37.3](https://github.com/dbt-labs/dbt-common/blob/main/CHANGELOG.md) * [dbt-athena 1.10.0](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-athena/CHANGELOG.md) * [dbt-bigquery 1.11.1](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-bigquery/CHANGELOG.md) * [dbt-databricks 1.11.7](https://github.com/databricks/dbt-databricks/blob/main/CHANGELOG.md) * [dbt-fabric 1.9.4](https://github.com/microsoft/dbt-fabric/releases/tag/v1.9.4) * [dbt-postgres 1.10.0](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-postgres/CHANGELOG.md) * [dbt-redshift 1.10.1](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-redshift/CHANGELOG.md) * [dbt-snowflake 1.11.4](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-snowflake/CHANGELOG.md) * [dbt-spark 1.10.1](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-spark/CHANGELOG.md) * [dbt-synapse 1.8.4](https://github.com/microsoft/dbt-synapse/blob/v1.8.latest/CHANGELOG.md) * [dbt-teradata 1.10.2](https://github.com/Teradata/dbt-teradata/releases/tag/v1.10.2) * [dbt-trino 1.10.1](https://github.com/starburstdata/dbt-trino/blob/master/CHANGELOG.md) #### March 2026 The compatible release scheduled for March 2026 will be skipped in order to further stabilize the minor upgrade of `dbt-core==1.11.6` across the dbt platform. Compatible releases will resume in April 2026. #### February 2026 Release date: February 27, 2026 ##### dbt cloud-based platform ##### Features * Support partial success result status for Advanced CI ##### Dependencies * Update dbt-databricks upper bound to 1.12 This compatible release includes functionality from the following versions of dbt v1 OSS: ```text dbt-core==1.11.6 # shared interfaces dbt-adapters==1.22.6 dbt-common==1.37.2 dbt-extractor==0.6.0 dbt-semantic-interfaces==0.9.0 dbt-sl-sdk[sync]==0.13.1 # adapters dbt-athena==1.10.0 dbt-bigquery==1.11.0 dbt-databricks==1.11.5 dbt-fabric==1.9.4 dbt-postgres==1.10.0 dbt-redshift==1.10.1 dbt-snowflake==1.11.2 dbt-spark==1.10.1 dbt-synapse==1.8.4 dbt-teradata==1.10.1 dbt-trino==1.10.1 ``` Changelogs: * [dbt-core 1.11.6](https://github.com/dbt-labs/dbt/blob/1.11.latest/CHANGELOG.md#dbt-core-1116---february-17-2026) * [dbt-adapters 1.22.6](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/CHANGELOG.md#dbt-adapters-1226---february-17-2026) * [dbt-common 1.37.2](https://github.com/dbt-labs/dbt-common/blob/main/CHANGELOG.md#dbt-common-1372---december-15-2025) * [dbt-athena 1.10.0](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-athena/CHANGELOG.md#dbt-athena-1100---december-22-2025) * [dbt-bigquery 1.11.0](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-bigquery/CHANGELOG.md#dbt-bigquery-1110---december-22-2025) * [dbt-databricks 1.11.5](https://github.com/databricks/dbt-databricks/blob/main/CHANGELOG.md#dbt-databricks-1115-feb-19-2026) * [dbt-fabric 1.9.4](https://github.com/microsoft/dbt-fabric/releases/tag/v1.9.4) * [dbt-postgres 1.10.0](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-postgres/CHANGELOG.md#dbt-postgres-1100---december-22-2025) * [dbt-redshift 1.10.1](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-redshift/CHANGELOG.md#dbt-redshift-1101---february-11-2026) * [dbt-snowflake 1.11.2](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-snowflake/CHANGELOG.md#dbt-snowflake-1112---february-11-2026) * [dbt-spark 1.10.1](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-spark/CHANGELOG.md#dbt-spark-1101---february-11-2026) * [dbt-synapse 1.8.4](https://github.com/microsoft/dbt-synapse/blob/v1.8.latest/CHANGELOG.md) * [dbt-teradata 1.10.1](https://github.com/Teradata/dbt-teradata/releases/tag/v1.10.1) * [dbt-trino 1.10.1](https://github.com/starburstdata/dbt-trino/blob/master/CHANGELOG.md#dbt-trino-1101---january-16-2026) #### January 2026 Release date: January 22, 2026 ##### dbt cloud-based platform ##### Under the Hood * Add debug log for local md5 hash for dbt v2 conformance * Resolve Click CLI UserWarning regarding --target and --profile usage in Advanced CI This compatible release includes functionality from the following versions of dbt v1 OSS: ```text dbt-core==1.10.19 # shared interfaces dbt-adapters==1.22.5 dbt-common==1.37.2 dbt-extractor==0.6.0 dbt-semantic-interfaces==0.9.0 dbt-sl-sdk[sync]==0.13.1 # adapters dbt-athena==1.10.0 dbt-bigquery==1.11.0 dbt-databricks==1.10.19 dbt-fabric==1.9.4 dbt-postgres==1.10.0 dbt-redshift==1.10.0 dbt-snowflake==1.11.1 dbt-spark==1.9.3 dbt-synapse==1.8.4 dbt-teradata==1.10.1 dbt-trino==1.10.1 ``` Changelogs: * [dbt-core 1.10.19](https://github.com/dbt-labs/dbt/blob/1.10.latest/CHANGELOG.md#dbt-core-11019---january-20-2026) * [dbt-adapters 1.22.5](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/CHANGELOG.md#dbt-adapters-1225---january-14-2026) * [dbt-common 1.37.2](https://github.com/dbt-labs/dbt-common/blob/main/CHANGELOG.md#dbt-common-1372---december-15-2025) * [dbt-athena 1.10.0](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-athena/CHANGELOG.md#dbt-athena-1100---december-22-2025) * [dbt-bigquery 1.11.0](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-bigquery/CHANGELOG.md#dbt-bigquery-1110---december-22-2025) * [dbt-databricks 1.10.19](https://github.com/databricks/dbt-databricks/blob/1.10.latest/CHANGELOG.md#dbt-databricks-11019-jan-21-2026) * [dbt-fabric 1.9.4](https://github.com/microsoft/dbt-fabric/releases/tag/v1.9.4) * [dbt-postgres 1.10.0](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-postgres/CHANGELOG.md#dbt-postgres-1100---december-22-2025) * [dbt-redshift 1.10.0](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-redshift/CHANGELOG.md#dbt-redshift-1100---december-22-2025) * [dbt-snowflake 1.11.1](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-snowflake/CHANGELOG.md#dbt-snowflake-1111---january-08-2026) * [dbt-spark 1.9.3](https://github.com/dbt-labs/dbt-adapters/blob/stable/dbt-spark/CHANGELOG.md#dbt-spark-193---july-16-2025) * [dbt-synapse 1.8.4](https://github.com/microsoft/dbt-synapse/blob/v1.8.latest/CHANGELOG.md) * [dbt-teradata 1.10.1](https://github.com/Teradata/dbt-teradata/releases/tag/v1.10.1) * [dbt-trino 1.10.1](https://github.com/starburstdata/dbt-trino/blob/master/CHANGELOG.md#dbt-trino-1101---january-16-2026) #### December 2025 Release date: December 9, 2025 ##### dbt cloud-based platform This compatible release includes functionality from the following versions of dbt v1 OSS: ```text dbt-core==1.10.15 # shared interfaces dbt-adapters==1.16.7 dbt-common==1.33.0 dbt-semantic-interfaces==0.9.0 # adapters dbt-athena==1.9.5 dbt-bigquery==1.10.3 dbt-databricks==1.10.15 dbt-extractor==0.6.0 dbt-fabric==1.9.4 dbt-postgres==1.9.1 dbt-redshift==1.9.5 dbt-sl-sdk[sync]==0.13.0 dbt-snowflake==1.10.3 dbt-spark==1.9.3 dbt-synapse==1.8.4 dbt-teradata==1.10.0 dbt-trino==1.9.3 ``` Changelogs: * [dbt-core 1.10.15](https://github.com/dbt-labs/dbt/blob/1.10.latest/CHANGELOG.md#dbt-core-11015---november-12-2025) * [dbt-adapters 1.16.7](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/CHANGELOG.md#dbt-adapters-1166---september-03-2025) * [dbt-common 1.33.0](https://github.com/dbt-labs/dbt-common/blob/main/CHANGELOG.md#dbt-common-1330---october-20-2025) * [dbt-athena 1.9.5](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-athena/CHANGELOG.md#dbt-athena-194---april-28-2025) * [dbt-bigquery 1.10.3](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-bigquery/CHANGELOG.md#dbt-bigquery-1101---july-29-2025) * [dbt-databricks 1.10.15](https://github.com/databricks/dbt-databricks/blob/1.10.latest/CHANGELOG.md#dbt-databricks-11015-nov-17-2025) * [dbt-fabric 1.9.4](https://github.com/microsoft/dbt-fabric/releases/tag/v1.9.4) * [dbt-postgres 1.9.1](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-postgres/CHANGELOG.md#changelog) * [dbt-redshift 1.9.5](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-redshift/CHANGELOG.md#dbt-redshift-195---may-13-2025) * [dbt-snowflake 1.10.3](http://github.com/dbt-labs/dbt-adapters/blob/main/dbt-snowflake/CHANGELOG.md) * [dbt-spark 1.9.3](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-spark/CHANGELOG.md#dbt-spark-193---july-16-2025) * [dbt-synapse 1.8.4](https://github.com/microsoft/dbt-synapse/blob/v1.8.latest/CHANGELOG.md) * [dbt-teradata 1.10.0](https://github.com/Teradata/dbt-teradata/releases/tag/v1.10.0) * [dbt-trino 1.9.3](https://github.com/starburstdata/dbt-trino/blob/master/CHANGELOG.md#dbt-trino-193---july-22-2025) #### November 2025 Release date: November 11, 2025 ##### dbt cloud-based platform ##### Under the Hood * Record source column schemas when `DBT_RECORDER_MODE` is set * Issue additional column schema retrieval for hardcoded relation references in SQL * Make source schema recording cache thread-safe * Record column schemas for deferred relations and unselected dependencies This compatible release includes functionality from the following versions of dbt v1 OSS: ```text dbt-core==1.10.15 # shared interfaces dbt-adapters==1.16.7 dbt-common==1.33.0 dbt-semantic-interfaces==0.9.0 # adapters dbt-athena==1.9.5 dbt-bigquery==1.10.3 dbt-databricks==1.10.15 dbt-extractor==0.6.0 dbt-fabric==1.9.4 dbt-postgres==1.9.1 dbt-redshift==1.9.5 dbt-sl-sdk[sync]==0.13.0 dbt-snowflake==1.10.3 dbt-spark==1.9.3 dbt-synapse==1.8.4 dbt-teradata==1.10.0 dbt-trino==1.9.3 ``` Changelogs: * [dbt-core 1.10.15](https://github.com/dbt-labs/dbt/blob/1.10.latest/CHANGELOG.md#dbt-core-11015---november-12-2025) * [dbt-adapters 1.16.7](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/CHANGELOG.md#dbt-adapters-1166---september-03-2025) * [dbt-common 1.33.0](https://github.com/dbt-labs/dbt-common/blob/main/CHANGELOG.md#dbt-common-1330---october-20-2025) * [dbt-athena 1.9.5](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-athena/CHANGELOG.md#dbt-athena-194---april-28-2025) * [dbt-bigquery 1.10.3](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-bigquery/CHANGELOG.md#dbt-bigquery-1101---july-29-2025) * [dbt-databricks 1.10.15](https://github.com/databricks/dbt-databricks/blob/1.10.latest/CHANGELOG.md#dbt-databricks-11015-nov-17-2025) * [dbt-fabric 1.9.4](https://github.com/microsoft/dbt-fabric/releases/tag/v1.9.4) * [dbt-postgres 1.9.1](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-postgres/CHANGELOG.md#changelog) * [dbt-redshift 1.9.5](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-redshift/CHANGELOG.md#dbt-redshift-195---may-13-2025) * [dbt-snowflake 1.10.3](http://github.com/dbt-labs/dbt-adapters/blob/main/dbt-snowflake/CHANGELOG.md) * [dbt-spark 1.9.3](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-spark/CHANGELOG.md#dbt-spark-193---july-16-2025) * [dbt-synapse 1.8.4](https://github.com/microsoft/dbt-synapse/blob/v1.8.latest/CHANGELOG.md) * [dbt-teradata 1.10.0](https://github.com/Teradata/dbt-teradata/releases/tag/v1.10.0) * [dbt-trino 1.9.3](https://github.com/starburstdata/dbt-trino/blob/master/CHANGELOG.md#dbt-trino-193---july-22-2025) #### October 2025 Release date: October 23, 2025 ##### dbt cloud-based platform ##### Under the Hood * Add instrumentation to adapter methods for reliable debugging traces at the adapter boundary This compatible release includes functionality from the following versions of dbt v1 OSS: ```text dbt-core==1.10.13 # shared interfaces dbt-adapters==1.16.7 dbt-common==1.33.0 dbt-semantic-interfaces==0.9.0 # adapters dbt-athena==1.9.5 dbt-bigquery==1.10.2 dbt-databricks==1.10.14 dbt-extractor==0.6.0 dbt-fabric==1.9.4 dbt-postgres==1.9.1 dbt-redshift==1.9.5 dbt-sl-sdk[sync]==0.13.0 dbt-snowflake==1.10.2 dbt-spark==1.9.3 dbt-synapse==1.8.4 dbt-teradata==1.10.0 dbt-trino==1.9.3 ``` Changelogs: * [dbt-core 1.10.13](https://github.com/dbt-labs/dbt/blob/1.10.latest/CHANGELOG.md#dbt-core-11013---september-25-2025) * [dbt-adapters 1.16.7](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/CHANGELOG.md#dbt-adapters-1166---september-03-2025) * [dbt-common 1.33.0](https://github.com/dbt-labs/dbt-common/blob/main/CHANGELOG.md#dbt-common-1330---october-20-2025) * [dbt-athena 1.9.5](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-athena/CHANGELOG.md#dbt-athena-194---april-28-2025) * [dbt-bigquery 1.10.2](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-bigquery/CHANGELOG.md#dbt-bigquery-1101---july-29-2025) * [dbt-databricks 1.10.14](https://github.com/databricks/dbt-databricks/blob/main/CHANGELOG.md#dbt-databricks-11014-october-22-2025) * [dbt-fabric 1.9.4](https://github.com/microsoft/dbt-fabric/releases/tag/v1.9.4) * [dbt-postgres 1.9.1](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-postgres/CHANGELOG.md#changelog) * [dbt-redshift 1.9.5](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-redshift/CHANGELOG.md#dbt-redshift-195---may-13-2025) * [dbt-snowflake 1.10.2](http://github.com/dbt-labs/dbt-adapters/blob/main/dbt-snowflake/CHANGELOG.md) * [dbt-spark 1.9.3](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-spark/CHANGELOG.md#dbt-spark-193---july-16-2025) * [dbt-synapse 1.8.4](https://github.com/microsoft/dbt-synapse/blob/v1.8.latest/CHANGELOG.md) * [dbt-teradata 1.10.0](https://github.com/Teradata/dbt-teradata/releases/tag/v1.10.0) * [dbt-trino 1.9.3](https://github.com/starburstdata/dbt-trino/blob/master/CHANGELOG.md#dbt-trino-193---july-22-2025) #### September 2025 Release date: September 10, 2025 This compatible release includes functionality from the following versions of dbt v1 OSS: ```text dbt-core==1.10.11 # shared interfaces dbt-adapters==1.16.6 dbt-common==1.29.0 dbt-semantic-interfaces==0.9.0 # adapters dbt-athena==1.9.5 dbt-bigquery==1.10.2 dbt-databricks==1.10.12 dbt-extractor==0.6.0 dbt-fabric==1.9.4 dbt-postgres==1.9.1 dbt-protos==1.0.348 dbt-redshift==1.9.5 dbt-sl-sdk[sync]==0.13.0 dbt-snowflake==1.10.2 dbt-spark==1.9.3 dbt-synapse==1.8.4 dbt-teradata==1.10.0 dbt-trino==1.9.3 ``` Changelogs: * [dbt-core 1.10.11](https://github.com/dbt-labs/dbt/blob/1.10.latest/CHANGELOG.md#dbt-core-11011---september-04-2025) * [dbt-adapters 1.16.6](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/CHANGELOG.md#dbt-adapters-1166---september-03-2025) * [dbt-common 1.29.0](https://github.com/dbt-labs/dbt-common/blob/main/CHANGELOG.md#dbt-common-1290---september-04-2025) * [dbt-athena 1.9.4](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-athena/CHANGELOG.md#dbt-athena-194---april-28-2025) * [dbt-bigquery 1.10.2](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-bigquery/CHANGELOG.md#dbt-bigquery-1101---july-29-2025) * [dbt-databricks 1.10.12](https://github.com/databricks/dbt-databricks/blob/main/CHANGELOG.md#dbt-databricks-11012-september-8-2025) * [dbt-fabric 1.9.4](https://github.com/microsoft/dbt-fabric/releases/tag/v1.9.4) * [dbt-postgres 1.9.1](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-postgres/CHANGELOG.md#changelog) * [dbt-redshift 1.9.5](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-redshift/CHANGELOG.md#dbt-redshift-195---may-13-2025) * [dbt-snowflake 1.10.2](http://github.com/dbt-labs/dbt-adapters/blob/main/dbt-snowflake/CHANGELOG.md) * [dbt-spark 1.9.3](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-spark/CHANGELOG.md#dbt-spark-193---july-16-2025) * [dbt-synapse 1.8.4](https://github.com/microsoft/dbt-synapse/blob/v1.8.latest/CHANGELOG.md) * [dbt-teradata 1.10.0](https://github.com/Teradata/dbt-teradata/releases/tag/v1.10.0) * [dbt-trino 1.9.3](https://github.com/starburstdata/dbt-trino/blob/master/CHANGELOG.md#dbt-trino-193---july-22-2025) #### August 2025 Release date: August 12, 2025 ##### Notable dbt v1 OSS changes This compatible upgrade brings in a minor update to `dbt-core`, from `dbt-core==1.9.8` to `dbt-core==1.10.8`. Some noteworthy changes from this minor version include: * Introduction of several new [deprecations](https://docs.getdbt.com/reference/deprecations.md) that warn about project incompatibilities between dbt v1 and v2. * Support for defining `meta` and `tags` within `config` of columns and exposures, as well as defining `freshness` within `config` of sources. These changes lead to manifest.json minor schema evolutions which may cause an intermittent increase in false positives during `state:modified` comparisons. ##### dbt cloud-based platform ##### Fixes * Update generate publications script to add project and env id in generated publication file * Use JSON stream for publication artifact generation script * Get environment variables correctly from environment for publication artifacts * Adding `--resource-type` and `--exclude-resource-type` flags to Semantic Layer commands * Azure DevOps Private Packages are now properly matched with Private Package Definition in packages.yml ##### Under the Hood * Prepare support for Private Package's URLs with multiple levels * Disable telemetry client logger * Update semantic layer SDK to 0.11 This release includes functionality from the following versions of dbt v1 OSS: ```text dbt-core==1.10.8 # shared interfaces dbt-adapters==1.16.3 dbt-common==1.27.1 dbt-semantic-interfaces==0.9.0 dbt-extractor==0.6.0 dbt-protos==1.0.348 # dbt-adapters dbt-athena==1.9.4 dbt-bigquery==1.10.1 dbt-databricks==1.10.10 dbt-fabric==1.9.4 dbt-postgres==1.9.0 dbt-redshift==1.9.5 dbt-snowflake==1.10.0 dbt-spark==1.9.3 dbt-synapse==1.8.2 dbt-teradata==1.9.3 dbt-trino==1.9.3 ``` Changelogs: * [dbt-core 1.10.8](https://github.com/dbt-labs/dbt/blob/1.10.latest/CHANGELOG.md#dbt-core-1108---august-12-2025) * [dbt-adapters 1.16.3](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/CHANGELOG.md#dbt-adapters-1163---july-21-2025) * [dbt-common 1.25.0](https://github.com/dbt-labs/dbt-common/blob/main/CHANGELOG.md#dbt-common-1271---july-21-2025) * [dbt-athena 1.9.4](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-athena/CHANGELOG.md#dbt-athena-194---april-28-2025) * [dbt-bigquery 1.10.1](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-bigquery/CHANGELOG.md#dbt-bigquery-1101---july-29-2025) * [dbt-databricks 1.9.7](https://github.com/databricks/dbt-databricks/blob/main/CHANGELOG.md#dbt-databricks-1109-august-7-2025) * [dbt-fabric 1.9.4](https://github.com/microsoft/dbt-fabric/releases/tag/v1.9.4) * [dbt-postgres 1.9.0](https://github.com/dbt-labs/dbt-postgres/blob/main/CHANGELOG.md#dbt-postgres-190---december-09-2024) * [dbt-redshift 1.9.5](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-redshift/CHANGELOG.md#dbt-redshift-195---may-13-2025) * [dbt-snowflake 1.10.0](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-snowflake/CHANGELOG.md#dbt-snowflake-1100-rc3---june-24-2025) * [dbt-spark 1.9.3](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-spark/CHANGELOG.md#dbt-spark-193---july-16-2025) * [dbt-synapse 1.8.2](https://github.com/microsoft/dbt-synapse/blob/v1.8.latest/CHANGELOG.md) * [dbt-teradata 1.9.3](https://github.com/Teradata/dbt-teradata/releases/tag/v1.9.3) * [dbt-trino 1.9.3](https://github.com/starburstdata/dbt-trino/blob/master/CHANGELOG.md#dbt-trino-193---july-22-2025) #### July 2025 The compatible release slated for July 2025 will be skipped in order to further stabilize the minor upgrade of `dbt-core==1.10.0` ([released June 16, 2025](https://pypi.org/project/dbt-core/1.10.0/)) across the dbt platform. Compatible releases will resume in August 2025. #### June 2025 Release date: June 12, 2025 This release includes functionality from the following versions of dbt v1 OSS: ```text dbt-core==1.9.8 # shared interfaces dbt-adapters==1.15.3 dbt-common==1.25.0 dbt-semantic-interfaces==0.7.4 # adapters dbt-athena==1.9.4 dbt-bigquery==1.9.1 dbt-databricks==1.9.7 dbt-extractor==0.6.0 dbt-fabric==1.9.4 dbt-postgres==1.9.0 dbt-protos==1.0.317 dbt-redshift==1.9.5 dbt-sl-sdk-internal[sync]==0.7.0 dbt-sl-sdk[sync]==0.7.0 dbt-snowflake==1.9.4 dbt-spark==1.9.2 dbt-synapse==1.8.2 dbt-teradata==1.9.2 dbt-trino==1.9.2 ``` Changelogs: * [dbt-core 1.9.8](https://github.com/dbt-labs/dbt/blob/1.9.latest/CHANGELOG.md#dbt-core-198---june-10-2025) * [dbt-adapters 1.15.3](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/CHANGELOG.md#dbt-adapters-1153---may-20-2025) * [dbt-common 1.25.0](https://github.com/dbt-labs/dbt-common/blob/main/CHANGELOG.md#dbt-common-1250---may-20-2025) * [dbt-athena 1.9.4](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-athena/CHANGELOG.md#dbt-athena-194---april-28-2025) * [dbt-bigquery 1.9.1](https://github.com/dbt-labs/dbt-bigquery/blob/1.9.latest/CHANGELOG.md#dbt-bigquery-191---january-10-2025) * [dbt-databricks 1.9.7](https://github.com/databricks/dbt-databricks/blob/main/CHANGELOG.md#dbt-databricks-197-feb-25-2025) * [dbt-fabric 1.9.4](https://github.com/microsoft/dbt-fabric/releases/tag/v1.9.4) * [dbt-postgres 1.9.0](https://github.com/dbt-labs/dbt-postgres/blob/main/CHANGELOG.md#dbt-postgres-190---december-09-2024) * [dbt-redshift 1.9.5](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-redshift/CHANGELOG.md#dbt-redshift-195---may-13-2025) * [dbt-snowflake 1.9.4](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-snowflake/CHANGELOG.md#dbt-snowflake-194---may-02-2025) * [dbt-spark 1.9.2](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-spark/CHANGELOG.md#dbt-spark-192---march-07-2025) * [dbt-synapse 1.8.2](https://github.com/microsoft/dbt-synapse/blob/v1.8.latest/CHANGELOG.md) * [dbt-teradata 1.9.2](https://github.com/Teradata/dbt-teradata/releases/tag/v1.9.2) * [dbt-trino 1.9.1](https://github.com/starburstdata/dbt-trino/blob/master/CHANGELOG.md#dbt-trino-192---june-03-2025) #### May 2025 Release date: May 19, 2025 ##### dbt cloud-based platform These changes reflect capabilities that are only available in the dbt platform. ##### Fixes * Get environment variables correctly from the environment for publication artifacts ##### Under the hood * Create JSON schemas for PublicationArtifact and ResolvedProjectsArtifact This release includes functionality from the following versions of dbt v1 OSS: ```text dbt-core==1.9.4 # shared interfaces dbt-adapters==1.14.8 dbt-common==1.24.0 dbt-semantic-interfaces==0.7.4 # adapters dbt-athena==1.9.4 dbt-bigquery==1.9.1 dbt-databricks==1.9.7 dbt-fabric==1.9.4 dbt-postgres==1.9.0 dbt-redshift==1.9.5 dbt-snowflake==1.9.4 dbt-spark==1.9.2 dbt-synapse==1.8.2 dbt-teradata==1.9.2 dbt-trino==1.9.1 ``` Changelogs: * [dbt-core 1.9.4](https://github.com/dbt-labs/dbt/blob/1.9.latest/CHANGELOG.md#dbt-core-194---april-02-2025) * [dbt-adapters 1.14.8](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/CHANGELOG.md#dbt-adapters-1148---april-25-2025) * [dbt-common 1.24.0](https://github.com/dbt-labs/dbt-common/blob/main/CHANGELOG.md#dbt-common-1240---may-09-2025) * [dbt-athena 1.9.4](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-athena/CHANGELOG.md#dbt-athena-194---april-28-2025) * [dbt-bigquery 1.9.1](https://github.com/dbt-labs/dbt-bigquery/blob/1.9.latest/CHANGELOG.md#dbt-bigquery-191---january-10-2025) * [dbt-databricks 1.9.7](https://github.com/databricks/dbt-databricks/blob/main/CHANGELOG.md#dbt-databricks-197-feb-25-2025) * [dbt-fabric 1.9.4](https://github.com/microsoft/dbt-fabric/releases/tag/v1.9.4) * [dbt-postgres 1.9.0](https://github.com/dbt-labs/dbt-postgres/blob/main/CHANGELOG.md#dbt-postgres-190---december-09-2024) * [dbt-redshift 1.9.5](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-redshift/CHANGELOG.md#dbt-redshift-195---may-13-2025) * [dbt-snowflake 1.9.4](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-snowflake/CHANGELOG.md#dbt-snowflake-194---may-02-2025) * [dbt-spark 1.9.2](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-spark/CHANGELOG.md#dbt-spark-192---march-07-2025) * [dbt-synapse 1.8.2](https://github.com/microsoft/dbt-synapse/blob/v1.8.latest/CHANGELOG.md) * [dbt-teradata 1.9.2](https://github.com/Teradata/dbt-teradata/releases/tag/v1.9.2) * [dbt-trino 1.9.1](https://github.com/starburstdata/dbt-trino/blob/master/CHANGELOG.md#dbt-trino-191---march-26-2025) #### April 2025 Release date: April 9, 2025 ##### dbt Cloud These changes reflect capabilities that are only available in dbt Cloud. ##### Under the Hood * Add secondary profiles to profile.py This release includes functionality from the following versions of dbt v1 OSS: ```text dbt-core==1.9.4 # shared interfaces dbt-adapters==1.14.5 dbt-common==1.17.0 dbt-semantic-interfaces==0.7.4 # adapters dbt-athena==1.9.3 dbt-bigquery==1.9.1 dbt-databricks==1.9.7 dbt-fabric==1.9.4 dbt-postgres==1.9.0 dbt-redshift==1.9.3 dbt-snowflake==1.9.2 dbt-spark==1.9.2 dbt-synapse==1.8.2 dbt-teradata==1.9.2 dbt-trino==1.9.1 ``` Changelogs: * [dbt-core 1.9.4](https://github.com/dbt-labs/dbt/blob/1.9.latest/CHANGELOG.md#dbt-core-194---april-02-2025) * [dbt-adapters 1.14.5](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/CHANGELOG.md#dbt-adapters-1145---april-07-2025) * [dbt-common 1.17.0](https://github.com/dbt-labs/dbt-common/blob/main/CHANGELOG.md#dbt-common-1170---march-31-2025) * [dbt-athena 1.9.3](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-athena/CHANGELOG.md#dbt-athena-193---april-07-2025) * [dbt-bigquery 1.9.1](https://github.com/dbt-labs/dbt-bigquery/blob/1.9.latest/CHANGELOG.md#dbt-bigquery-191---january-10-2025) * [dbt-databricks 1.9.7](https://github.com/databricks/dbt-databricks/blob/main/CHANGELOG.md#dbt-databricks-197-feb-25-2025) * [dbt-fabric 1.9.4](https://github.com/microsoft/dbt-fabric/releases/tag/v1.9.4) * [dbt-postgres 1.9.0](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-postgres/CHANGELOG.md#dbt-postgres-190---december-09-2024) * [dbt-redshift 1.9.3](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-redshift/CHANGELOG.md#dbt-redshift-193---april-01-2025) * [dbt-snowflake 1.9.2](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-snowflake/CHANGELOG.md#dbt-snowflake-192---march-07-2025) * [dbt-spark 1.9.2](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-spark/CHANGELOG.md#dbt-spark-192---march-07-2025) * [dbt-synapse 1.8.2](https://github.com/microsoft/dbt-synapse/blob/v1.8.latest/CHANGELOG.md) * [dbt-teradata 1.9.2](https://github.com/Teradata/dbt-teradata/releases/tag/v1.9.2) * [dbt-trino 1.9.1](https://github.com/starburstdata/dbt-trino/blob/master/CHANGELOG.md#dbt-trino-191---march-26-2025) #### March 2025 Release date: March 11, 2025 This release includes functionality from the following versions of dbt v1 OSS: ```text dbt-core==1.9.3 # shared interfaces dbt-adapters==1.14.1 dbt-common==1.15.0 dbt-semantic-interfaces==0.7.4 # adapters dbt-athena==1.9.2 dbt-bigquery==1.9.1 dbt-databricks==1.9.7 dbt-fabric==1.9.2 dbt-postgres==1.9.0 dbt-redshift==1.9.1 dbt-snowflake==1.9.2 dbt-spark==1.9.2 dbt-synapse==1.8.2 dbt-teradata==1.9.1 dbt-trino==1.9.0 ``` Changelogs: * [dbt 1.9.3](https://github.com/dbt-labs/dbt/blob/1.9.latest/CHANGELOG.md#dbt-core-193---march-07-2025) * [dbt-adapters 1.14.1](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/CHANGELOG.md#dbt-adapters-1141---march-04-2025) * [dbt-common 1.15.0](https://github.com/dbt-labs/dbt-common/blob/main/CHANGELOG.md#dbt-common-1150---february-14-2025) * [dbt-bigquery 1.9.1](https://github.com/dbt-labs/dbt-bigquery/blob/1.9.latest/CHANGELOG.md#dbt-bigquery-191---january-10-2025) * [dbt-databricks 1.9.7](https://github.com/databricks/dbt-databricks/blob/main/CHANGELOG.md#dbt-databricks-197-feb-25-2025) * [dbt-fabric 1.9.2](https://github.com/microsoft/dbt-fabric/releases/tag/v1.9.2) * [dbt-postgres 1.9.0](https://github.com/dbt-labs/dbt-postgres/blob/main/CHANGELOG.md#dbt-postgres-190---december-09-2024) * [dbt-redshift 1.9.1](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-redshift/CHANGELOG.md#dbt-redshift-191---march-07-2025) * [dbt-snowflake 1.9.2](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-snowflake/CHANGELOG.md#dbt-snowflake-192---march-07-2025) * [dbt-spark 1.9.2](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-spark/CHANGELOG.md#dbt-spark-192---march-07-2025) * [dbt-synapse 1.8.2](https://github.com/microsoft/dbt-synapse/blob/v1.8.latest/CHANGELOG.md) * [dbt-teradata 1.9.1](https://github.com/Teradata/dbt-teradata/releases/tag/v1.9.1) * [dbt-trino 1.9.0](https://github.com/starburstdata/dbt-trino/blob/master/CHANGELOG.md#dbt-trino-190---december-20-2024) #### February 2025 Release date: February 12, 2025 ##### dbt Cloud These changes reflect capabilities that are only available in dbt. ##### Features * Add [`event_time`](https://docs.getdbt.com/reference/resource-configs/event-time.md) to cross-project ref artifact. * Include debug exception message in ObservabilityMetric. ##### Fixes * Adding support for deferral against the new time spine definition. * Fix error messages for SL query. * Semantic Layer commands now respect `--favor-state` when running with `--defer`. This release includes functionality from the following versions of dbt v1 OSS: ```text dbt-core==1.9.2 # shared interfaces dbt-adapters==1.14.0 dbt-common==1.14.0 dbt-semantic-interfaces==0.7.4 # adapters dbt-athena==1.9.1 dbt-bigquery==1.9.1 dbt-databricks==1.9.4 dbt-fabric==1.9.0 dbt-postgres==1.9.0 dbt-redshift==1.9.0 dbt-snowflake==1.9.1 dbt-spark==1.9.1 dbt-synapse==1.8.2 dbt-teradata==1.9.1 dbt-trino==1.9.0 ``` Changelogs: * [dbt 1.9.2](https://github.com/dbt-labs/dbt/blob/1.9.latest/CHANGELOG.md#dbt-core-192---january-29-2025) * [dbt-adapters 1.14.0](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/CHANGELOG.md#dbt-adapters-1140---february-07-2025) * [dbt-common 1.14.0](https://github.com/dbt-labs/dbt-common/blob/main/CHANGELOG.md) * [dbt-bigquery 1.9.1](https://github.com/dbt-labs/dbt-bigquery/blob/1.9.latest/CHANGELOG.md#dbt-bigquery-191---january-10-2025) * [dbt-databricks 1.9.4](https://github.com/databricks/dbt-databricks/blob/main/CHANGELOG.md#dbt-databricks-194-jan-30-2024) * [dbt-fabric 1.9.0](https://github.com/microsoft/dbt-fabric/releases/tag/v1.9.0) * [dbt-postgres 1.9.0](https://github.com/dbt-labs/dbt-postgres/blob/main/CHANGELOG.md#dbt-postgres-190---december-09-2024) * [dbt-redshift 1.9.0](https://github.com/dbt-labs/dbt-redshift/blob/1.9.latest/CHANGELOG.md#dbt-redshift-190---december-09-2024) * [dbt-snowflake 1.9.1](https://github.com/dbt-labs/dbt-snowflake/blob/1.9.latest/CHANGELOG.md#dbt-snowflake-191---february-07-2025) * [dbt-spark 1.9.1](https://github.com/dbt-labs/dbt-spark/blob/1.9.latest/CHANGELOG.md#dbt-spark-191---february-07-2025) * [dbt-synapse 1.8.2](https://github.com/microsoft/dbt-synapse/blob/v1.8.latest/CHANGELOG.md) * [dbt-teradata 1.9.1](https://github.com/Teradata/dbt-teradata/releases/tag/v1.9.1) * [dbt-trino 1.9.0](https://github.com/starburstdata/dbt-trino/blob/master/CHANGELOG.md#dbt-trino-190---december-20-2024) #### January 2025 Release date: January 14, 2025 ##### dbt Cloud These changes reflect capabilities that are only available in dbt. ##### Features * Filter out external exposures in dbt compare. ##### Fixes * Use `meta.dbt_cloud_id` to `build unique_id` for manually defined exposure for merging against a duplicated exposure. This release includes functionality from the following versions of dbt v1 OSS: ```text dbt-core==1.9.1 # shared interfaces dbt-adapters==1.13.1 dbt-common==1.14.0 dbt-semantic-interfaces==0.7.4 # adapters dbt-athena==1.9.0 dbt-bigquery==1.9.1 dbt-databricks==1.9.1 dbt-fabric==1.9.0 dbt-postgres==1.9.0 dbt-redshift==1.9.0 dbt-snowflake==1.9.0 dbt-spark==1.9.0 dbt-synapse==1.8.2 dbt-teradata==1.9.0 dbt-trino==1.9.0 ``` Changelogs: * [dbt 1.9.1](https://github.com/dbt-labs/dbt/blob/1.9.latest/CHANGELOG.md#dbt-core-191---december-16-2024) * [dbt-adapters 1.13.1](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/CHANGELOG.md#dbt-adapters-1131---january-10-2025) * [dbt-common 1.14.0](https://github.com/dbt-labs/dbt-common/blob/main/CHANGELOG.md) * [dbt-bigquery 1.9.1](https://github.com/dbt-labs/dbt-bigquery/blob/1.9.latest/CHANGELOG.md#dbt-bigquery-191---january-10-2025) * [dbt-databricks 1.9.1](https://github.com/databricks/dbt-databricks/blob/main/CHANGELOG.md#dbt-databricks-191-december-16-2024) * [dbt-fabric 1.9.0](https://github.com/microsoft/dbt-fabric/releases/tag/v1.9.0) * [dbt-postgres 1.9.0](https://github.com/dbt-labs/dbt-postgres/blob/main/CHANGELOG.md#dbt-postgres-190---december-09-2024) * [dbt-redshift 1.9.0](https://github.com/dbt-labs/dbt-redshift/blob/1.9.latest/CHANGELOG.md#dbt-redshift-190---december-09-2024) * [dbt-snowflake 1.9.0](https://github.com/dbt-labs/dbt-snowflake/blob/1.9.latest/CHANGELOG.md#dbt-snowflake-190---december-09-2024) * [dbt-spark 1.9.0](https://github.com/dbt-labs/dbt-spark/blob/1.9.latest/CHANGELOG.md#dbt-spark-190---december-10-2024) * [dbt-synapse 1.8.2](https://github.com/microsoft/dbt-synapse/blob/v1.8.latest/CHANGELOG.md) * [dbt-teradata 1.9.0](https://github.com/Teradata/dbt-teradata/releases/tag/v1.9.0) * [dbt-trino 1.9.0](https://github.com/starburstdata/dbt-trino/blob/master/CHANGELOG.md#dbt-trino-190---december-20-2024) #### December 2024 Release date: December 12, 2024 This release includes functionality from the following versions of dbt v1 OSS: ```text dbt-core==1.9.0 # shared interfaces dbt-adapters==1.10.4 dbt-common==1.14.0 dbt-semantic-interfaces==0.7.4 # adapters dbt-athena==1.9.0 dbt-bigquery==1.9.0 dbt-databricks==1.9.0 dbt-fabric==1.8.8 dbt-postgres==1.9.0 dbt-redshift==1.9.0 dbt-snowflake==1.9.0 dbt-spark==1.9.0 dbt-synapse==1.8.2 dbt-teradata==1.8.2 dbt-trino==1.8.5 ``` Changelogs: * [dbt 1.9.0](https://github.com/dbt-labs/dbt/blob/1.9.latest/CHANGELOG.md#dbt-core-190---december-09-2024) * [dbt-adapters 1.10.4](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-adapters/CHANGELOG.md#dbt-adapters-1104---november-11-2024) * [dbt-common 1.14.0](https://github.com/dbt-labs/dbt-common/blob/main/CHANGELOG.md) * [dbt-bigquery 1.9.0](https://github.com/dbt-labs/dbt-bigquery/blob/1.9.latest/CHANGELOG.md#dbt-bigquery-190---december-09-2024) * [dbt-databricks 1.9.0](https://github.com/databricks/dbt-databricks/blob/main/CHANGELOG.md#dbt-databricks-190-december-9-2024) * [dbt-fabric 1.8.8](https://github.com/microsoft/dbt-fabric/blob/v1.8.latest/CHANGELOG.md) * [dbt-postgres 1.9.0](https://github.com/dbt-labs/dbt-adapters/blob/main/dbt-postgres/CHANGELOG.md#dbt-postgres-190---december-09-2024) * [dbt-redshift 1.9.0](https://github.com/dbt-labs/dbt-redshift/blob/1.9.latest/CHANGELOG.md#dbt-redshift-190---december-09-2024) * [dbt-snowflake 1.9.0](https://github.com/dbt-labs/dbt-snowflake/blob/1.9.latest/CHANGELOG.md#dbt-snowflake-190---december-09-2024) * [dbt-spark 1.9.0](https://github.com/dbt-labs/dbt-spark/blob/1.9.latest/CHANGELOG.md#dbt-spark-190---december-10-2024) * [dbt-synapse 1.8.2](https://github.com/microsoft/dbt-synapse/blob/v1.8.latest/CHANGELOG.md) * [dbt-teradata 1.8.2](https://github.com/Teradata/dbt-teradata/releases/tag/v1.8.2) * [dbt-trino 1.8.5](https://github.com/starburstdata/dbt-trino/blob/master/CHANGELOG.md#dbt-trino-185---december-11-2024) --- ### dbt release notes dbt platform dbt release notes for recent and historical changes. Release notes fall into one of the following categories: * **New:** New products and features * **Enhancement:** Performance improvements and feature enhancements * **Fix:** Bug and security fixes * **Behavior change:** A change to existing behavior that doesn't fit into the other categories, such as feature deprecations or changes to default settings Release notes are grouped by month for both multi-tenant and virtual private cloud (VPC) environments. ![RSS](/img/fontawesome/rss.svg)Subscribe to release note updates via [RSS](https://docs.getdbt.com/assets/files/release-notes-rss-49475018c79805e7351fe5b3f5227b83.xml), [Atom](https://docs.getdbt.com/assets/files/release-notes-atom-78c7d9ff5c18eef0b3cece35aabf68cb.xml), or [JSON Feed](https://docs.getdbt.com/assets/files/release-notes-rss-15a44a9349c64f09e5c6a2e02952c701.json). For dbt v2 updates, refer to the [v2 changelog](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md). #### September 2026 * **New:** The **dbt State** page now includes a [**Lag tolerance recommendations**](https://docs.getdbt.com/docs/deploy/dbt-state-interface.md#lag-tolerance-recommendations) section that identifies models that could safely tolerate more lag. For each model, it shows the current lag tolerance, the recommended value, the estimated percentage of build time you'd save, and projected build time savings over the next 30 days. You can search by model name or filter by project, and apply recommendations by updating the [`lag_tolerance`](https://docs.getdbt.com/reference/resource-configs/lag-tolerance.md) config. * **New:** The Snowflake adapter now supports the `interactive_table` materialization in beta on dbt v2 (dbt-snowflake v1.13+), covering both static and dynamic (auto-refreshing) interactive tables. For more information, see [Interactive tables](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md). * **Enhancement:** The [Analyst Read](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#analyst-read) permission set is now available to all accounts without requiring a feature flag. You can assign it to groups so read-only users can view Catalog and project configuration such as connections, environments, and Semantic Layer settings. * **Enhancement:** When you connect to Snowflake through the Semantic Layer, authentication failures and permission errors now return distinct messages prefixed with `[WAREHOUSE_AUTHENTICATION_FAILED]` or `[WAREHOUSE_PERMISSION_DENIED]`, so you can tell credential issues apart from missing grants. * **Fix:** Saving a job with an invalid day-of-month value such as `*,L` now shows a validation error and prevents the broken schedule from being saved. Use either `*` or `L` in the day-of-month field, not both. * **Behavior change:** The **Fusion admin** permission set is now called [**v2 Migration Admin**](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#v2-migration-admin). The permissions it grants haven't changed, and existing assignments carry over. * **Behavior change:** When you assign a project-scoped permission set to a group or service token, you must now specify either all projects or a specific project. Requests that leave project scope unset return a `400` error. Account-level permission sets such as Billing Admin and Notification Manager are not affected. Existing legacy assignments continue to work until you change them. * **Preview:** Explore mode is now available in the dbt Wizard [home tab](https://docs.getdbt.com/docs/platform/wizard-home.md#ask-questions-in-explore-mode) and [Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md#ask-questions-in-explore-mode). Explore mode lets users ask questions of governed production data in plain language, with the SQL or metric definition behind every answer. Read-only users can now be [invited](https://docs.getdbt.com/docs/platform/wizard-read-only-users.md) to ask questions about your data without a developer license in the Wizard home tab. * **New:** Jobs with [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) enabled now include an **Explain** tab on the run details page. This tab shows why dbt State rebuilt, reused, or cloned each resource in the run, so you can audit State behavior and debug unexpected decisions directly on the dbt platform. Refer to [Monitor dbt State activity](https://docs.getdbt.com/docs/deploy/dbt-state-interface.md#explain-tab) for more information. * **Behavior change:** The following behavior change flags on the dbt platform **Latest** release track have reached maturity, switching their default values from `false` to `true`. To preserve the previous behavior for any of these flags, set them to `false` in your `dbt_project.yml`. * [`skip_nodes_if_on_run_start_fails`](https://docs.getdbt.com/reference/global-configs/behavior-flags/skip_nodes_if_on_run_start_fails.md) * [`state_modified_compare_more_unrendered_values`](https://docs.getdbt.com/reference/global-configs/behavior-flags/state_modified_compare_more_unrendered_values.md) * [`require_yaml_configuration_for_mf_time_spines`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_yaml_configuration_for_mf_time_spines.md) * [`require_batched_execution_for_custom_microbatch_strategy`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_batched_execution_for_custom_microbatch_strategy.md) * [`require_nested_cumulative_type_params`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_nested_cumulative_type_params.md) * [`validate_macro_args`](https://docs.getdbt.com/reference/global-configs/behavior-flags/validate_macro_args.md) * [`require_all_warnings_handled_by_warn_error`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_all_warnings_handled_by_warn_error.md) * **Behavior change:** AI features in the dbt platform are being enabled by default. They're already on for new accounts and are rolling out soon to existing accounts. Organizations that opted out stay off, and admins can turn AI features on or off anytime in **Account settings**. Refer to [AI features in dbt platform](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md). * **New:** dbt Wizard now uses usage-based billing for dbt-managed AI. Usage is metered per token against your consumption pool, with monthly included usage or a trial to start and admin-set spend limits. Refer to [Trial and billing](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/trial-and-billing.md) and the [Billing and access FAQs](https://docs.getdbt.com/docs/dbt-ai/wizard-billing-faqs.md). * **New:** [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) onboarding and billing updates: * **New:** New users can now sign up for a free dbt platform account and start a 30-day dbt State trial through a guided onboarding flow. For details, refer to [Setting up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md?type=platform-signup). * **New:** dbt v1.7-1.12 users who are using the dbt State plugin can now create a free dbt platform account to start a 30-day dbt State trial. For details, refer to [Setting up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md). * **New:** dbt State billing is now available for self-serve accounts. You can set up usage-based billing in the **Usage-based features** dashboard. Note that you're billed per target table that dbt State reuses each day, not per dbt platform seat. Refer to [Setting up usage-based billing](https://docs.getdbt.com/docs/deploy/dbt-state-trial.md#setting-up-usage-based-billing) for more information. * **Behavior change**: The standalone dbt State app (`app.state.dbt.com`) is being retired and is no longer accepting new users. Existing dbt State app users should [create a free dbt platform account](http://us1.dbt.com/register?_dbtsrc=dbt-state) to continue using dbt State — your free trial will be extended upon account creation. #### August 2026 * **New:** A new "State explain" tab on the run details page shows dbt State's decision for each model in a run (rebuilt, reused, or cloned), with expandable details, search, and Comma-Separated Values (CSV) download. You can use this tab to investigate why each model was rebuilt or reused. Contact your account manager to enable. * **Enhancement:** Non-admin users now see an "Ask an admin to enable" message on the dbt Wizard and dbt State cards in Billing & Usage when a trial is available but they lack permission to start it, instead of a blank space. The same message appears if a non-admin tries to start a trial from a dbt State or dbt Wizard link. * **Enhancement:** dbt State is now available for jobs running on the Compatible, dbt v2 Extended, and dbt v2 Fallback release tracks, in addition to previously supported tracks. * **Enhancement:** When compare results are larger than 50 MB, pull request comments now show a "too large to summarize" notice with a link to the full compare report, instead of failing with no message. * **Enhancement:** The Studio IDE browser tab now displays "dbt Studio" instead of a generic editor title. * **Enhancement:** When you connect an MCP client with OAuth, more clients can now complete sign-in successfully. * **Enhancement:** [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) now reuses views that only use `select *` on CTEs. Previously, any `select *` anywhere in a view caused a rebuild. Views that use `select *` directly on a `ref()` or `source()` still force a rebuild, because dbt cannot safely determine the output columns at parse time. For more information, refer to [Views with `select *`](https://docs.getdbt.com/faqs/State/views-rebuilt.md#views-with-select). * **Enhancement:** dbt State now fetches table metadata in the background at the start of each run, so execution doesn't stall. Any node that is ready to skip or clone proceeds immediately, without waiting for the fetch to complete. Previously, dbt waited for the entire metadata fetch to complete before any node could execute. For more details, refer to [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md). * **Enhancement:** On Snowflake, when [`metadata_warehouse`](https://docs.getdbt.com/reference/resource-configs/metadata-warehouse.md) is configured, dbt State now issues multiple, individual queries (one per schema) in parallel on the dedicated warehouse — faster than the single, consolidated query dbt runs by default. Without a dedicated warehouse, dbt now emits a warning if the metadata fetch takes longer than 15 seconds. * **New:** The [`allow_clones`](https://docs.getdbt.com/reference/resource-configs/allow-clones.md) profile-level setting lets you control whether dbt State can clone tables into a target environment. Previously, there was no way to disable cloning — dbt State always cloned into any environment when a matching table was found. * **New**: [`compare_unrendered_code`](https://docs.getdbt.com/reference/resource-configs/compare-unrendered-code.md) is a new dbt State config that checks the Jinja template for unrendered code changes. If dbt detects unrendered code changes, it then compares the rendered SQL. A rebuild only occurs when *both* have changed. This prevents unnecessary rebuilds for nodes that use non-deterministic macros or environment variables. * **New:** When dbt State is enabled, you can run `dbt state explain` (dbt v2) or `dbt-state explain` (dbt v1 plugin) in the CLI after a job finishes to see why dbt State made each decision and whether each node was built, reused, or cloned. For a detailed breakdown, run the command with `--verbose -s my_node_name` to see the table analysis, query analysis, and data freshness analysis for a specific node. For more information, refer to [`dbt state explain`](https://docs.getdbt.com/reference/commands/state-explain.md). * **Enhancement:** New sessions open on the Wizard tab when available, and the Studio IDE remembers your last-used tab for each project so you can pick up where you left off. * **Enhancement:** A new `relationName` field on the `ModelAppliedStateNode` and `ModelAppliedStateNestedNode` GraphQL types exposes the fully-qualified, adapter-rendered relation name (for example, `"database"."schema"."model_name"`) from the last successful model build. * **New:** When dbt State is enabled, you can run `dbt state explain` (dbt v2) or `dbt-state explain` (dbt v1 plugin) in the CLI after a job finishes to see why dbt State made each decision and whether each node was built, reused, or cloned. For a detailed breakdown, run the command with `--verbose -s my_node_name` to see the table analysis, query analysis, and data freshness analysis for a specific node. For more information, refer to [`dbt state explain`](https://docs.getdbt.com/reference/commands/state-explain.md). * **Enhancement:** New sessions open on the Wizard tab when available, and the Studio IDE remembers your last-used tab for each project so you can pick up where you left off. * **Enhancement:** A new `relationName` field on the `ModelAppliedStateNode` and `ModelAppliedStateNestedNode` GraphQL types exposes the fully-qualified, adapter-rendered relation name (for example, `"database"."schema"."model_name"`) from the last successful model build. * **Beta**: [dbt Core 2.0](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md) is now available in beta! * **New:** The [Analyst read](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#analyst-read) permission set is now generally available (GA) for Enterprise plans. Analyst read is a project-level permission set that provides read-only access to analyze dbt models and project resources, and read-only users can connect to analysis features such as the [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md). * **Enhancement:** [Cost Insights](https://docs.getdbt.com/docs/explore/cost-insights.md) now supports cost attribution for [Snowflake Adaptive Warehouses](https://docs.snowflake.com/en/user-guide/warehouses-adaptive). For setup details, refer to [Assign required permissions](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md#assign-required-permissions). * **New:** The [Model timing tab](https://docs.getdbt.com/docs/deploy/run-visibility.md#model-timing-tab) in job run details has been redesigned with a richer, scalable view that includes metric tiles, an execution timeline with grouping and highlight controls, a concurrency-over-time chart, and a searchable resource details table. * **New:** Semantic Layer development connections to Redshift now support external OAuth using Okta or Microsoft Entra with AWS IAM Identity Center. * **Enhancement:** System for Cross-domain Identity Management (SCIM) API errors for seat or licensing failures now include email addresses so you can identify which users are blocking provisioning. * **Behavior change:** Semantic Layer GraphQL queries that exceed the complexity limit of 200,000 now return an error instead of completing with a warning. If you hit this error, request fewer fields, use pagination, narrow your filters, or split the query into smaller ones. * **Enhancement:** The [Analyst read](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#analyst-read) permission set now includes read access to **Connections** (account and project), **Projects**, Git repository settings, Semantic Layer configuration, **Environments**, custom environment variables, and Catalog metadata. Analysts can view that configuration without assigning additional permission sets, once added to a group with Analyst read. #### July 2026 * **Enhancement:** The [dbt State usage page](https://docs.getdbt.com/docs/deploy/dbt-state-interface.md) now shows daily active target tables (DATTs) split into **Billable** and **Free**. During a trial, all DATTs are counted as free. * **Preview**: [The dbt Wizard home tab in dbt platform](https://docs.getdbt.com/docs/platform/wizard-home.md) is now available in public preview. You can build and change dbt projects through natural language, with inline diffs, DAG previews, and validation built in. * **New:** The [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/mcp-available-tools.md#discovery) now uses one `get_node_details` tool for all resource types. The older type-specific tools are deprecated and will be removed in a future release. * **New:** When a job is deactivated, the banner now shows a specific reason (repeated run failures, account inactivity, or a generic fallback) with tailored reactivation instructions for each case. * **Enhancement:** The agent now automatically retries transient LLM provider failures (network timeouts, rate limits, and server errors) with exponential backoff, so brief provider blips are less likely to surface as errors during your session. * **Enhancement:** Client tool loops now run until the agent finishes rather than stopping after 50 iterations, eliminating premature termination of long-running agentic workflows. * **Enhancement:** Code blocks in dbt Wizard responses now include a Copy button on hover, so you can reuse generated SQL or YAML more easily. * **Enhancement:** When a Bring-Your-Own-Key (BYOK) OpenAI model is configured with a deployment that does not support embeddings (for example, a `gpt-4o` Azure deployment), the similar models feature now returns an actionable error message prompting you to use a text-embedding model instead of a generic internal error. * **Enhancement:** The `get_lineage` tool now accepts a `direction` parameter (`upstream`, `downstream`, or `both`) to narrow results to only ancestors or only descendants of a target node, reducing response size for large graphs. The response also now includes a `description` field on each returned node. * **Enhancement:** The `list_metrics` tool now accepts a `meta_filter` parameter to restrict results to metrics whose `config.meta` contains specified key-value pairs (for example, `{"agent_accessible": true}`), keeping result sets small enough to preserve description and metadata in the response. * **Enhancement:** The `ModelAppliedFilter` input type now includes a `health` field, letting you filter applied models by health status (`unknown`, `degraded`, `caution`, or `healthy`) directly in the Discovery API. * **Enhancement:** The `RunStatus` enum and the `lastRunStatus` field on model execution information now include `warn`, so models whose last run completed with warnings correctly reflect that status. * **Enhancement:** When searching from within a project environment route (for example, a Staging page), the Catalog search now defaults the environment filter to that environment type rather than always defaulting to Production. * **Enhancement:** A redesigned search result card replaces tooltip-based match pills with inline expandable snippets for columns, tags, descriptions, and code matches. Please contact your account manager to enable. * **Enhancement:** The `latest-fusion` release track is now dbt v2 Stable across all settings. Existing configurations have been updated automatically. No action is needed. * **Enhancement:** On the Enable dbt v2 Environments page, environments already running dbt v2 now show a disabled checkbox, preventing unnecessary saves. * **Enhancement:** When saving a dbt v2 upgrade fails, the platform now displays the top-level user message from the API instead of internal field-level error details. * **Enhancement:** The command panel now shows live status updates as commands run, so you see progress sooner without waiting for a refresh. * **Enhancement:** The "Committed spend" card is now labeled "Consumption pool" with copy explaining that usage-based features like dbt State draw from it. The card now appears between the current plan metric tiles and the product-specific sections on billing Overview and usage tab pages. * **Enhancement:** The Daily Active Target Tables (DATTs) chart now stacks billable and free series, so trial users whose usage is entirely free see real bars instead of an empty chart. * **Enhancement:** Memory-tuning optimizations are now applied automatically to all dbt v2 runs, reducing out-of-memory kill rates and improving overall uptime. * **Fix:** Claude-backed agents can return longer answers and handle some previously broken interactions more reliably. * **New**: [Apache Ossie](https://github.com/apache/ossie) semantic layer support: * Open Semantic Interchange (OSI) has been renamed to Apache Ossie. For more information, refer to [OSI is now Apache Ossie (Incubating)](https://www.getdbt.com/blog/osi-is-now-apache-ossie). * dbt writes an `osi_document.json` file to your `target/` directory alongside `semantic_manifest.json` at parse time. This artifact provides an Ossie representation of your project's Semantic Layer. For more information, refer to [Semantic manifest](https://docs.getdbt.com/reference/artifacts/sl-manifest.md#apache-ossie-document). * dbt supports the Ossie standard for defining semantic models and metrics. You can place Ossie-format `.json` files in an `osi/` directory at the root of your project, and dbt parses them into the manifest alongside any native dbt semantic models. To use a different directory, configure [`osi-paths`](https://docs.getdbt.com/reference/project-configs/osi-paths.md) in `dbt_project.yml`. Ossie versions `0.1.0` and `0.1.1` are supported; any other version raises a parse error. For more information, refer to [Ossie semantic layer documents](https://docs.getdbt.com/docs/build/ossie-semantic-models.md). * **New:** [Cost Insights](https://docs.getdbt.com/docs/explore/cost-insights.md) is now generally available (GA) for Snowflake, BigQuery, and Databricks. * **Preview:** [Cost Insights](https://docs.getdbt.com/docs/explore/cost-insights.md) for Amazon Redshift is now in preview. * **Enhancement:** Users with `user_credential_write` access can now view and manage their credentials without needing `credentials_read` privileges. This update reduces the need for additional, broader permissions when performing credential updates. * **Enhancement:** The [dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-platform.md) in dbt platform has a redesigned empty state with updated suggested prompts to help you discover different ways to get started. A new wayfinder bar keeps your current project and branch visible and highlights the next step as you move from asking questions to changing code and opening a pull request. * **Enhancement:** Catalog now supports a **Warn** last-run status. Resources whose last run completed with warnings show a distinct status and tooltip, and you can filter by **Warn** alongside other run statuses. * **New:** You can now create hybrid jobs to track runs triggered by an external orchestrator. Hybrid jobs have a simplified setup that omits execution steps, triggers, advanced settings, and cost-optimization controls. They display **Externally triggered** as their next-run schedule and are available only for projects configured as [Hybrid projects](https://docs.getdbt.com/docs/deploy/hybrid-projects.md). * **Enhancement:** Runs using a dbt v2 dbt version now invoke the built-in [`dbt lint`](https://docs.getdbt.com/reference/commands/lint.md?version=2.0) command instead of SQLFluff. dbt v2 virtual environments do not include SQLFluff, so linting now works for all Fusion-version runs and runs faster. * **Enhancement:** When the agent compresses conversation context in the background, a spinner labeled **Optimizing conversation context…** now appears in the chat area. Submitting new messages and stopping the agent are disabled while compaction is in progress to prevent conflicts. * **Enhancement:** When [dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-platform.md) is unavailable (not activated, trial expired, or spend limit reached), Studio IDE now shows a dedicated screen with the specific reason and an appropriate action instead of a generic message. * **Enhancement:** The users table, group member lists, and user edit drawer now search, filter, sort, and paginate server-side. On large accounts, all users are findable by name, email, or license type, group member search no longer misses results beyond the first page, and users beyond the first page can be opened and edited in the user edit drawer. * **Enhancement:** The **Enable global account discovery** setting on the **Account settings** page is now visible to all entitled accounts without requiring a feature flag. You can allow or restrict account discovery from [Account settings](https://docs.getdbt.com/docs/platform/account-settings.md#enable-global-account-discovery). * **Enhancement:** Credential-level [connection overrides](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) (such as Databricks catalog, Snowflake warehouse, role, and database) are now surfaced as a read-only **Connection overrides** section in the profile details view, without requiring you to open the edit form. * **Fix:** When a run pod is Out of Memory (OOM)-killed and restarted, the platform now passes the correct status code and message to the config API so the run transitions to a failed state in the dbt platform UI instead of remaining **running** indefinitely. * **Fix:** The Secure Shell (SSH) connection and authentication timeouts for Semantic Layer data platform connections are now 30 seconds (previously 1 second). If your bastion host or network path has higher latency, you will no longer experience deterministic connection failures. Refer to [Set up the Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md) for more information. * **Fix:** Some types of Compile SQL queries are now rejected if they are too complex. If a request fails with a validation error, try reducing the number of metrics or group-by dimensions in the query. * **Behavior change:** You can no longer create a [service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) using an account-scoped [personal access token](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) (PAT). Requests to the service tokens endpoint authenticated with a PAT now return a `400` error. Use a service token to create new ones instead. * **New:** You can now access dbt State settings from **Account settings** > **Billing & Usage**, previously found under **State**. You can manage your trial, enable dbt State on environments and jobs, and set spend alerts — all in one place. For details, refer to [dbt State trial and billing](https://docs.getdbt.com/docs/deploy/dbt-state-trial.md). * **New:** Redshift development connections now support [external OAuth](https://docs.getdbt.com/docs/platform/manage-access/redshift-external-oauth.md) (Okta or Entra ID) through AWS IAM Identity Center. ###### Docs changes To simplify the docs experience, clarify availability, and make it easier to find what applies to you, we made the following changes to the docs site: *tl;dr:* The docs are now organized around v1 and v2 for simplified docs versioning and navigation. We've clarified dbt Core and licensing, reorganized v2 content, and refreshed adapter and Fusion availability guidance. If you notice anything off or have any feedback, we'd love to hear it! Open up a [docs issue here](https://github.com/dbt-labs/docs.getdbt.com/issues). * **Enhancement**: We've updated the version switcher on the docs site. The version switcher now just shows v1 and v2. v2 is the current generation of dbt, built on Rust for a faster, richer dev experience; v1 is the Python-based generation of dbt. Refer to [dbt versions](https://docs.getdbt.com/docs/introduction.md#dbt-versions) for what's different between v1 and v2. * **New:** We've added a dedicated page explaining dbt Core and its distributions. dbt Core 2.0 is the Rust-based open-source runtime. dbt Core v1.x is the Python-based runtime. Refer to [About dbt Core](https://docs.getdbt.com/docs/introduction.md) for more info. * **New:** Licensing across dbt Core now has its own page, so you can see what applies to your setup in one place. Refer to [dbt licensing](https://docs.getdbt.com/docs/dbt-licensing.md). * **Enhancement:** [Static analysis](https://docs.getdbt.com/docs/build/about-static-analysis.md) now lives with the rest of your build docs and available in v2. * **Enhancement:** The Fusion upgrade readiness checklist now sits right next to the [v2 upgrade guide](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md), and the networking and telemetry references moved in [local install](https://docs.getdbt.com/docs/local/dbt-networking-requirements.md) and [Reference](https://docs.getdbt.com/reference/telemetry-observability.md). * **Enhancement:** More adapters are closer to general availability — Snowflake, BigQuery, Databricks, and Redshift are now in **Preview**, and Spark and DuckDB are in **Beta**. Refer to [Adapter lifecycles](https://docs.getdbt.com/docs/dbt/dbt-availability.md?version=2.0#adapter-lifecycle) for the current status of each adapter. * **Enhancement:** Simplified and clarified the [Fusion feature tables](https://docs.getdbt.com/docs/dbt/dbt-availability.md?version=2.0#what-you-get-with-fusion) to make it easier to see what's available and how to get it. * **New:** Added availability badges to pages and sections so you can quickly see what applies to your setup at a glance. #### June 2026 * **Enhancement:** [Column-level tags](https://docs.getdbt.com/reference/resource-configs/tags.md) defined in your dbt project now appear on the **Columns** tab of resource details pages in Catalog. You can click any tag badge to filter the lineage view, or search for columns directly by tag name. Refer to [View resource details](https://docs.getdbt.com/docs/explore/explore-projects.md#view-resource-details). * **Enhancement:** You can now enable [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) on continuous integration and merge job types, in addition to deploy jobs. For more information, refer to [Enabling dbt State on individual jobs](https://docs.getdbt.com/docs/deploy/dbt-state-enable-jobs.md). * **Enhancement**: The [Cost Insights](https://docs.getdbt.com/docs/explore/cost-insights.md) table view now includes **All** and **Jobs** buttons to switch between an aggregated cost view and a per-job cost breakdown. Available in the project dashboard and the **Model performance** section in Catalog. When **Jobs** is selected, the CSV export includes job-level data. For more information, refer to [Explore cost data](https://docs.getdbt.com/docs/explore/explore-cost-data.md). * **Enhancement:** [dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-platform.md) tool calls for dbt command invocations now stream their output live in chat, in both the Studio IDE and [Wizard home](https://docs.getdbt.com/docs/platform/wizard-home.md). * **Enhancement:** You can now download files from the Studio IDE File explorer. Right-click a file and select **Download** to save it to your computer. For more information, refer to the [Studio IDE user interface](https://docs.getdbt.com/docs/platform/studio-ide/ide-user-interface.md#basic-layout). * **Fix:** If you use the Administrator API to manage [SCIM](https://docs.getdbt.com/docs/platform/manage-access/scim.md) to sync users from your identity provider, the `/api/v3/accounts/{account_id}/scim/v2/Users` response now returns `value` and `display` on each embedded group reference. `id` and `displayName` are retained so existing integrations keep working — this is a non-breaking change. * **Enhancement**: The [Administrative API v3](https://docs.getdbt.com/dbt-cloud/api-v3) now supports private endpoint operations — [`list`](https://docs.getdbt.com/dbt-cloud/api-v3?version=2.0#/operations/List%20Private%20Endpoints), [`create`](https://docs.getdbt.com/dbt-cloud/api-v3?version=2.0#/operations/Create%20Private%20Endpoint), [`retrieve`](https://docs.getdbt.com/dbt-cloud/api-v3?version=2.0#/operations/Retrieve%20Private%20Endpoint), [`update`](https://docs.getdbt.com/dbt-cloud/api-v3?version=2.0#/operations/Update%20Private%20Endpoint), and [`delete`](https://docs.getdbt.com/dbt-cloud/api-v3?version=2.0#/operations/Delete%20Private%20Endpoint). Use these endpoints to manage private connectivity programmatically. * **Enhancement**: You can [download OpenTelemetry (OTel) logs](https://docs.getdbt.com/docs/deploy/run-visibility.md#access-logs) for individual dbt command steps in Fusion job runs. * **Enhancement**: You can now configure [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) for the Studio IDE directly in the dbt platform UI — either as a team-wide default on your development environment, or as a personal override. For more information, refer to [Enabling dbt State in Studio](https://docs.getdbt.com/docs/deploy/dbt-state-enable-studio.md). * **New:** [Model query history](https://docs.getdbt.com/docs/explore/model-query-history.md) for Redshift and Databricks is now generally available (GA). * **Behavior change:** On September 1, 2026, several behavior change flags on the dbt platform **Latest** release track will reach maturity (enabled by default). Refer to [About behavior changes](https://docs.getdbt.com/reference/global-configs/behavior-changes.md) to see which flags may affect your project and how to opt out before then. * **Beta:** dbt v2 now supports the Salesforce Data 360 connection in the dbt platform. For more information, refer to [Connect Salesforce Data 360](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-salesforce.md). * **Private beta**: The [Analyst read](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#analyst-read) permission set is available for Enterprise plans. * Analyst read is a project-level permission set that provides read-only access to analyze dbt models and project resources. The OAuth integration that lets read-only users connect to analysis features (such as the [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md)) is available to use, while the Analyst read permission set and read-only permission changes are in private beta. To enable them, contact your account manager. * **Beta**: Workspace-level Private Link for Microsoft Fabric is now available in beta. Configure a private connection between the dbt platform and your Fabric workspace so SQL traffic stays on Azure's private network. For more information, refer to [Configuring Private Link for Microsoft Fabric](https://docs.getdbt.com/docs/platform/secure/private-connectivity/azure/azure-fabric.md). * **Beta**: [Cost Insights](https://docs.getdbt.com/docs/explore/cost-insights.md) now supports Amazon Redshift Serverless and provisioned clusters. Configure your platform metadata credentials with the `sys:monitor` role or `SYSLOG ACCESS UNRESTRICTED` permission to allow dbt to read cross-user query history, then set your pricing in Cost Insights settings. For more information, refer to [Set up Cost Insights](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md). ##### Snowflake Summit 2026 announcements The following features are new or enhanced as part of dbt Labs announcements at [Snowflake Summit 2026](https://www.getdbt.com/events/snowflake-summit-2026) in San Francisco from June 1–4, 2026: * **Alpha**: [dbt Core 2.0](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md) is now available in alpha! * **New**: dbt Core 2.0 is the open-source Apache 2.0 foundation that dbt v2 builds on, delivering a faster, Rust-based runtime. It ships as two distributions: `dbt-core` (OSS, Apache 2.0) and `dbt` (dbt v2 distribution, proprietary). * **Beta**: [`dbt lint`](https://docs.getdbt.com/reference/commands/lint.md?version=2.0) is now available in beta! * **New**: `dbt lint` is a high-performance SQL linter built into the dbt platform, available on projects running dbt v2. It is SQLFluff-compatible; it reads your existing `.sqlfluff` config, uses the same rule codes, and respects `-- noqa` suppression comments. In benchmarks, it runs roughly 50× faster than single-threaded SQLFluff.. * **Preview**: [dbt Docs v2](https://docs.getdbt.com/docs/build/view-documentation.md#dbt-docs-v2) is now available in preview! * **New**: dbt Docs v2 is a next-generation open-source catalog experience available with dbt v2 and dbt Core 2.0. It uses a compact binary index instead of loading the full `manifest.json` in the browser, making it significantly faster for large projects. * **New**: dbt Docs v2 includes a redesigned UI, Semantic Layer metadata, column-level lineage (dbt v2 only), and a REST API at `/api/v1/` so AI agents and MCP servers can query your dbt project metadata without a browser. * **New**: Generate and serve [dbt Docs v2](https://docs.getdbt.com/docs/build/view-documentation.md#dbt-docs-v2) with dbt v2 or dbt Core 2.0 by running a dbt command with `--use-index`, then `dbt docs serve`. Add [`--write-catalog`](https://docs.getdbt.com/reference/commands/cmd-docs.md#--write-catalog-flag) for richer column type metadata. * **Preview**: [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) is now available in preview! * **New**: dbt State skips or clones nodes when the logic and data haven't changed, rather than rebuilding everything on every run. Available natively in dbt v2.0, the dbt platform, and dbt v2, and as a plugin for dbt v1.7-1.12. To get started, refer to [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md). * **New**: [dbt State pricing](https://docs.getdbt.com/docs/platform/billing/dbt-state-usage.md) is usage-based at $0.094 per daily unique reuse. New organizations receive a 30-day free trial with no usage limit. * **Behavior change**: State-aware orchestration is no longer being enabled for new customers. Refer to [Migrate to dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md) for more information. * **New**: dbt Wizard is available in dbt platform as a public preview. Introducing dbt Wizard CLI as a public beta. Purpose-built for agentic governed data development in dbt, dbt Wizard understands your project through a [native metadata engine](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md#native-metadata-engine), unlike general-purpose coding agents. * **New**: [Support for Anthropic as a BYOK provider for dbt AI](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md#configure-ai-provider). * **New**: [`dbt login`](https://docs.getdbt.com/reference/commands/login.md?version=2.0) is a new CLI command available in dbt Core 2.0 and later. It opens browser-based authentication and shares your login state across the CLI, dbt VS Code extension, dbt State, and dbt Wizard CLI with no separate sign-in flows needed. * **New:** OAuth client registrations now accept custom-scheme redirect URIs (for example, `cursor://` or `vscode://`), so you can build native app OAuth integrations with Cursor and VS Code. * **New:** Public REST API endpoints at `/api/ide/v3/{environment_id}/files/` support Studio IDE workspace file operations, including stat, read, write, list, delete, mkdir, and rename. Pass file paths as query parameters. * **New:** The `GET /api/ide/v3/{environment_id}/status` endpoint returns the `dbt_version` and `is_fusion` status for a given environment. * **New:** The dbt platform CLI Python client's `create_invocation()` method now supports a `workspace` parameter, so you can run invocations against persisted workspace files on workers. #### May 2026 * **Enhancement:** Repository clone failures now surface a more actionable diagnostic message to help you resolve common issues faster. For guidance, refer to [Troubleshooting clone errors](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md#troubleshooting-clone-errors). * **Fix:** The connection test failure message now prompts you to verify your connection details and confirm that your credentials have access to the data warehouse, rather than showing a generic failure message. * **Enhancement:** Users granted `user_credential_write` can access **Your profile** > **Credentials** without `develop_access` (including read-only users). Environment variable overrides and dbt version overrides still require `develop_access`. Refer to [Enterprise permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) for more information. * **New:** The [Job creator permission set](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#job-creator) is now available for Enterprise accounts. Assign it to users who need to create, edit, and run jobs within assigned projects and environments without access to edit environments or environment variables. * **Enhancement:** The admin API toolset (job management and run operations) is now always available in the dbt [](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md)and no longer requires a feature flag. You no longer need to contact your account manager to enable these tools. * **Fix:** When a job cannot clone its repository because no remote URL is configured, the error message now explains the most likely causes (an invalid Git remote URL, a Git provider outage, or a deprecated HTTPS connection) and directs you to verify the URL, confirm your provider is operational, and ensure the repository uses SSH with deploy keys before retrying. * **New:** The **Notification Manager** [permission set](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) is now available for Enterprise accounts. Assign it to users who need to manage Slack, Microsoft Teams, and email job notifications across all projects without requiring full Account Admin access. * **Beta**: [Cost Insights](https://docs.getdbt.com/docs/explore/cost-insights.md), available in public beta, shows estimated warehouse compute costs and run times for dbt projects and models in dbt platform, highlighting efficiency gains from [state-aware orchestration](https://docs.getdbt.com/docs/deploy/state-aware-about.md). Refer to [Set up Cost Insights](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md) and [Explore cost data](https://docs.getdbt.com/docs/explore/explore-cost-data.md) to learn more. * **New:** Fusion release tracks are now being rolled out across accounts in phases. Refer to [Fusion release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md?version=2.0#dbt-v2-release-tracks) for more information. * **Enhancement:** Commands run by and the [](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md)now appear in the Studio IDE **Commands** tab with a icon and **Run by Copilot** tooltip, so you can tell agent-run commands apart from manually run ones. * **Fix:** [`state:modified`](https://docs.getdbt.com/reference/node-selection/methods.md#state) now detects changes to [UDF](https://docs.getdbt.com/docs/build/udfs.md) properties (such as `arguments` and `returns`) defined in `.yml` files. Previously, only changes to the SQL or Python function body were detected. * **New:** [Native private packages](https://docs.getdbt.com/docs/build/packages.md#native-private-packages) are now generally available (GA). * **Preview**: The [Developer agent](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) is now in preview. Use natural language prompts to build or refactor models, and generate SQL, tests, documentation, and semantic models from scratch. For more information, refer to the [Developer agent](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md). * **Behavior change:** When you set up single sign-on (SSO) in the dbt platform, the SSO slug is now system-generated and read-only. Existing SSO configurations remain valid, but you can’t change the slug. If you delete and recreate your SSO configuration, the new configuration uses a new, system-generated slug. Refer to [Single sign-on overview](https://docs.getdbt.com/docs/platform/manage-access/sso-overview.md) for more information. * **Enhancement:** The [dbt VS Code extension](https://docs.getdbt.com/docs/install-dbt-extension.md?version=2.0) now supports account creation. If you sign in with an existing dbt user that doesn't have an associated dbt platform account, the registration flow prompts you to create one instead of requiring a separate workflow. * **Enhancement:** Delete individual [dbt Wizard chat conversations](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md#availability-and-considerations) from the conversation list (three dots → **Delete**). Deleting the open conversation clears the panel. * **New:** The Fusion + Snowflake connection experience is now generally available on the dbt platform. See our [Fusion upgrade guides](https://docs.getdbt.com/guides/prepare-v2-upgrade.md?step=1) for information on enabling the upgrade workflows for your environments today! * **Enhancement:** In the Discovery API [Tests object schema](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-environment-applied-tests.md), you can now filter `environment.applied.tests` by multiple test result statuses in a single query using the new `lastKnownResults: [TestStatus]` filter field on `TestAppliedFilter`. The single-value `lastKnownResult` filter field is still supported but deprecated. Update your queries to use `lastKnownResults` going forward. * **Enhanced** dbt v2 eligibility job prompts now use a **Debug on dbt v2** dropdown instead of a standalone **Run once on dbt v2** button. For more information, refer to [Update your jobs](https://docs.getdbt.com/guides/prepare-v2-upgrade.md?step=7). * **Enhancement:** The [](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md)input bar now supports arrow key history navigation. Press the up arrow at the start of the input to cycle through previous inputs, and the down arrow at the end to return to more recent ones. dbt stores up to 5 previous inputs per session. * **Enhancement:** Tool approval and file edit dialogs in the [](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md)now support number key shortcuts (1, 2, 3) to select options. The first option is auto-focused when a dialog appears, so you can act immediately without clicking. #### April 2026 * **Enhancement:** When a dbt command run by the [](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md)times out, the agent now automatically attempts to cancel the stuck invocation on the server and returns a retry-friendly message, letting you decide whether to retry. Previously, timeouts resulted in an unhandled error. This applies to both model invocations and autofix runs. * **Enhancement:** In dbt platform run logs, `dbt ls` and `dbt list` now display node results as **No-op** instead of **Unknown** when using dbt v2. Refer to [dbt ls (list)](https://docs.getdbt.com/reference/commands/list.md) for more information. * **New:** A universal login URL is available at <https://login.dbt.com>, making it easier for you to view accounts you have access to across instances (regions and tenancies). This is currently available for multi-tenant accounts with an account-specific domain, and support for single-tenant accounts is coming soon. For more information, refer to [Log in to dbt platform](https://docs.getdbt.com/docs/platform/about-platform/login.md). * **Fix:** Refreshing the same browser tab now restores your active dbt Wizard conversation instead of showing the empty state. Opening a new tab, or returning after closing the tab, still starts in the empty state. The dbt Wizard is currently in beta. * **Enhancement:** The dbt VS Code extension's **Get started** panel has been redesigned and surfaces the exact next setup step you need to install the extension and dbt v2. The new panel also supports a new **agentic migration** option that helps you upgrade your project to dbt v2 automatically in Copilot or Cursor. For more info, see [Getting started](https://docs.getdbt.com/docs/install-dbt-extension.md#getting-started). * **Beta**: [Model query history](https://docs.getdbt.com/docs/explore/model-query-history.md) now also supports Databricks and Redshift. Refer to [Credential permissions](https://docs.getdbt.com/docs/explore/model-query-history.md#credential-permissions) for more information. * **Enhancement:** [Slack notifications (account-level)](https://docs.getdbt.com/docs/deploy/job-notifications.md#slack-notifications-account) and [Microsoft Teams notifications](https://docs.getdbt.com/docs/deploy/job-notifications.md#microsoft-teams-notifications) are now generally available, enabling you to send job notifications directly to Slack channels configured at the account level, and to Teams channels. * **Enhancement:** When using the [dbt autofix](https://github.com/dbt-labs/dbt-autofix) tool in the Studio IDE, you can now compile your project directly from the results panel after a successful `dbt parse`. Click **Compile** next to the **Successfully resolved** result to kick off a compile. For more information, refer to [Fix deprecation warnings](https://docs.getdbt.com/docs/platform/studio-ide/autofix-deprecations.md). * **Beta**: DuckDB is now supported in the dbt v2 CLI, which lets you run local dbt projects without a warehouse account. For more information, refer to [Connect DuckDB](https://docs.getdbt.com/docs/local/connect-data-platform/duckdb-setup.md). * **New**: You can now configure Snowflake PrivateLink endpoints directly in dbt platform without contacting dbt Support, available in private beta. Go to **Account settings → Integrations → Private endpoints** to request and manage Snowflake PrivateLink endpoints on AWS. This feature is available for Snowflake on AWS only. For more information, refer to [AWS PrivateLink for Snowflake](https://docs.getdbt.com/docs/platform/secure/private-connectivity/aws/aws-snowflake.md?version=1.12). * **Enhancement:** You can now use arrays as values for keys in the dbt platform extended attributes YAML editor. For example, `db_groups: [db_editor, db_viewer]` is now valid. Previously, array values were only supported using the API. For more information, refer to [Extended attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes). * **Beta**: The Redshift adapter now supports a `datasharing` profile credential on the dbt platform **Latest** release track. When set to `true`, dbt uses Redshift's native `SHOW` commands (for example, `SHOW TABLES`, `SHOW COLUMNS`, `SHOW SCHEMAS`) for metadata queries instead of PostgreSQL catalog tables, enabling cross-database and cross-cluster access with [Redshift Datasharing](https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html). For more information, refer to [Redshift setup](https://docs.getdbt.com/docs/local/connect-data-platform/redshift-setup.md#datasharing). * **Enhancement:** When a connection does not have platform metadata credentials configured yet, the credentials form now renders in edit mode immediately — you no longer need to click **Add credentials** first. If you cancel, the **Add credentials** button appears so you can return to the form. Existing connections with configured platform metadata credentials are unaffected. Refer to [Configure the warehouse connection](https://docs.getdbt.com/docs/explore/external-metadata-ingestion.md#configure-the-warehouse-connection) for more information. * **New**: The [dbt Remote dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md?version=2.0) now supports Admin API calls! This allows users to troubleshoot job-related errors in agents like Claude and Cursor. * **New**: The [Developer agent](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) is now in beta. Use the Developer agent to write or refactor dbt models from natural language, generate documentation, tests, semantic models, and SQL code from scratch, giving you the flexibility to modify or fix generated code. For more information, refer to the [Developer agent](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md). * **Enhancement:** The Studio IDE now validates dbt YAML using dbt v2 aligned JSON Schema from [dbt-jsonschema](https://github.com/dbt-labs/dbt-jsonschema) across [dbt platform release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md), including for development environments on dbt v1. This improves autocomplete and structural feedback in the editor. Diagnostics can occasionally disagree with what your environment accepts; use dbt runs and previews as the source of truth. For context, review [Migrate to the latest YAML spec](https://docs.getdbt.com/docs/build/latest-metrics-spec.md) and [dbt YAML validation in Studio](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md#dbt-yaml-validation). This will be a phased rollout starting the week of April 6th. * **Enhancement:** The Studio IDE status bar now offers more control, more detailed information, and quicker access to settings for deferral, dbt version, and project status. For more information, refer to the [Studio IDE docs](https://docs.getdbt.com/docs/platform/studio-ide/ide-user-interface.md#the-command-and-status-bar). These updates roll out in phases to existing accounts starting April 6. * **Enhancement:** In Snowflake **Private endpoints**, output validation errors now display inline beneath the text area (instead of as a page-level banner). The **Submit request** button is also disabled when the output is invalid (for example, empty, malformed JSON, or missing required fields). * **Enhancement:** The Studio IDE now supports deep links to a specific console tab using the `?consoleTab=` query parameter. For example, append `?consoleTab=problems` to open Studio with the **Problems** tab pre-selected. The `problems` tab applies only when it is available for the current session. #### March 2026 * **Enhancement:** The environment [Connection profiles](https://docs.getdbt.com/docs/platform/about-profiles.md#environment-profiles-table) page has been updated. The profile name is now a clickable button that opens the view/edit drawer, the Connection column links to the connection details page in a new tab, and in edit mode a **swap icon** button lets you change the assigned profile. The previous ellipsis menu has been removed. For details, refer to [About profiles](https://docs.getdbt.com/docs/platform/about-profiles.md). * **Beta:** Apache Spark is now supported in the dbt v2 CLI, enabling faster compilation and execution for Spark-based dbt projects. dbt v2 currently supports only Apache Spark 3.0. For more information, refer to [Connect Apache Spark to Fusion](https://docs.getdbt.com/docs/local/connect-data-platform/spark-setup.md). * **Enhancement:** [Cost Insights](https://docs.getdbt.com/docs/explore/cost-insights.md) charts now include an **Assets** filter (**Models** / **Tests** / **All**) on the **Cost**, **Usage**, **Query run time**, and **Builds** tabs. Use the dropdown on each chart to filter the data you want to view; your selection is stored per tab. The former **Model builds** tab is now labeled **Builds**. For more information, refer to [Explore cost data](https://docs.getdbt.com/docs/explore/explore-cost-data.md). * **Enhancement:** [Deferral](https://docs.getdbt.com/reference/node-selection/defer.md) now supports [user-defined functions (UDFs)](https://docs.getdbt.com/docs/build/udfs.md). When you run a dbt command with `--defer` and `--state`, dbt resolves `function()` calls from the state manifest. This lets you run models that depend on UDFs without first building those UDFs in your current target. * **Fix**: Status messages that exceed the 1024 character limit are now automatically truncated to prevent validation errors and run timeouts. Previously, long status messages could cause runs to fail with unhandled exceptions or result in lost status information. The system now logs when truncation occurs to help identify and optimize verbose status messages. * **Fix:** Resolved an issue where [retrying failed runs](https://docs.getdbt.com/docs/deploy/retry-jobs.md) that were triggered from Git tags would use the wrong commit. Previously, when runs were triggered from Git tags instead of branches, the system would enter a detached HEAD state, causing retries to use the latest commit on HEAD rather than the original tagged commit. The fix now correctly preserves and uses the original Git tag reference when retrying runs, ensuring consistency between the initial run and any retries. * **New**: The [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md?version=2.0#product-docs) now includes product docs tools (`search_product_docs` and `get_product_doc_pages`) that let your AI assistant search and fetch pages from docs.getdbt.com in real time. Get responses grounded in the latest official dbt documentation rather than relying on training data or web searches, so you can stay in your development flow and trust the answers. This allows you to stay in your development flow and trust. These tools are enabled by default with no additional configuration. Restart your MCP server if you don't see the product docs tools in your MCP config. For more information, refer to [the dbt MCP repo](https://github.com/dbt-labs/dbt-mcp?tab=readme-ov-file#product-docs). * **Enhancement**: The Model Timing tab displays an informative banner for dbt v2 runs instead of the timing chart. The banner explains "Model timing is not yet available for Fusion runs" and provides context about threading differences. Non-Fusion runs continue to show the timing chart normally. * **Behavior change**: [Snowflake plans to increase](https://docs.snowflake.com/en/release-notes/bcr-bundles/un-bundled/bcr-2118) the default column size for string and binary data types in September 2026. `dbt-snowflake` versions below v1.10.6 may fail to build certain incremental models when this change is deployed. [Assess impact and take any required actions](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#assess-impact-and-required-actions). * **New**: The new Semantic Layer YAML specification is now available on the dbt platform **Latest** release track. For an overview of the changes and steps how to migrate to the latest YAML spec, refer to [Migrate to the latest YAML spec](https://docs.getdbt.com/docs/build/latest-metrics-spec.md). * **Behavior change:** New projects in trial, starter, or Enterprise accounts now default to **Fusion Stable** for all new environments with a supported adapter (Redshift, Snowflake, BigQuery, and Databricks). You can revert to another version by changing the dbt version in your [environment settings](https://docs.getdbt.com/docs/dbt-platform-environments.md#change-environment-settings). #### February 2026 * **New**: Advanced CI (dbt compare in orchestration) is now supported in dbt v2. For more information, review [Advanced CI](https://docs.getdbt.com/docs/deploy/advanced-ci.md). * **Beta**: The `dbt-salesforce` adapter available in the dbt v2 CLI is now in beta. For more information, refer to [Salesforce Data 360 setup](https://docs.getdbt.com/docs/local/connect-data-platform/salesforce-data-cloud-setup.md). * **Enhancement:** The Analyst permission now has the project-level access to read repositories. Review [Project access for project permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#project-access-for-project-permissions) for more information. * **Enhancement:** After a user accepts an email [invite](https://docs.getdbt.com/docs/platform/manage-access/invite-users.md) to access an [SSO-protected](https://docs.getdbt.com/docs/platform/manage-access/sso-overview.md) dbt platform account, the UI now prompts them to log in with SSO to complete the process. This replaces the previous "Joined successfully" message, helping avoid confusion when users accept an invite but do not complete the SSO login flow. * **New:** [Profiles](https://docs.getdbt.com/docs/platform/about-profiles.md) let you define and manage connections, credentials, and attributes for deployment environments at the project level. dbt automatically creates profiles for existing projects and environments based on the current configurations, so you don't need to take any action. This is being rolled out in phases during the coming weeks. * **New**: [Python UDFs](https://docs.getdbt.com/docs/build/udfs.md) are now supported and available in dbt v2 when using Snowflake or BigQuery. * **Enhancement:** Minor enhancements and UI updates to the Studio IDE, file explorer that replicate the VS Code IDE experience. * **Enhancement:** Profile creation now displays specific validation error messages (such as "Profile keys cannot contain spaces or special characters") instead of generic error text, making it easier to identify and fix configuration issues. * **Private beta**: [Cost Insights](https://docs.getdbt.com/docs/explore/cost-insights.md) shows estimated warehouse compute costs and run times for your dbt projects and models, directly in the dbt platform. It highlights cost reductions and efficiency gains from optimizations like [state-aware orchestration](https://docs.getdbt.com/docs/deploy/state-aware-about.md) across your project dashboard, model pages, and job details. Refer to [Set up Cost Insights](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md) and [Explore cost data](https://docs.getdbt.com/docs/explore/explore-cost-data.md) to learn more. * **New**: The [dbt Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) now supports [Omni](https://docs.omni.co/integrations/dbt/semantic-layer) as a partner integration. For more information, refer to [Available integrations](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md). * **Enhancement**: We clarified documentation for cumulative log size limits on run endpoints, originally introduced in [October 2025](https://docs.getdbt.com/docs/dbt-versions/2025-release-notes.md#october-2025). When logs exceed the cumulative size limit, dbt omits them and displays a banner. No functional changes were made in February 2026. For more information, review [Run visibility](https://docs.getdbt.com/docs/deploy/run-visibility.md#log-size-limits). * **New**: The `immutable_where` configuration is now supported for Snowflake dynamic tables. For more information, refer to [Snowflake configurations](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#immutable-where). * **Fix**: The user invite details now show more information in invite status, giving admins visibility into users who accepted an invite to an SSO-protected account but haven't yet logged in via SSO. Previously, these invites were hidden, making it appear as if the user hadn't been invited. The Invites endpoints of the dbt platform Admin v2 API now include these additional statuses: * `4` (PENDINGEMAIL\_VERIFICATION) * `5` (EMAIL\_VERIFIED\_SSO). * **Enhancement**: Improved performance on Runs endpoint for Admin V2 API and run details in dbt platform when connecting with GCP. #### January 2026 * **Enhancement:** The `defer-env-id` setting for choosing which deployment environment to defer to is [now available](https://docs.getdbt.com/docs/platform/about-defer.md#configure-deferral-environment-id) in the Studio IDE. Previously, this configuration only worked for the dbt platform CLI * **Beta:** The [Analyst agent](https://docs.getdbt.com/docs/explore/navigate-dbt-insights.md#dbt-copilot) in dbt Insights is now in beta. * dbt dbt Wizard's AI assistant in Insights now uses a dropdown menu to select between **Agent** and **Generate SQL**, replacing the previous tab interface. * **Enhancement:** The [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/ide-user-interface.md#search-your-project) now includes search and replace functionality and a command palette, enabling you to quickly find and replace text across your project, navigate files, jump to symbols, and run IDE configuration commands. This feature is being rolled out in phases and will become available to all dbt platform accounts by mid-February. * **Enhancement:** [State-aware orchestration](https://docs.getdbt.com/docs/deploy/state-aware-about.md) improvements: * When a model fails a data test, state-aware orchestration rebuilds it on subsequent runs instead of reusing it from prior state to ensure dbt reevaluates data quality issues. * State-aware orchestration now detects and rebuilds models whose tables are deleted from the warehouse, even when there are no code or data changes. Previously, tables deleted externally were not detected, and therefore not rebuilt, unless code or data had changed. For more information, review [Handling deleted tables](https://docs.getdbt.com/docs/deploy/state-aware-about.md#handling-deleted-tables). State-aware orchestration is in private preview. refer to the [prerequisites for using the feature](https://docs.getdbt.com/docs/deploy/state-aware-setup.md#prerequisites). * **Enhancement:** [dbt dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-platform.md) correctly detects column names across various `schema.yml` files, adds only missing descriptions, and preserves existing ones. * **Enhancement**: v2 now automatically reads environment variables from a `.env` file in your current working directory (the folder you `cd` into and run dbt commands from in your terminal), if one exists. This provides a simple way to manage credentials and configuration without hardcoding them in your `profiles.yml`. The [dbt VS Code extension](https://docs.getdbt.com/docs/about-dbt-extension.md) also supports `.env` files and LSP-powered features. For more information, refer to [Configure environment variables](https://docs.getdbt.com/docs/local/configure-environment-variables.md). * **New**: The new Semantic Layer YAML specification creates an open standard for defining metrics and dimensions that works across multiple platforms. The new spec is now live in dbt v2. Key changes: * Semantic models are now embedded within model YAML entries. This removes the need to manage YAML entries across multiple files. * Measures are now simple metrics. * Frequently used options are now top-level keys, reducing YAML nesting depth. For an overview of the changes and steps how to migrate to the latest YAML spec, check [Migrate to the latest YAML spec](https://docs.getdbt.com/docs/build/latest-metrics-spec.md). * **Fix:** Debug logs in the **Run summary** tab are now properly truncated to improve performance and user interface responsiveness. Previously, debug logs were not truncated correctly, causing slower page loads. You can access the full debug logs by clicking **Download > Download all debug logs**. For more information, review [Run visibility](https://docs.getdbt.com/docs/deploy/run-visibility.md#run-summary-tab). * **New:** The [Semantic Layer querying](https://docs.getdbt.com/docs/explore/navigate-dbt-insights.md#semantic-layer-querying) within dbt Insights is now generally available (GA), enabling you to build SQL queries against the Semantic Layer without writing SQL code. * **Enhancement**: Eligible dbt platform accounts in the dbt v2 private preview can now use [Exposures](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures.md). --- ### dbt Semantic Layer dbt platform | Starter, Enterprise, Enterprise+ The dbt Semantic Layer eliminates duplicate coding by allowing data teams to define metrics on top of existing models and automatically handling data joins. The dbt Semantic Layer, powered by [MetricFlow](https://docs.getdbt.com/docs/build/about-metricflow.md), simplifies the process of defining and using critical business metrics, like `revenue` in the modeling layer (your dbt project). By centralizing metric definitions, data teams can ensure consistent self-service access to these metrics in downstream data tools and applications. Moving metric definitions out of the BI layer and into the modeling layer allows data teams to feel confident that different business units are working from the same metric definitions, regardless of their tool of choice. If a metric definition changes in dbt, it’s refreshed everywhere it’s invoked and creates consistency across all applications. To ensure secure access control, the Semantic Layer implements robust [access permissions](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md#set-up-dbt-semantic-layer) mechanisms. Refer to the [Semantic Layer FAQs](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-faqs.md) or [Why we need a universal semantic layer](https://www.getdbt.com/blog/universal-semantic-layer/) blog post to learn more. [YouTube video player](https://www.youtube.com/embed/DS7Ub_CmBR0?si=m92hLmxw1VuE6KKO) #### Get started with the dbt Semantic Layer To define and query metrics with the dbt Semantic Layer, you must be on a [dbt Starter or Enterprise-tier](https://www.getdbt.com/pricing/) account. [](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses)Suitable for both Multi-tenant and Single-tenant accounts. Note: Single-tenant accounts should contact their account representative for necessary setup and enablement. <br /> <br /> This page points to various resources available to help you understand, configure, deploy, and integrate the Semantic Layer. The following sections contain links to specific pages that explain each aspect in detail. Use these links to navigate directly to the information you need, whether you're setting up the Semantic Layer for the first time, deploying metrics, or integrating with downstream tools. Refer to the following resources to get started with the Semantic Layer: * [Quickstart with the Semantic Layer](https://docs.getdbt.com/guides/sl-qs.md): Build and define metrics, set up the Semantic Layer, and query them in Google Sheets and other tools. * [Build your metrics](https://docs.getdbt.com/docs/build/build-metrics-intro.md) — Use MetricFlow in dbt to centrally define your metrics. * [Semantic Layer FAQs](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-faqs.md) — Discover answers to frequently asked questions about the Semantic Layer, such as availability, integrations, and more. #### Configure the dbt Semantic Layer The following resources provide information on how to configure the Semantic Layer: * [Administer the Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md) — Seamlessly set up the credentials and tokens to start querying the Semantic Layer. * [Architecture](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-architecture.md) — Explore the powerful components that make up the Semantic Layer. #### Deploy metrics This section provides information on how to deploy the Semantic Layer and materialize your metrics: * [Deploy your Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/deploy-sl.md) — Run a dbt job to deploy the Semantic Layer and materialize your metrics. * [Write queries with exports](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md) — Use exports to write commonly used queries directly within your data platform, on a schedule. * [Cache common queries](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-cache.md) — Leverage result caching and declarative caching for common queries to speed up performance and reduce query computation. #### Consume metrics and integrate Consume metrics and integrate the Semantic Layer with downstream tools and applications: * [Consume metrics](https://docs.getdbt.com/docs/use-dbt-semantic-layer/consume-metrics.md) — Query and consume metrics in downstream tools and applications using the Semantic Layer. * [Available integrations](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) — Review a wide range of partners you can integrate and query with the Semantic Layer. * [Semantic Layer APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md) — Use the Semantic Layer APIs to query metrics in downstream tools for consistent, reliable data metrics. --- ### dbt Semantic Layer architecture dbt platform The Semantic Layer allows you to define metrics and use various interfaces to query them. The Semantic Layer does the heavy lifting to find where the queried data exists in your data platform and generates the SQL to make the request (including performing joins). [![This diagram shows how the dbt Semantic Layer works with your data stack.](/img/docs/dbt-platform/semantic-layer/sl-concept.png?v=2 "This diagram shows how the dbt Semantic Layer works with your data stack.")](#)This diagram shows how the dbt Semantic Layer works with your data stack. [![The diagram displays how your data flows using the dbt Semantic Layer and the variety of integration tools it supports.](/img/docs/dbt-platform/semantic-layer/sl-architecture.jpg?v=2 "The diagram displays how your data flows using the dbt Semantic Layer and the variety of integration tools it supports.")](#)The diagram displays how your data flows using the dbt Semantic Layer and the variety of integration tools it supports. #### Components The Semantic Layer includes the following components: | Components | Information | local dbt | Developer plans | Starter plans | Enterprise-tier plans | License | | ----------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | --------------- | ------------- | --------------------- | ------------------------------------------------------------------------------------------- | | **[MetricFlow](https://docs.getdbt.com/docs/build/about-metricflow.md)** | MetricFlow in dbt allows users to centrally define their semantic models and metrics with YAML specifications. | ✅ | ✅ | ✅ | ✅ | [Apache 2.0 license](https://github.com/dbt-labs/metricflow/blob/main/LICENSE) | | **dbt Semantic interfaces** | A configuration spec for defining metrics, dimensions, and how they link to each other. The [dbt-semantic-interfaces](https://github.com/dbt-labs/dbt-semantic-interfaces) is available under Apache 2.0. | ✅ | ✅ | ✅ | ✅ | [Apache 2.0 license](https://github.com/dbt-labs/dbt-semantic-interfaces/blob/main/LICENSE) | | **Service layer** | Coordinates query requests and dispatching the relevant metric query to the target query engine. This is provided through dbt and is available to all users on dbt version 1.6 or later. The service layer includes a Gateway service for executing SQL against the data platform. | ❌ | ❌ | ✅ | ✅ | Proprietary, Cloud (Starter, Enterprise, Enterprise+) | | **[Semantic Layer APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md)** | The interfaces allow users to submit metric queries using GraphQL and JDBC APIs. They also serve as the foundation for building first-class integrations with various tools. | ❌ | ❌ | ✅ | ✅ | Proprietary, Cloud (Starter, Enterprise, Enterprise+) | #### Feature comparison The following table compares the features available in dbt and source available in dbt v1: | Feature | MetricFlow Source available | Semantic Layer with dbt | | -------------------------------------------------------------------------------------- | --------------------------- | ----------------------- | | Define metrics and semantic models in dbt using the MetricFlow spec | ✅ | ✅ | | Generate SQL from a set of config files | ✅ | ✅ | | Query metrics and dimensions through the command line interface (CLI) | ✅ | ✅ | | Query dimension, entity, and metric metadata through the CLI | ✅ | ✅ | | Query metrics and dimensions through semantic APIs (ADBC, GQL) | ❌ | ✅ | | Connect to downstream integrations (Tableau, Hex, Dot, Mode, Google Sheets, and so on) | ❌ | ✅ | | Create and run Exports to save metrics queries as tables in your data platform | ❌ | ✅ | #### Related docs * [Semantic Layer FAQs](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-faqs.md) --- ### dbt Semantic Layer FAQs dbt platform The [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) is a dbt offering that allows users to centrally define their metrics within their dbt project using [MetricFlow](https://docs.getdbt.com/docs/build/about-metricflow.md). The Semantic Layer offers: * Dynamic SQL generation to compute metrics * APIs to query metrics and dimensions * First-class [integrations](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) to query those centralized metrics in downstream tools The Semantic Layer is powered by MetricFlow, which is a source-available component. #### Overview of the dbt Semantic Layer  What are the main benefits of using the dbt Semantic Layer? The primary value of the dbt Semantic Layer is to centralize and bring consistency to your metrics across your organization. Additionally, it allows you to: * **Meet your users where they are** by being agnostic to where your end users consume data through the supporting of different APIs for integrations. * **Optimize costs** by spending less time preparing data for consumption. * **Simplify your code** by not duplicating metric logic and allowing MetricFlow to perform complex calculations for you. * **Empower stakeholders** with rich context and flexible, yet governed experiences. [![This diagram shows how the dbt Semantic Layer works with your data stack.](/img/docs/dbt-platform/semantic-layer/sl-concept.png?v=2 "This diagram shows how the dbt Semantic Layer works with your data stack.")](#)This diagram shows how the dbt Semantic Layer works with your data stack.  What's the main difference between the dbt Semantic Layer and dbt Metrics? dbt Metrics is the now-deprecated dbt package that was used to define metrics within dbt. dbt Metrics has been replaced with [MetricFlow](https://docs.getdbt.com/docs/build/about-metricflow.md), a more flexible and powerful engine, which powers the foundation of the dbt Semantic Layer today. MetricFlow introduces SQL generation to the dbt Semantic Layer and offers more advanced capabilities than dbt Metrics, for example: * **Query construction** — MetricFlow iteratively constructs queries using a dataflow plan, our internal DAG for generating SQL. By comparison, dbt Metrics relied on templated Jinja to construct SQL. * **Joins** — MetricFlow also has a sophisticated way of handling joins, which dbt Metrics did not support. With MetricFlow you can effortlessly access all valid dimensions for your metrics on the fly, even when they are defined in different semantic models.  Is there a dbt Semantic Layer discussion hub? Yes, absolutely! Join the [dbt Slack community](https://app.slack.com/client/T0VLPD22H) and [#dbt-cloud-semantic-layer](https://getdbt.slack.com/archives/C046L0VTVR6) slack channel for all things related to the dbt Semantic Layer.  How does the dbt Semantic Layer fit with different modeling approaches (Medallion, Data Vault, Dimensional modeling)? The dbt Semantic Layer is flexible enough to work with many common modeling approaches. It references dbt models, which means how you configure your Semantic Layer will mirror the modeling approach you've taken with the underlying data. The primary consideration is the flexibility and performance of the underlying queries. For example: * A star schema data model offers more flexibility for dimensions that are available for a given metric, but will require more joins. * A fully denormalized data model is simpler, will be materialized to a specific grain, but won’t be able to join to other tables. While the dbt Semantic Layer will work for both cases, it's best to allow MetricFlow to handle some level of denormalization for you in order to provide more flexibility to metric consumers.  How is the dbt Semantic Layer priced? The dbt Semantic Layer measures usage in distinct 'Queried Metrics'. Refer to the [Billing](https://docs.getdbt.com/docs/platform/billing/how-pricing-works.md#what-counts-as-a-queried-metric) to learn more about pricing. #### Availability  What data platforms are supported by the dbt Semantic Layer? The dbt Semantic Layer supports the following data platforms: * Snowflake * BigQuery * Databricks * Redshift * Postgres * Trino Support for other data platforms, such as Fabric, isn't available at this time. If you're interested in using the dbt Semantic Layer with a data platform not on the list, please [contact us](https://www.getdbt.com/contact).  Do I need to be on a specific version of dbt to use dbt Semantic Layer? Yes, the dbt Semantic Layer is compatible with [dbt v1.6 or higher](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md).  Does dbt Semantic Layer require a specific dbt plan? Yes, dbt [Starter, Enterprise, or Enterprise+](https://www.getdbt.com/pricing) plan customers can access the dbt Semantic Layer. Certain features like caching and using multiple credentials are available for Enterprise and Enterprise+ plans.  Is there a way to leverage dbt Semantic Layer capabilities outside the dbt platform? The dbt Semantic Layer is proprietary to dbt, however some components of it are open-source. dbt users can use MetricFlow features, like defining metrics in their projects, without a dbt platform plan. dbt users can also query their semantic layer locally using the command line. However, they won't be able to use the [APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md) or [available integrations](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) to access metrics dynamically.  Is there a solution or licensing path for an organization that doesn't use dbt for pipelining, but might like to implement the dbt Semantic Layer? If you're interested in the this type of implementation, please reach out to us [here](https://www.getdbt.com/contact). #### How does the dbt Semantic Layer work?  Why is the dbt Semantic Layer better than using tables or dbt models to calculate metrics? You can use tables and dbt models to calculate metrics as an option, but it's a static approach that is rigid and cumbersome to maintain. That’s because metrics are seldom useful on their own: they usually need dimensions, grains, and attributes for business users to analyze (or slice and dice) data effectively. If you create a table with a metric, you’ll need to create numerous other tables derived from that table to show the desired metric cut by the desired dimension or time grain. Mature data models have thousands of dimensions, so you can see how this will quickly result in unnecessary duplication, maintenance, and costs. It's also incredibly hard to predict all the slices of data that a user is going to need ahead of time. With the dbt Semantic Layer, you don’t need to pre-join or build any tables; rather, you can simply add a few lines of code to your semantic model, and that data will only be computed upon request. [![This diagram shows how the dbt Semantic Layer works with your data stack.](/img/docs/dbt-platform/semantic-layer/sl-concept.png?v=2 "This diagram shows how the dbt Semantic Layer works with your data stack.")](#)This diagram shows how the dbt Semantic Layer works with your data stack.  Do I materialize anything when I define a semantic model? No, you don't. When querying the dbt Semantic Layer through the [Semantic Layer APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md), you're not materializing any data by default. The dbt Semantic Layer dynamically computes the metric using the underlying data tables. Then it returns the output to the end user.  Is the dbt Semantic Layer a physical copy of your data stored on your data warehouse? The dbt Semantic Layer does not store a physical copy of your data. It uses underlying tables to construct or compute the requested output.  How does the Semantic Layer handle data? The dbt Semantic Layer is part of the dbt platform. It allows data teams to define metrics once, centrally, and access them from any integrated analytics tool, ensuring consistent answers across diverse datasets. In providing this service, dbt Labs permits clients to access Semantic Layer metrics. Client data passes through the Semantic Layer on the way back from the data warehouse. dbt Labs handles this in a secure way using encryption and authentication from the client’s data warehouse. In certain cases, such data may be cached on dbt Labs system ephemerally (data is not persistently stored). dbt Labs employees cannot access cached data during normal business operations and must have a business need and/or direct manager approval for access to the underlying infrastructure. Access would only be when necessary for providing a client services and never with the purpose of enriching dbt Labs. No client warehouse data is retained on dbt Labs's systems. We offer a caching solution to optimize query performance. The caching feature uses client data warehouse storage rather than being stored on dbt Labs’s systems. In addition, this feature is activated only through a client opt-in. Therefore, caching is always in client hands and at client discretion  Does our agreement, the Terms of Service (ToS) for dbt, apply to the Semantic Layer? Yes it does.  Where is MetricFlow hosted? How do queries pass through MetricFlow and dbt and back to the end user? MetricFlow is hosted in dbt. Requests from the [Semantic Layer APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md) are routed from our API gateway to MetricFlow, which generates the SQL to compute what's requested by the user. MetricFlow hands the SQL back to our gateway, which then executes it against the data platform.  How do I configure the dbt Semantic Layer? 1. You define [semantic models](https://docs.getdbt.com/docs/build/semantic-models.md) in YAML files that describe your data, including entities (for joins), measures (with aggregation types as a building block to your metrics), and dimensions (to slice and dice your metrics). 2. Then you build your metrics on top of these semantic models. This is all done in `.yml` configurations alongside your dbt models in your projects. 3. Once you've defined your metrics and semantic models, you can [configure the dbt Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md) in dbt. Read our [dbt Semantic Layer quickstart](https://docs.getdbt.com/guides/sl-qs.md) guide for more information.  How does caching work in the dbt Semantic Layer? Beginning in March 2024, the dbt Semantic Layer will offer two layers of caching: * The result cache, which caches query results in the data platform so that subsequent runs of the same query are faster. * A declarative cache which also lives in your data platform.  Does the dbt Semantic Layer expect all models to be in normalized format? No, the dbt Semantic Layer is flexible enough to work with many data modeling approaches including Snowflake, Star schemas, Data vaults, or other normalized tables.  How are queries optimized to not scan more data than they should? MetricFlow always tries to generate SQL in the most performant way, while ensuring the metric value is correct. It generates SQL in a way that allows us to add optimizations, like predicate pushdown, to ensure we don’t perform full table scans.  What are the latency considerations of using the dbt Semantic Layer? The latency of query runtimes is low, in the order of milliseconds.  What if different teams have different definitions? If the underlying metric aggregation is different, then these would be different metrics. However, if teams have different definitions because they're using specific filters or dimensions, it's still the same metric. They're just using it in different ways. This can be managed by adjusting how the metric is viewed in downstream tools or setting up [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md) to handle the various permutations of it. #### Build metrics and semantic models  Can I define my own aggregations? MetricFlow does not currently support custom aggregations on measures. You can find supported aggregation types [here](https://docs.getdbt.com/docs/build/measures.md#aggregation).  How are joins identified in the semantic model? [Joins](https://docs.getdbt.com/docs/build/join-logic.md) are identified through [entities](https://docs.getdbt.com/docs/build/entities.md) defined in a [semantic model](https://docs.getdbt.com/docs/build/semantic-models.md). These are the keys in your dataset. You can specify `foreign`, `unique`, `primary`, or `natural` joins. With multiple semantic models and the entities within them, MetricFlow creates a graph using the semantic models as nodes and the join paths as edges to perform joins automatically. MetricFlow chooses the appropriate join type and avoids fan-out or chasm joins with other tables based on the entity types. You can find supported join types [here](https://docs.getdbt.com/docs/build/join-logic.md#types-of-joins).  What is the benefit of “expr” used in semantic models and metric configurations? Expr (short for “expression”) allows you to put any arbitrary SQL supported by your data platform in any definition of a measure, entity, or dimension. This is useful if you want the object name in the semantic model to be different than what it’s called in the database. Or if you want to include logic in the definition of the component you're creating. The MetricFlow spec is deliberately opinionated, and we offer “expr” as an escape hatch to allow developers to be more expressive.  Do you support semi-additive metrics? Yes, we approach this by specifying a [dimension](https://docs.getdbt.com/docs/build/dimensions.md) that a metric cannot be aggregated across (such as `time`). You can learn how to configure semi-additive dimensions [here](https://docs.getdbt.com/docs/build/measures.md#non-additive-dimensions).  Can I use an entity as a dimension? Yes, while [entities](https://docs.getdbt.com/docs/build/entities.md) must be defined under “entities,” they can be queried like dimensions in downstream tools. Additionally, if the entity isn't used to perform joins across your semantic models, you may optionally define it as a dimension.  Can I test my semantic models and metrics? Yes! You can validate your semantic nodes (semantic models, metrics, saved queries) in a few ways: * [Query and validate you metrics](https://docs.getdbt.com/docs/build/metricflow-commands.md) in your development tool before submitting your code changes. * [Validate semantic nodes in CI](https://docs.getdbt.com/docs/deploy/ci-jobs.md#semantic-validations-in-ci) to ensure code changes made to dbt models don't break these metrics. #### Available integrations  What integrations are supported today? There are a number of data applications that have integrations with the dbt Semantic Layer, including Tableau, Google Sheets, Hex, Dot, and Mode, among others. Refer to [Available integrations](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) for more information.  How can I benefit from using the dbt Semantic Layer if my visualization tool is not currently supported? You can use [exports](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md) to materialize your metrics into a table or view in your data platform. From there, you can connect your visualization tool to your data platform. Although this approach doesn't provide the dynamic benefits of the dbt Semantic Layer, you still benefit from centralized metrics and from using MetricFlow configurations to define, generate, and compute SQL for your metrics.  Why should I use exports as opposed to defining a view within my data platform? Creating an [export](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md) allows you to bring your governed metric definitions into your data platform as a table or view. This means your metric logic is managed centrally in dbt, instead of as a view in your data platform and ensures that metric values remain consistent across all interfaces.  Can metric descriptions be viewed from third-party tools? Yes, all of our interfaces or APIs expose metric descriptions, which you can surface in downstream tools. #### Permissions and access  How do fine-grained access controls work with the dbt Semantic Layer? The dbt Semantic Layer uses service or personal tokens for authentication. [Service tokens](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) are mapped to underlying data platform credentials. These credentials control physical access to the raw data. The credential configuration allows admins to create a credential and map it to service tokens, which can then be shared to relevant teams for BI connection setup. You can configure credentials and service tokens to reflect your teams and their roles. Personal access tokens [(PATs)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) enable user-level authentication. When you use PATs to authenticate, your personal user credentials are used when running queries against the Semantic Layer. Currently, the credentials you configure when setting up the dbt Semantic Layer are used for every request. Any physical access policies you have tied to your credentials will be respected. #### Implementation  How can I implement dbt Mesh with the dbt Semantic Layer? When using the dbt Semantic Layer in a [dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md) setting, we recommend the following: * You have one standalone project that contains your semantic models and metrics. * Then as you build your Semantic Layer, you can [cross-reference dbt models](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md) across your various projects or packages to create your semantic models using the [two-argument `ref` function](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models) (`ref('project_name', 'model_name')`). * Your dbt Semantic Layer project serves as a global source of truth across the rest of your projects. (Applies to dbt v1.12 and later) Cross-project refs unsupported in latest SL YAML spec When using [dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md) with the [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md), [referencing models from different projects](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models) is only supported in the legacy YAML spec, where semantic models are defined as top-level resources and can reference models across projects. In the [latest YAML spec](https://docs.getdbt.com/docs/build/latest-metrics-spec.md), semantic models are defined within model YAML files, and cross-project references are not yet supported. Support for this capability in the latest spec is planned for a future release. ###### Usage example For example, let's say you have a public model (`fct_orders`) that lives in the `jaffle_finance` project. As you build your semantic model, use the following syntax to ref the model: models/metrics/semantic\_model\_name.yml ```yaml semantic_models: - name: customer_orders defaults: agg_time_dimension: first_ordered_at description: | Customer grain mart that aggregates customer orders. model: ref('jaffle_finance', 'fct_orders') # ref('project_name', 'model_name') entities: ...rest of configuration... dimensions: ...rest of configuration... measures: ...rest of configuration... ``` Notice that in the `model` parameter, we're using the `ref` function with two arguments to reference the public model `fct_orders` defined in the `jaffle_finance` project. <br />  Which ‘staging layer’ should the dbt Semantic Layer talk to? Raw, staging, or marts? We recommend to build your semantic layer on top of the [marts layer](https://docs.getdbt.com/best-practices/how-we-structure/4-marts.md), which represents the clean and transformed data from your dbt models.  Should semantic layer credentials mirror those for production environments? Or should they be different? Semantic layer credentials are different than the credentials you use to run dbt models. Specifically, we recommend a less privileged set of credentials since consumers are only reading data.  How does the dbt Semantic Layer support a dbt Mesh architecture design? Currently, semantic models can be created from dbt models that live across projects ([dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md)). In the future, users will also be able to use mesh concepts on semantic objects and define metrics across dbt projects. (Applies to dbt v1.12 and later) Cross-project refs unsupported in latest SL YAML spec When using [dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md) with the [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md), [referencing models from different projects](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models) is only supported in the legacy YAML spec, where semantic models are defined as top-level resources and can reference models across projects. In the [latest YAML spec](https://docs.getdbt.com/docs/build/latest-metrics-spec.md), semantic models are defined within model YAML files, and cross-project references are not yet supported. Support for this capability in the latest spec is planned for a future release. --- ### dbt State trial and billing Preview ### dbt State trial and billing [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Login required | Usage-based Start a dbt State trial and manage paid access from the dbt platform **Billing & Usage** dashboard. #### How the trial works * Eligible new organizations receive 30 days of free use with no usage limit. After the free period, a credit card or enterprise contract (for dbt platform managed plans) is required to continue. For more information, refer to [Setting up usage-based billing](https://docs.getdbt.com/docs/deploy/dbt-state-trial.md#setting-up-usage-based-billing). * To start a dbt State trial, you need a dbt account so you can manage dbt State usage, billing, and spend limits from one dashboard. A paid dbt platform plan is *not* required to use dbt State locally. * Once started, you cannot pause the trial. * If you were using state-aware orchestration prior to June 1, 2026, your dbt State trial will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. #### Starting your trial To start your 30-day trial, refer to the instructions in [Setting up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md). #### Setting up usage-based billing dbt State pricing is usage-based; you're billed per target table that dbt State reuses each day, *not* per dbt platform seat. You can set up billing during your trial or after it ends. Once your trial ends, dbt notifies your billing admin — they must set up billing to keep using dbt State. ###### Managed accounts If your account has consumption spend on contract, go to the **State** tab of the **Usage-based features** page and click **Allow** to bill against your committed spend. Otherwise, [contact the dbt Labs sales team](https://www.getdbt.com/contact). ###### Self-serve accounts 1. Go to **Billing & Usage** > **Usage-based features**. 2. Click **Set up billing** (if your trial is still active) or **Add billing info** (if your trial has ended). 3. In **Add a payment method**, enter your card details and billing address. 4. Click **Save card**. 5. Under the **State** tab, the card you added is displayed. Click **Confirm and activate**. After setting up billing, you can optionally [set a spend alert](#setting-spend-alerts) to get notified when your monthly costs reach a threshold. #### Setting spend alerts You can set a spend alert to get notified when your monthly dbt State costs reach a defined threshold. 1. In your dbt platform account, click your account name in the lower-left corner above your username and click **Account settings**. 2. Go to **Billing & Usage** > **Usage-based features**. 3. In the **Spend alert** section, click **Set a spend alert**. 4. Enable the toggle to receive email notifications when monthly spend reaches your threshold. 5. In the **Alert threshold** field, enter the amount in USD that triggers the alert. 6. Click **Save**. #### How billing works ##### Daily active target tables For purposes of pricing, daily active target tables (DATT) are measured as the number of distinct target tables (as defined below) for which dbt State performs at least one of the following unique operations on a given day (based on UTC time): a skip, clone, or test reuse. A target table is a database object managed by your dbt project for a given database and schema name. It includes seeds, snapshots, dbt models (including incremental models). It also includes each distinct test (even if the tests are not built into the database because `store_failures` is disabled). For example, if `stg_customers` has `not_null` and `unique` tests on its `id` column, that's three target tables: the model and its two tests. When you run `dbt build` or a similar command, a target table is selected for execution. It counts as an active target table if dbt State can reuse it based on your configuration rules. All reuses of the same active target table in a single day (based on UTC time) count as a single daily active target table (DATT). ##### Monthly cost calculation dbt State calculates cost per billing period using the unit price (USD $0.094) x sum of daily active target tables (DATT) for all account users and all days in that billing period. For example, if you have 100 DATT in a billing period, you'll be billed for 100 \* $0.094 = $9.40. For current unit price and more information, refer to the [dbt Labs Service Consumption Table](https://www.getdbt.com/legal/service-consumption-table). ##### DATT chart You can view your DATT count in **Account settings** > **Billing & Usage** > **Usage-based features**. Under the **State** tab, the DATT chart shows DATTs split into **Billable** and **Free**. During a trial, all DATTs are counted as free. ##### Cancellation Usage is tracked through your cancellation date. You're billed at month end for usage incurred before cancellation and not charged for usage after. #### Related docs * [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) * [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) * [dbt State usage and pricing](https://docs.getdbt.com/docs/platform/billing.md#dbt-state-usage) --- ### dbt support Support for dbt is available to all users through the following channels: * Dedicated dbt Support team (dbt users). * [The Community Forum](https://discourse.getdbt.com/). * [dbt Community slack](https://www.getdbt.com/community/join-the-community/). #### dbt support If you're developing on the command line (CLI) and have questions or need some help — reach out to the helpful dbt community through [the Community Forum](https://discourse.getdbt.com/) or [dbt Community slack](https://www.getdbt.com/community/join-the-community/). #### dbt platform support The global dbt Support team is available to dbt customers by [email](mailto:support@getdbt.com) or by clicking **Create a support ticket** through the dbt navigation. ##### Create a support ticket To create a support ticket in dbt: 1. In the dbt navigation, click on **Get resources**. 2. Click **Create a support ticket**. 3. Fill out the form and click **Create Ticket**. 4. A dbt Support team member will respond to your ticket through email. ##### Ask dbt Support Assistant dbt Support Assistant is an AI widget that provides instant, AI-generated responses to common questions. This feature is available to dbt users and can help answer troubleshooting questions, give a synopsis of features and functionality, or link to relevant documentation. The dbt Support Assistant AI widget is separate from [dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-platform.md), a powerful AI engine that helps with code generation to accelerate your analytics workflows. The dbt Support Assistant focuses on answering documentation and troubleshooting-related questions. Enabling or disabling AI features in dbt won't affect the dbt Support Assistant's availability. info We recommend validating information received in AI responses for any scenario using our documentation. Please [contact support](mailto:support@getdbt.com) to report incorrect information provided by the Support Assistant. ##### Support plans and resources We want to help you work through implementing and utilizing dbt platform at your organization. Have a question you can't find an answer to in [our docs](https://docs.getdbt.com/) or [the Community Forum](https://discourse.getdbt.com/)? Our Support team is here to `dbt help` you! * **Enterprise and Enterprise+ plans** — Priority [support](#severity-level-for-enterprise-support), optional premium plans, enhanced SLAs, implementation assistance, dedicated management, and dbt Labs security reviews depending on price point. * **Developer and Starter plans** — 24x5 support (no service level agreement (SLA); [contact Sales](https://www.getdbt.com/pricing/) for Enterprise plan inquiries). * **Support team help** — Assistance with [common dbt questions](https://docs.getdbt.com/category/troubleshooting.md), like project setup, login issues, error understanding, setup private packages, link to a new GitHub account, [how to generate a har file](https://docs.getdbt.com/faqs/Troubleshooting/generate-har-file.md), and so on. * **Resource guide** — Check the [guide](https://docs.getdbt.com/community/resources/getting-help.md) for effective help-seeking strategies. Example of common support questions Types of dbt cloud-based platform related questions our Support team can assist you with, regardless of your dbt plan:<br /><br />**How do I...**<br /> * set up a dbt project? <br /> * set up a private package in dbt? <br /> * configure custom branches on git repos? <br /> * link dbt to a new GitHub account? <br /> <br /> **Help! I can't...** <br /> * log in. <br /> * access logs. <br /> * update user groups. <br /> <br /> **I need help understanding...** <br /> * why this run failed. <br /> * why I am getting this error message in dbt? <br /> * why my CI jobs are not kicking off as expected. <br /> #### dbt Enterprise accounts Basic assistance with dbt project troubleshooting. Help with errors and issues in macros, models, and dbt Labs' packages. For strategic advice, best practices, or expansion conversations, consult your Account team. For customers on a dbt Enterprise-tier plan, we **also** offer basic assistance in troubleshooting issues with your dbt project: * **Something isn't working the way I would expect it to...** * in a macro I created... * in an incremental model I'm building... * in one of dbt Labs' packages like dbt\_utils or audit\_helper... * **I need help understanding and troubleshooting this error...** * `Server error: Compilation Error in rpc request (from remote system) 'dbt_utils' is undefined` * `SQL compilation error: syntax error line 1 at position 38 unexpected '<EOF>'.` * `Compilation Error Error reading name_of_folder/name_of_file.yml - Runtime Error Syntax error near line 9` Types of questions you should ask your Account team: * How should we think about setting up our dbt projects, environments, and jobs based on our company structure and needs? * I want to expand my account! How do I add more people and train them? * Here is our data road map for the next year - can we talk through how dbt fits into it and what features we may not be utilizing that can help us achieve our goals? * It is time for our contract renewal, what options do I have? ##### Severity level for Enterprise support Support tickets are assigned a severity level based on the impact of the issue on your business. The severity level is assigned by dbt Labs, and the level assigned determines the priority level of support you will receive. For specific ticket response time or other questions that relate to your Enterprise or Enterprise+ account’s SLA, please refer to your Enterprise contract. | Severity Level | Description | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Severity Level 1 | Any Error which makes the use or continued use of the Subscription or material features impossible; Subscription is not operational, with no alternative available. | | Severity Level 2 | Feature failure, without a workaround, but Subscription is operational. | | Severity Level 3 | Feature failure, but a workaround exists. | | Severity Level 4 | Error with low-to-no impact on Client’s access to or use of the Subscription, or Client has a general question or feature enhancement request. | #### Leave feedback Leave feedback or submit a feature request for dbt or dbt v1. ###### Share feedback or feature request for the dbt platform 1. In the dbt navigation, click **Leave feedback**. 2. In the **Leave feedback** pop up, fill out the form. 3. Upload any relevant files to the feedback form (optional). 4. Confirm if you'd like dbt Labs to contact you about the feedback (optional). 5. Click **Send Feedback**. [![Leave feedback in dbt](/img/docs/leave-feedback.gif?v=2 "Leave feedback in dbt")](#)Leave feedback in dbt ###### Share feedback or feature request for dbt * [Create a GitHub issue here](https://github.com/dbt-labs/dbt/issues). #### External help For SQL writing, project performance review, or project building, refer to dbt Preferred Consulting Providers and dbt Labs' Services. For help writing SQL, reviewing the overall performance of your project, or want someone to actually help build your dbt project, refer to the following pages: * List of [dbt Consulting Partners](https://www.getdbt.com/partner-directory). * dbt Labs' [Services](https://www.getdbt.com/dbt-labs/services/). --- ### dbt tips and tricks Use this page for valuable insights and practical advice to enhance your dbt experience. Whether you're new to dbt or an experienced user, these tips are designed to help you work more efficiently and effectively. The following tips are organized into the following categories: * [Package tips](#package-tips) to help you streamline your workflow. * [Advanced tips and techniques](#advanced-tips-and-techniques) to help you get the most out of dbt. If you're developing with the Studio IDE, you can refer to the [keyboard shortcuts](https://docs.getdbt.com/docs/platform/studio-ide/keyboard-shortcuts.md) page to help make development more productive and easier for everyone. #### YAML tips This section clarifies where you can use [Jinja](https://docs.getdbt.com/docs/build/jinja-macros.md), nest [vars](https://docs.getdbt.com/reference/dbt-jinja-functions/var.md) and [`env_var`](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md) in your YAML files. * You can use Jinja in almost every YAML file in dbt *except* the [`dependencies.yml` file](https://docs.getdbt.com/docs/build/packages.md#use-cases). This is because the `dependencies.yml` file doesn't support Jinja. * Use `vars` in any YAML file that supports Jinja (like `schema.yml`, `snapshots.yml`). However, note that: * In `dbt_project.yml`, `packages.yml`, and `profiles.yml` files, you must pass `vars` through the CLI using `--vars`, not defined inside the `vars:` block in the YAML file. This is because these files are parsed before Jinja is rendered. * You can use `env_var()` in all YAML files that support Jinja. Only `profiles.yml` and `packages.yml` support environment variables for secure values (using the `DBT_ENV_SECRET_` prefix). These are masked in logs and intended for credentials or secrets. For additional information, check out [dbt v1's context docs](https://github.com/dbt-labs/dbt/blob/1.latest/core/dbt/context/README.md). #### Package tips Leverage these dbt packages to streamline your workflow: | Package | Description | | --------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [`dbt_codegen`](https://hub.getdbt.com/dbt-labs/codegen/latest/) | Use the package to help you generate YML files for your models and sources and SQL files for your staging models. | | [`dbt_utils`](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/) | The package contains macros useful for daily development. For example, `date_spine` generates a table with all dates between the ones provided as parameters. | | [`dbt_project_evaluator`](https://hub.getdbt.com/dbt-labs/dbt_project_evaluator/latest) | The package compares your dbt project against a list of our best practices and provides suggestions and guidelines on how to update your models. | | [`dbt_expectations`](https://hub.getdbt.com/metaplane/dbt_expectations/latest/) | The package contains many tests beyond those built into dbt. | | [`dbt_audit_helper`](https://hub.getdbt.com/#:~:text=adwords-,audit_helper,-codegen) | The package lets you compare the output of 2 queries. Use it when refactoring existing logic to ensure that the new results are identical. | | [`dbt_artifacts`](https://hub.getdbt.com/brooklyn-data/dbt_artifacts/latest) | The package saves information about your dbt runs directly to your data platform so that you can track the performance of models over time. | | [`dbt_meta_testing`](https://hub.getdbt.com/tnightengale/dbt_meta_testing/latest) | This package checks that your dbt project is sufficiently tested and documented. | #### Advanced tips and techniques * Use your folder structure as your primary selector method. `dbt build --select marts.marketing` is simpler and more resilient than relying on tagging every model. * Think about jobs in terms of build cadences and SLAs. Run models that have hourly, daily, or weekly build cadences together. * Use the [where config](https://docs.getdbt.com/reference/resource-configs/where.md) for tests to test an assertion on a subset of records. * [store\_failures](https://docs.getdbt.com/reference/resource-configs/store_failures.md) lets you examine records that cause tests to fail, so you can either repair the data or change the test as needed. * Use [severity](https://docs.getdbt.com/reference/resource-configs/severity.md) thresholds to set an acceptable number of failures for a test. * Use [incremental\_strategy](https://docs.getdbt.com/docs/build/incremental-strategy.md) in your incremental model config to implement the most effective behavior depending on the volume of your data and reliability of your unique keys. * Set `vars` in your `dbt_project.yml` to define global defaults for certain conditions, which you can then override using the `--vars` flag in your commands. * Use [for loops](https://docs.getdbt.com/guides/using-jinja.md?step=3) in Jinja to DRY up repetitive logic, such as selecting a series of columns that all require the same transformations and naming patterns to be applied. * Instead of relying on post-hooks, use the [grants config](https://docs.getdbt.com/reference/resource-configs/grants.md) to apply permission grants in the warehouse resiliently. * Define [source-freshness](https://docs.getdbt.com/docs/build/sources.md#source-data-freshness) thresholds on your sources to avoid running transformations on data that has already been processed. * Use the `+` operator on the left of a model `dbt build --select +model_name` to run a model and all of its upstream dependencies. Use the `+` operator on the right of the model `dbt build --select model_name+` to run a model and everything downstream that depends on it. * Use `dir_name` to run all models in a package or directory. * Use the `@` operator on the left of a model in a non-state-aware CI setup to test it. This operator runs all of a selection’s parents and children, and also runs the parents of its children, which in a fresh CI schema will likely not exist yet. * Use the [--exclude flag](https://docs.getdbt.com/reference/node-selection/exclude.md) to remove a subset of models out of a selection. * Use the [--full-refresh](https://docs.getdbt.com/reference/commands/run.md#refresh-incremental-models) flag to rebuild an incremental model from scratch. * Use [seeds](https://docs.getdbt.com/docs/build/seeds.md) to create manual lookup tables, like zip codes to states or marketing UTMs to campaigns. `dbt seed` will build these from CSVs into your warehouse and make them `ref` able in your models. * Use [target.name](https://docs.getdbt.com/docs/build/custom-schemas.md#an-alternative-pattern-for-generating-schema-names) to pivot logic based on what environment you’re using. For example, to build into a single development schema while developing, but use multiple schemas in production. #### Related docs * [Quickstart guide](https://docs.getdbt.com/guides.md) * [About dbt](https://docs.getdbt.com/docs/platform/about-platform/dbt-platform-features.md) * [Develop in the Cloud](https://docs.getdbt.com/docs/platform/about-develop-dbt.md) --- ### dbt v2 networking requirements Preview ### dbt v2 networking requirements [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt v2 requires outbound HTTPS access to several endpoints depending on your usage. This page describes each requirement and provides guidance for enterprise environments that restrict outbound traffic. The following table summarizes all endpoints. See each section below for details. | Resource | URL | Required for | | ----------------------------------------- | ---------------------------------------------- | --------------------------- | | [Adapter drivers](#adapter-drivers) | `https://public.cdn.getdbt.com` | All users | | [Telemetry](#telemetry) | `https://p.vx.dbt.com` | All users (can be disabled) | | [Manifest downloads](#manifest-downloads) | Cloud provider storage URLs (varies by region) | dbt platform users only | #### Adapter drivers The dbt v2 binary does *not* bundle database drivers. Instead, dbt v2 automatically downloads the correct [ADBC](https://arrow.apache.org/adbc/) driver for your data platform the first time you run a dbt command (such as `dbt run`, `dbt debug`, or `dbt compile`). dbt v2 detects which driver you need based on your `profiles.yml` configuration and downloads it from the dbt Labs CDN. dbt v2 distributes all checksums with the binary itself to guarantee authenticity of the downloaded drivers. Adapter driver downloads require outbound HTTPS access to the dbt CDN: | Resource | URL | Purpose | | ------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | | **Adapter drivers** | `https://public.cdn.getdbt.com` | Downloads ADBC adapter driver libraries (`.dylib`, `.so`, `.dll`) on first use or when running `dbt system install-drivers` | info dbt v2 handles driver download automatically on first use. The `dbt system install-drivers` command downloads **all** supported drivers (Snowflake, BigQuery, Postgres, Databricks, Redshift, DuckDB, and Salesforce) at once. This is useful if you work across multiple data platforms and want to pre-cache every driver before going offline or switching projects. ##### Enterprise proxy considerations Adapter drivers are native shared libraries (`.dylib` on macOS, `.so` on Linux, `.dll` on Windows). Some enterprise proxy filters and security tools classify these file types as executables and may block the download — even if you allowlist `public.cdn.getdbt.com` at the domain level. If your organization's proxy blocks adapter driver downloads, work with your IT team to ensure both: 1. You allowlist the domain `public.cdn.getdbt.com`. 2. Content inspection rules permit downloading native library file types (`.dylib`, `.so`, `.dll`) from that domain. If you cannot change your proxy configuration, see [Restricted network installation](#restricted-network-installation). ##### Restricted network installation If your environment cannot access `public.cdn.getdbt.com` for adapter driver downloads, you can pre-build a bundle of the dbt v2 binary and the adapter drivers into a single `.tar.gz` or Docker image and host it on an internally approved fileshare. When you assemble a bundle for air-gapped or firewall-restricted machines: 1. Use the [version compatibility matrix](https://docs.getdbt.com/docs/dbt-versions/dbt-version-compatibility.md#compatibility-matrix) to choose a dbt v2 version that matches the dbt VS Code extension your users run. 2. Verify the binary against its published SHA-256 checksum before distributing it. Refer to [Verify binaries for manual and air-gapped installs](https://docs.getdbt.com/docs/dbt-versions/dbt-version-compatibility.md#verify-binaries-for-manual-and-air-gapped-installs). 3. Point the dbt VS Code extension at the bundled binary with the `dbt.fusionPath` setting. Refer to [dbt extension settings](https://docs.getdbt.com/docs/configure-dbt-extension.md#dbt-extension-settings). For supported adapters, refer to [v2 requirements](https://docs.getdbt.com/docs/dbt/supported-features.md#requirements). #### Telemetry dbt v2 sends anonymous usage statistics to help improve the product. If the telemetry endpoint is unreachable (for example, blocked by a firewall or proxy), dbt v2 logs errors on each invocation. | Resource | URL | Purpose | | ------------- | ---------------------- | -------------------------------- | | **Telemetry** | `https://p.vx.dbt.com` | Sends anonymous usage statistics | To suppress these errors without allowlisting the URL, disable telemetry using the options described in [Anonymous usage stats](https://docs.getdbt.com/reference/global-configs/usage-stats.md). #### Manifest downloads (dbt platform only) [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") For [dbt platform](https://docs.getdbt.com/docs/platform/about-platform/dbt-platform-features.md) customers using dbt v2 locally, dbt v2 downloads production manifests from dbt platform to enable features like [deferral](https://docs.getdbt.com/reference/node-selection/defer.md) and [cross-project references](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md). The [cloud storage provider](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) hosting your dbt platform cell serves these manifests via **pre-signed URLs**. The specific hostnames depend on your dbt platform deployment region and the underlying cloud provider. To ensure dbt v2 can download manifests, allowlist the appropriate storage domain for your region: | Cloud provider | URL pattern | Example | | ------------------------ | ----------------------------------------- | ----------------------------------------------- | | **AWS (S3)** | `https://s3.<region>.amazonaws.com` | `https://s3.ap-northeast-1.amazonaws.com` (JP1) | | **Azure (Blob Storage)** | `https://<account>.blob.core.windows.net` | `https://prodeu2.blob.core.windows.net` (EU2) | | **GCP (Cloud Storage)** | `https://storage.googleapis.com` | `storage.googleapis.com` | Because pre-signed URLs contain region and account-specific hostnames that may change over time, we recommend allowlisting the **base storage domain** for your cloud provider rather than individual URLs: * **AWS** — `s3.*.amazonaws.com` * **Azure** — `*.blob.core.windows.net` * **GCP** — `storage.googleapis.com` --- ### dbt VS Code extension features Preview ### dbt VS Code extension features [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Local development The dbt VS Code extension uses a dynamic Language Server Protocol (LSP) to provide a fast, intelligent, and cost-efficient dbt development experience with enhanced workflows and easy navigation. Registration for advanced features All dbt VS Code extension features are available to all users for 14 days. After the 14-day trial period, most features remain available without registration. To keep using advanced features, [register](https://docs.getdbt.com/docs/sign-in-dbt-extension.md) for a free dbt platform account. Existing registered users keep access automatically. See the [feature availability](#feature-availability) table for the full list of features and what each needs. (Applies to dbt v1.13 and later) In dbt v2.0 and later, authentication for registration and features that require sign-in is handled by [`dbt login`](https://docs.getdbt.com/reference/commands/login.md?version=2.0), so your login state is shared across dbt tools like the dbt VS Code extension and, in supported versions, dbt State. #### Feature availability The dbt VS Code extension is free to install. All features work for 14 days with no login; after that, the vast majority keep working, and a few advanced features need a free dbt platform account (sign in or register with your email, or run `dbt login`). | Feature | Works without login/registration | Register or login<br />Any dbt platform account, free or paid | | ---------------------------------------------------------- | -------------------------------- | ------------------------------------------------------------- | | Error diagnostics for Jinja, YAML, and SQL syntax | ✅ | ✅ | | Jinja LSP go-to ref, source, and macro | ✅ | ✅ | | Linter warning diagnostics | ✅ | ✅ | | Table-level lineage | ✅ | ✅ | | Basic dbt command UI (run, build, test, and query results) | ✅ | ✅ | | Ref autocomplete | ✅ | ✅ | | Refactor ref and column names | ✅ | ✅ | | Dialect-aware function autocomplete | ✅ | ✅ | | SQL type and schema error diagnostics | - | ✅ | | Preview CTE | - | ✅ | | Query cache for faster incremental compiles | - | ✅ | | Model docs tab with platform metadata | - | ✅ | | Column-level lineage | - | ✅ | | Compare changes | - | ✅ | | SQL LSP go-to column and CTE | - | ✅ | | SQL LSP hover to see the schema for `select *` | - | ✅ | #### Lightning-fast parse times Parse even the largest projects up to 30x faster than with dbt v1. **Requires registration** — LSP query cache (for faster incremental compiles). [](/img/docs/extension/zoomzoom.mp4) #### View compiled code Get a live view of the SQL code your models will build — right alongside your dbt code. Usage: * Click the **code icon** to view compiled code side-by-side with source code. * Compiled code will update as you save your source code. * Clicking on a dbt macro will focus the corresponding compiled code. * Clicking on a compiled code block will focus the corresponding source code. [](/img/docs/extension/compiled-code.mp4) #### Build flexibly Use the command palette to quickly build models using complex selectors. Usage: * Click the **dbt icon** or use keyboard shortcut `cmd+shift+enter` (macOS) / `ctrl+shift+enter` (Windows/Linux) to launch a quickpick menu. * Select a command to run. [](/img/docs/extension/build-flexibly.mp4) #### Live error detection Automatically validate your SQL code to detect errors and surface warnings without hitting the warehouse. **Available to all users:** Syntax-tree diagnostics for Jinja, YAML, and SQL syntax errors (L1): * Syntax errors (missing commas, misspelled keywords, and more) * Hover over red squiggles to display errors * Full diagnostic information is available in the **Problems** panel **Requires registration:** L2 dbt v2 SQL comprehension diagnostics (depends on strict static analysis): * Missing `group by` clauses, or columns that are neither grouped nor aggregated * Invalid function names or arguments * SQL type and schema errors * Linter warning diagnostics [](/img/docs/extension/live-error-detection.mp4) #### Powerful IntelliSense Autocomplete SQL functions, model names, macros, and more. **Available to all users:** * Autocomplete `ref`s and `source` calls. For example, type `{{ ref(` or `{{ source(` and you will see a list of available resources and their type complete the function call. Autocomplete doesn't trigger when replacing existing model names inside parentheses. * Dialect-aware SQL function autocomplete [![Example of the VS Code extension IntelliSense](/img/docs/extension/vsce-intellisense.gif?v=2 "Example of the VS Code extension IntelliSense")](#)Example of the VS Code extension IntelliSense #### Instant refactoring Rename models or columns and see references update project-wide. **Available to all users:** Renaming models: * Right-click on a file in the file tree and select **Rename**. * After renaming the file, you'll get a prompt asking if you want to make refactoring changes. * Select **OK** to apply the changes, or **Show Preview** to display a preview of refactorings. * After applying your changes, `ref`s should be updated to use the updated model name. **Requires registration:** Renaming columns (depends on strict static analysis): Column renaming depends on strict static analysis, which validates column references across your project before the extension updates downstream models. * Right-click on a column alias and select **Rename Symbol**. * After renaming the column, you'll get a prompt asking if you want to make refactoring changes. * Select **OK** to apply the changes, or **Show Preview** to show a preview of refactorings. * After applying your changes, downstream references to the column should be updated to use the new column name. Note: Renaming models and columns is not yet supported for snapshots, or any resources defined in a .yml file. [](/img/docs/extension/refactor.mp4) #### Go-to-definition and reference Jump to the definition of any `ref`, macro, model, or column with a single click. Particularly useful in large projects with many models and macros. Excludes definitions from installed packages. **Available to all users:** * Command or Ctrl-click to go to the definition for an identifier. * Right-click an identifier and select **Go to Definition** or **Go to References**. * Jinja LSP go-to-definition for `ref()`, `source()`, and macros. **Requires registration:** Column and CTE go-to-definition (depends on `strict` static analysis): * Go-to-definition for column names * Go-to-definition for CTE names [](/img/docs/extension/go-to-definition.mp4) #### Rich lineage in context See lineage at the column or table level as you develop — no context switching or breaking flow. **Available to all users:** Table-level lineage: Using the lineage tab in Cursor If you're using the dbt VS Code extension in Cursor, the lineage tab works best in Editor mode and doesn't render in Agent mode. If you're in Agent mode and the lineage tab isn't rendering, just switch to Editor mode to view your project's table and column lineage. View table lineage: * Open the **Lineage** tab in your editor. It will reflect table lineage focused on the currently-open file. * Double-click nodes to open the files in your editor. * The lineage pane updates as you navigate the files in your dbt project. * Right-click on a node to update the DAG, or view column lineage for a node. **Requires registration:** Column-level lineage (depends on strict static analysis): View column lineage: * Right-click on a filename, or in the SQL contents of a model file. * Select **dbt: View Lineage** --> **Show column lineage**. * Select the column to view lineage for. * Double-click on a node to update the DAG selector. * You can also use column selectors in the lineage window by adding the `column:` prefix and appending the column name. [](/img/docs/extension/lineage.mp4) #### Hover insights See context on tables, columns, and functions without leaving your code. Simply hover over any SQL element to see details like column names and data types. **Requires registration:** Hover insights depend on strict static analysis, which lets the extension understand column types and function signatures across your project. Usage: * Hover over `*` to see expanded list of columns and their types. * Hover over column name or alias to see its type. [](/img/docs/extension/hover-insights.mp4) #### Live preview for models and CTEs Preview query output directly from inside your editor for faster validation and debugging. **Available to all users:** * Click the **table icon** or use keyboard shortcut `cmd+enter` (macOS) / `ctrl+enter` (Windows/Linux) to preview query results for a model or selected SQL snippet. * Results are displayed in the **Query Results** tab in the bottom panel. * The preview table is sortable and results are stored until the tab is closed. **Requires registration:** CTE preview: * Click the **Preview CTE** codelens to preview CTE results. [](/img/docs/extension/preview-cte.mp4) #### Explore your catalog [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") **Requires registration:** Open the **Catalog** tab to see information for the model you're working on — enriched by your dbt platform account — without leaving your editor. For the current model, the catalog tab surfaces: * The build status, last build time, and run duration from the dbt platform. * The model's **Description**. * The model's **Columns**, including each column's type, description, and test results. Sort columns alphabetically or by test name. * A **View in dbt platform** link to open the resource in the dbt platform. The **Catalog** tab is an advanced feature. Before your 14-day trial expires, [register](https://docs.getdbt.com/docs/sign-in-dbt-extension.md) for a dbt platform account to continue using it. [![Example of the Catalog tab in the dbt VS Code extension](/img/docs/extension/vsce-catalog-tab.png?v=2 "Example of the Catalog tab in the dbt VS Code extension")](#)Example of the Catalog tab in the dbt VS Code extension #### Generate a system report Generate a system report to collect your VS Code extension logs and system information into a zip file. This is useful when troubleshooting issues with the dbt VS Code extension. You can share the zip file with dbt Labs support to help diagnose problems. To generate and download a system report: 1. Open the Command Palette (`Cmd+Shift+P` on macOS, `Ctrl+Shift+P` on Windows/Linux). 2. Search for and select **dbt: Generate System Report**. 3. Choose a location to save the .zip file when prompted. 4. A notification will confirm where the file was saved. #### Compare changes in development [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") Requires registration Advanced capabilities are available to all users for 14 days. After the 14-day trial, [sign in or register](https://docs.getdbt.com/docs/sign-in-dbt-extension.md) for a dbt platform account to keep using advanced features. Existing registered dbt VS Code extension users keep access automatically. (Applies to dbt v2.0 and later) Authentication is handled by [`dbt login`](https://docs.getdbt.com/reference/commands/login.md?version=2.0), so your login state is shared across the CLI, dbt VS Code extension, and . You can use compare changes, powered by dbt v2, in your local development environment to compare your current working copy against your `manifest.json` (for example, your last production state) directly in your editor. For more details on how to use this feature, refer to [Compare changes in local development](https://docs.getdbt.com/docs/dbt/vs-compare-changes.md). [![Example of the Compare tab](/img/docs/extension/vs-compare-changes.png?v=2 "Example of the Compare tab")](#)Example of the Compare tab --- ### dbt Wizard billing and AI access FAQs Common questions about AI being turned on by default, how dbt Wizard usage is measured, what your usage credits covers, and how spend limits work. What's changing from September 1, 2026 From September 1, 2026, a couple of things are changing for dbt AI features: * **AI features are being enabled by default.** They're already on for new accounts and are rolling out soon to existing accounts. If your organization opted out, they'll remain off. Admins can turn AI features on or off anytime in **Account settings**. * **dbt Wizard is moving to usage-based billing** for [dbt-managed AI](#dbt-managed-inference). Usage is metered per token against your consumption pool, and an admin can set a monthly spend limit in dbt platform. #### AI enabled by default AI features are being enabled by default for dbt platform accounts. They're already on for new accounts and are rolling out soon to existing accounts. If your organization opted out, they'll remain off. Admins can turn AI features on or off anytime in **Account settings**. Turning AI on doesn't create a charge on its own — refer to [Billing FAQs](#billing-faqs) in the next section to understand how usage is metered.  Which AI features are enabled by default? The following surfaces are on by default: * dbt Wizard in Studio IDE * dbt Wizard home tab * dbt Copilot in dbt platform (includes Canvas and Insights). * Any future dbt AI features will automatically become available as well.  AI features aren't on for my account yet. How do I turn them on? AI features are already on for new accounts and are rolling out soon to existing accounts, so they may not be on for your account right away. An account admin can turn them on now in **Account settings** — refer to [Manage AI features in dbt platform](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md). If your organization opted out, they'll remain off until an admin turns them on.  I previously asked for AI to be permanently disabled. Will it turn on anyway? No. If your organization already opted out of AI features contractually or had them permanently disabled, they stay off. You don't need to do anything before September 1, 2026.  Can I opt out of AI features? Yes. An account admin can turn AI off at any time in **Account settings**. Refer to [Manage AI features in dbt platform](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md) for the steps — the same toggle controls both dbt Wizard and dbt Copilot.  If AI is enabled by default, will I be charged automatically? No. Enabling AI doesn't authorize paid usage by itself. dbt Wizard usage draws from your included consumption pool or trial pool. On Developer, Starter, and self-hosted plans, dbt-managed dbt Wizard pauses once that pool is used up, and going beyond it requires explicit purchase. Enterprise and Enterprise+ accounts should add a committed spend amount to their contract to keep using dbt Wizard. You may lose access to dbt Wizard without this commit in place. If you've set an optional monthly dbt Wizard spend limit, that still applies and pauses usage once reached. If you keep AI disabled, you incur no AI charges after September 1, 2026.  Which AI features use consumption-based billing? Only dbt Wizard with dbt-managed inference, across dbt Wizard in dbt platform and the dbt Wizard CLI. dbt Copilot stays on its existing actions-based model and isn't moving to consumption-based billing.  How do I check whether AI is enabled and what my account has used? An account admin will be able to check the AI toggle in **Account settings**. To see usage and remaining credit from September 1st, 2026, go to **Account settings** > **Billing & Usage**. The overview and the dbt Wizard usage-based feature page will show your consumption pool, amount used and remaining, and the reset date. Historical dbt Copilot Actions usage appears there too.  Who do I contact about AI access or usage limits? Contact your dbt Labs account team for questions about enabling or disabling AI features, purchasing additional usage credits, or contract-specific billing questions. If you're on Developer or Starter plan, [reach out to dbt Support](mailto:support@getdbt.com) for help. #### Billing FAQs The following questions cover how dbt-managed inference is metered for dbt Wizard usage, what your plan's consumption pool will include, and how you can track and cap your spend once Wizard usage-based billing goes live on September 1st, 2026. ##### Wizard usage overview  How is Wizard usage measured and priced? Wizard usage with the dbt-managed inference will be measured per token. Every token is processed as input or output counts. Each model has its own unique per-token pricing. Cost will depend on the model used, prompt length and complexity, and response size. Refer to [Model Provider Rate table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) and [Service Consumption Table](https://www.getdbt.com/legal/service-consumption-table) for the currently available models.  What is the dbt Wizard usage consumption pool? The dbt Wizard consumption pool is the usage balance available when using dbt-managed inference for dbt Wizard — this includes Developer and Starter plan trial Wizard usage credits and the monthly Enterprise/Enterprise+ included usage credits. As you use dbt Wizard, your token usage will automatically convert into a dollar amount and is deducted from your active usage credit. Once your usage credit is used up, additional usage draws from a purchased consumption pool, if one exists. What happens next depends on your plan: * **Developer, Starter, and self-hosted:** dbt Wizard usage pauses until you add paid usage or your pool refreshes. * **Enterprise and Enterprise+**: Accounts should add a committed spend amount to their contract to keep using dbt Wizard. You may lose access to dbt Wizard without this commit in place. If you've set an optional monthly dbt Wizard spend limit, that still applies and pauses usage once reached.  Is the Wizard consumption pool shared between dbt Wizard in the dbt platform and Wizard CLI? Yes. The consumption pool and usage credits will be shared across all users within an account and across Wizard in dbt platform and local Wizard CLI. Usage from either surface draws from the same account-level consumption pool.  Which AI models are available for use through dbt managed inference? Refer to the [Model Provider Rate table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) and [Service Consumption Table](https://www.getdbt.com/legal/service-consumption-table) for the models available as of September 1, 2026. dbt Labs bills usage of these models through your dbt account. ##### Developer and Starter plan (self-serve free trial)  What does the dbt Wizard Developer and Starter plan free trial include? Developer and Starter plans get $100 in usage credits for dbt-managed Wizard inference, free for 30 days. The $100 is *per account, not per user*, which means everyone on the account shares the same credits. On Developer, that's a single user, since it's a single-user plan. The credit covers Wizard usage across both the dbt platform and the CLI, and the trial ends when your account hits $100 in usage or 30 days, whichever comes first. The trial credits can only be spent on dbt Wizard, not on dbt State or other consumption-based features.  Who is eligible to start a dbt Wizard 30-day free trial? Developer or Starter plan accounts are eligible for the 30-day, $100/account free trial. Trials require a business email address, so personal email domains such as Gmail aren't eligible. An account admin or billing admin must start the trial.  Do unused trial consumption pools roll over or expire? Unused trial usage credits don't roll over. Your trial ends when you use up the entire $100 usage credit or 30 days pass, whichever happens first.  What happens if I deplete my trial usage credit before the 30-day trial ends? Your trial ends when you reach 30 days or use up the full $100 credits, whichever comes first. Because the credit is shared across the account, usage from any user on it counts toward the same $100. To continue using dbt managed providers, you will need to add a payment method and set a monthly spend limit. You can also continue with your own AI provider ("Bring Your Own Key (BYOK)") if you configure credentials for a supported AI provider.  What happens when my dbt Wizard trial ends? dbt-managed Wizard usage pauses unless paid access is configured by purchasing additional consumption pools. Self-serve accounts will be able to add a payment method and choose a monthly spend limit. Enterprise and Enterprise+ accounts should contact their dbt Labs account team. BYOK usage remains separate and is billed by your provider.  Will dbt automatically charge me when my trial ends? No. Starting a trial doesn't automatically create paid usage. For self-serve access, you must add a payment method and choose a spend limit. If you set up payment while the trial is active, paid usage begins only after the trial ends or its credit is exhausted. ##### Enterprise & Enterprise+ plans  What does the dbt Wizard Enterprise plan monthly usage credits include? Enterprise plans automatically include $100/month in usage credits at no cost, and Enterprise+ includes $200/month. These amounts are *per account, not per user*, which means a 5-person account and a 500-person account both get the same monthly credit, and everyone on the account draws from the same shared balance. No billing setup or opt-in is required to receive the monthly included consumption usage credits as it renews each billing period and doesn't roll over. These included monthly credits can only be spent on dbt Wizard. They can't be used for dbt State or any other consumption-based feature.  What happens when your account uses up its monthly usage credits? Enterprise and Enterprise+ account should add a committed spend amount to their contract to keep using dbt Wizard. You may lose access to dbt Wizard without this commit in place. If you've set an optional monthly dbt Wizard spend limit, that still applies and pauses usage once reached. If you have a committed spend amount, usage beyond your included credits draws from that commitment and appears on your invoice. BYOK usage is unaffected because your AI provider bills it separately. ##### Consumption pool add-on  How does the consumption pool work? It's the balance that covers dbt-managed inference usage (which must be purchased once you've used up any freely available monthly or trial usage credits that may be available), metered per token at cost. Pool dollars don't roll over at the end of a committed term. Unlike free Wizard usage credits, purchased committed spend isn't limited to dbt Wizard — it covers both dbt Wizard and dbt State.  Do I pay the full consumption pool or only for what I use? It depends on how you purchase dbt Wizard: * **Pay-as-you-go (self-service):** You pay only for actual dbt-managed Wizard usage, up to your selected spend limit. The spend limit is a cap, not a prepaid charge. Typically for Developer, Starter, and self-hosted plans. * **Pre-committed spend:** You commit to a specific amount upfront through your account team and are billed for that amount. Your usage is deducted from the committed amount as you use dbt Wizard. Typically for Enterprise-tiered plans. Talk to your account team to set up a pre-committed spend.  Who can set or change the dbt Wizard consumption pool limit? An account admin or billing admin can manage Wizard billing and spend controls. In the dbt platform, go to **Billing & Usage** > **Usage-based features** > **Wizard** to view or update the limit.  Is the consumption pool for dbt Wizard also shared with dbt State or dbt Copilot? It depends on which credits you're using: * Free dbt Wizard usage credits (The Developer and Starter trial pool, and the Enterprise and Enterprise+ monthly included usage credits) are scoped to dbt Wizard only. * Consumption pool add-on that you purchase covers both dbt Wizard and dbt State, so usage from either feature draws down the same account-level pool. Either way, Copilot Actions are metered separately on an actions-based model and never touch your dbt Wizard consumption pool. dbt Wizard also has its own feature-level spend limit, configured separately from dbt State. ##### Tracking usage & spend limits  Do I need a paid dbt plan or credit card to try dbt Wizard? No. You need a free dbt account to manage usage, billing, and spend limits, but you don't need a paid dbt platform plan or credit card to start the trial. If you don't have an account, you can [create one](https://www.getdbt.com/signup) during setup.  How can I track my dbt Wizard usage and remaining trial credit? From September 1st, 2026, you'll be able to track in dbt platform by going to **Account settings** > **Billing & Usage**. The overview and Wizard usage-based feature pages will show your current consumption pool/usage, trial balance, and spend controls across the platform and CLI.  Where can I find the current token rates for each supported model? Refer to the [Model Provider Rate Table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) and [Service Consumption Table](https://www.getdbt.com/legal/service-consumption-table), which includes the current input, cache-write, cached-read, and output token rates. Rates vary by model and can change over time.  How does the spend limit work? You'll be able to set a monthly spend limit for dbt Wizard in dbt platform. You'll be alerted as you approach it, and usage pauses if you reach it until the limit is raised or the next billing period begins. ##### Bring Your Own Key (BYOK)  Does bring your own key (BYOK) usage consume dbt Wizard consumption pools? No. With BYOK, your AI provider bills you directly. BYOK usage doesn't draw from your dbt-managed consumption pools.  How does BYOK work? With BYOK, you connect your own AI provider credentials and pay the provider directly. BYOK usage doesn't consume your dbt Wizard consumption pool. Refer to the [BYOK setup guide](https://docs.getdbt.com/docs/platform/wizard-byok-platform.md) for configuration details. Refer to [Model Provider Rate table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) and [Service Consumption Table](https://www.getdbt.com/legal/service-consumption-table) for more information. #### dbt-managed inference  Why use dbt-managed inference instead of bringing my own key? With dbt-managed inference, there's nothing to configure or maintain. dbt Labs selects and maintains the underlying models for cost, speed, and accuracy, so your team focuses on data work, not agent upkeep. Usage is billed through your existing dbt account and covered by your consumption pool, so there's one bill instead of a second vendor relationship to manage.  Which models are available with dbt-managed inference, and who picks them? dbt-managed inference includes several frontier models, including models from OpenAI and Anthropic, plus a set of open weight models. dbt Labs maintains and updates this list, so new models become available without you having to evaluate or configure a new provider yourself. Refer to [Model Provider Rate table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) and [Service Consumption Table](https://www.getdbt.com/legal/service-consumption-table) for more information.  Do I need to worry about rate limits or provider outages with dbt-managed inference? No — dbt manages the underlying provider relationships and infrastructure for dbt-managed inference. You interact with a single consumption pool and spend limit in dbt platform, regardless of which model is handling a given request.  Can I mix dbt-managed inference and BYOK? Yes. BYOK usage is billed by your provider and never draws from your dbt-managed consumption pool, so you can use dbt-managed inference for some work and BYOK for other work without either affecting the other's usage or billing.  Is dbt-managed inference more expensive than using my own provider key? Cost depends on the model and your usage pattern. dbt-managed inference is metered per token at the model's rate, while BYOK usage is billed directly by your provider at their own rates. Compare the two based on which models and volume you expect to use. Refer to [Model Provider Rate table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) and [Service Consumption Table](https://www.getdbt.com/legal/service-consumption-table) for more information.  Does read-only user usage in Explore mode count against my consumption pool? Yes. Questions asked in [Explore mode](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md#agent-modes) — including by [read-only users](https://docs.getdbt.com/docs/platform/wizard-read-only-users.md) — use dbt-managed inference and draw from your account's dbt Wizard consumption pool, the same as any other dbt Wizard usage. #### Related docs * [Manage AI features in dbt platform](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md) to turn AI features on or off * [How dbt Wizard works](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md) * [dbt AI usage](https://docs.getdbt.com/docs/platform/billing/dbt-ai-usage.md) for how dbt AI usage is metered and limited * [BYOK for the dbt platform](https://docs.getdbt.com/docs/platform/wizard-byok-platform.md) or [BYOK for the CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) * [Billing](https://docs.getdbt.com/docs/platform/billing.md) for general dbt platform billing --- ### dbt Wizard CLI data use and telemetry Local development dbt Wizard CLI collects anonymous product telemetry to improve the AI agent experience, understand usage patterns, optimize performance, and attribute compute costs without capturing your code, queries, prompts, responses, or file contents. #### Opt out of client telemetry dbt Wizard CLI respects the following opt-out mechanisms, checked in order: 1. Set `DO_NOT_TRACK=1`. 2. Set `DBT_SEND_ANONYMOUS_USAGE_STATS=false`. 3. Store telemetry consent in `~/.dbt/.user.yml`. Setting any of these disables telemetry from the dbt Wizard CLI client. #### What dbt Wizard does not collect dbt Wizard CLI does not collect: * Prompt or response content. * SQL queries, file paths, or dbt node names. * MCP tool call arguments or outputs. * Raw API keys or tokens. Identifiers are hashed before transmission. * Error messages that contain user content. Errors are limited to class or code. #### Events collected | Event | When it is collected | Why it is collected | | ------------------------------- | ---------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | LLM request completed or failed | When an LLM request completes | Token usage, model, latency monitoring, error rates, and cost attribution | | Wizard session started or ended | When a user opens or closes a dbt Wizard CLI session | Weekly active users, session duration, model adoption, and client surface usage | | Wizard turn completed | After a user message and AI response complete | Engagement depth, token consumption, model usage, status, and duration | | Wizard tool use | Each time the agent invokes a tool | Tool adoption, reliability, and performance | | Conversation feedback | If you submit feedback on a dbt Wizard conversation. | We may retain the associated transcript for up to 400 days to investigate the feedback. These transcripts are not used for model training and can be deleted upon written request. | Tool telemetry records the tool type, tool name, whether the call failed, and execution time. Tool arguments and outputs are not collected. #### Data handling * Telemetry is transmitted over HTTPS to dbt Labs ingestion infrastructure. * Events are stored in an internal dbt Labs data warehouse. * Telemetry is not shared with third parties. * API keys and tokens are not transmitted in raw form. * Local development users who opt out with the supported environment variables generate no dbt Wizard CLI client telemetry events. #### Related docs * [Install dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-cli.md) * [Configure BYOK for dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) * [dbt AI FAQs](https://docs.getdbt.com/docs/dbt-ai/dbt-ai-faqs.md) — including [privacy and data](https://docs.getdbt.com/docs/dbt-ai/dbt-ai-faqs.md#privacy-and-data) for dbt Wizard in the dbt platform --- ### dbt Wizard command reference Beta ### dbt Wizard command reference [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Local development Full reference for all `wizard` subcommands and global flags. Not the same as dbt commands This page is auto-generated and covers `wizard` commands and flags. For standard dbt project commands (`dbt run`, `dbt build`, `dbt test`, and so on.) refer to the [dbt command reference](https://docs.getdbt.com/reference/dbt-commands.md). If you see any issues, please [file an issue](https://github.com/dbt-labs/docs.getdbt.com/issues) and we'll be happy to sort it out. #### Common commands Most people use a handful of commands to get started. View the following table and then refer to the rest of the page or the [examples](#examples) section for more details. | I want to... | Command | What it does | | ---------------------------------- | ---------------------------------------------- | --------------------------------------------------------------------------- | | Start an interactive session | `wizard` | Opens the interactive TUI where you chat with the agent. | | Run a one-off task without the TUI | `wizard exec "add tests to my staging models"` | Runs the agent once, prints the result, and exits. Good for scripts and CI. | | Review my uncommitted changes | `wizard review --uncommitted` | Runs a code review on your staged, unstaged, and untracked changes. | | Pick up where I left off | `wizard resume --last` | Reopens your most recent session with its full history. | | Check that my install is healthy | `wizard doctor` | Diagnoses your install, config, auth, runtime health | | Update to the latest version | `wizard update` | Updates wizard to the newest release. | #### How to read this reference * **Flag** — the full flag as typed. Short flags (e.g. `-m`) are listed in the Short column. * **Type** — `boolean` (presence/absence), `string`, `path`, or an enumeration of allowed values. * **Description** — what the flag does and when to use it. * Global flags apply to the base interactive `wizard` command. Subcommands can have different flag sets; use each command's section for automation. See it in action and share your feedback Want to see dbt Wizard in action? Check out the [demo video](https://www.youtube.com/watch?v=-lIzh1xQWMA). We'd love to hear how dbt Wizard is working for you. Share your feedback by either running the `/feedback` slash command in your interactive terminal session or by going to the [#dbt-wizard](https://getdbt.slack.com/archives/C0B6KLW6T26) channel in the [dbt Community Slack](https://docs.getdbt.com/community/join?version=2.0). Thanks so much for your help in improving dbt Wizard and dbt data development! #### Global flags These flags work on the base interactive `wizard` command. | Flag | Short | Type | Description | | -------------------------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | | `--remote <ADDR>` | — | string | Connect the TUI to a remote app server endpoint. | | `--remote-auth-token-env <ENV_VAR>` | — | string | Name of the environment variable containing the bearer token to send to a remote app server websocket | | `--strict-config` | — | boolean | Error out when config.toml contains fields that are not recognized by this version of wizard | | `-i, --image <FILE>...` | `-i` | path | Optional image(s) to attach to the initial prompt | | `-m, --model <MODEL>` | `-m` | enum | Model the agent should use | | `--oss` | — | boolean | Use open-source provider | | `--local-provider <OSS_PROVIDER>` | — | enum | Specify which local provider to use (lmstudio or ollama). If not specified with --oss, will use config default or show selection | | `-p, --profile <CONFIG_PROFILE>` | `-p` | path | Configuration profile from config.toml to specify default options | | `--profile-v2 <CONFIG_PROFILE_V2>` | — | path | Layer $DBT\_WIZARD\_HOME/\<name>.config.toml on top of the base user config | | `-s, --sandbox <SANDBOX_MODE>` | `-s` | enum | Select the sandbox policy to use when executing model-generated shell commands | | `--dangerously-bypass-approvals-and-sandbox` | — | boolean | Skip all confirmation prompts and execute commands without sandboxing. EXTREMELY DANGEROUS. Intended solely for running in environments that are externally sandboxed | | `--dangerously-bypass-hook-trust` | — | boolean | Run enabled hooks without requiring persisted hook trust for this invocation. DANGEROUS. Intended only for automation that already vets hook sources | | `-C, --cd <DIR>` | `-C` | path | Tell the agent to use the specified directory as its working root | | `--add-dir <DIR>` | — | path | Additional directories that should be writable alongside the primary workspace | | `-a, --ask-for-approval <APPROVAL_POLICY>` | `-a` | enum | Configure when the model requires human approval before executing a command | | `--search` | — | boolean | Enable live web search. When enabled, the native Responses `web_search` tool is available to the model (no per‑call approval) | | `--no-alt-screen` | — | boolean | Disable alternate screen mode | #### Commands | Command | Aliases | Description | | ----------------------------------- | ------- | ------------------------------------------------------------------------------------------------- | | [`exec`](#exec) | `e` | Run wizard non-interactively | | [`review`](#review) | — | Run a code review non-interactively | | [`login`](#login) | — | Manage login | | [`logout`](#logout) | — | Remove stored authentication credentials | | [`mcp`](#mcp) | — | Manage external MCP servers for wizard | | [`plugin`](#plugin) | — | Manage wizard plugins | | [`providers`](#providers) | — | Manage model providers | | [`mcp-server`](#mcp-server) | — | Start wizard as an MCP server (stdio) | | [`app-server`](#app-server) | — | \[experimental] Run the app server or related tooling | | [`remote-control`](#remote-control) | — | \[experimental] Manage the app-server daemon with remote control enabled | | [`completion`](#completion) | — | Generate shell completion scripts | | [`update`](#update) | — | Update wizard to the latest version | | [`doctor`](#doctor) | — | Diagnose local wizard installation, config, auth, and runtime health | | [`sandbox`](#sandbox) | — | Run commands within a wizard-provided sandbox | | [`debug`](#debug) | — | Debugging tools | | [`apply`](#apply) | `a` | Apply the latest diff produced by wizard agent as a `git apply` to your local working tree | | [`resume`](#resume) | — | Resume a previous interactive session (picker by default; use --last to continue the most recent) | | [`fork`](#fork) | — | Fork a previous interactive session (picker by default; use --last to fork the most recent) | | [`cloud`](#cloud) | — | \[EXPERIMENTAL] Browse tasks from wizard in dbt platform and apply changes locally | | [`exec-server`](#exec-server) | — | \[EXPERIMENTAL] Run the standalone exec-server service | | [`features`](#features) | — | Inspect feature flags | #### exec Run wizard non-interactively ```bash wizard exec [OPTIONS] <COMMAND> [ARGS] ``` **Arguments:** | Argument | Description | | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `[PROMPT]` | Initial instructions for the agent. If not provided as an argument (or if `-` is used), instructions are read from stdin. If stdin is piped and a prompt is also provided, stdin is appended as a `<stdin>` block | | Flag | Short | Type | Description | | -------------------------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | | `--strict-config` | — | boolean | Error out when config.toml contains fields that are not recognized by this version of wizard | | `-i, --image <FILE>...` | `-i` | path | Optional image(s) to attach to the initial prompt | | `-m, --model <MODEL>` | `-m` | enum | Model the agent should use | | `--oss` | — | boolean | Use open-source provider | | `--local-provider <OSS_PROVIDER>` | — | enum | Specify which local provider to use (lmstudio or ollama). If not specified with --oss, will use config default or show selection | | `-p, --profile <CONFIG_PROFILE>` | `-p` | path | Configuration profile from config.toml to specify default options | | `--profile-v2 <CONFIG_PROFILE_V2>` | — | path | Layer $DBT\_WIZARD\_HOME/\<name>.config.toml on top of the base user config | | `-s, --sandbox <SANDBOX_MODE>` | `-s` | enum | Select the sandbox policy to use when executing model-generated shell commands | | `--dangerously-bypass-approvals-and-sandbox` | — | boolean | Skip all confirmation prompts and execute commands without sandboxing. EXTREMELY DANGEROUS. Intended solely for running in environments that are externally sandboxed | | `--dangerously-bypass-hook-trust` | — | boolean | Run enabled hooks without requiring persisted hook trust for this invocation. DANGEROUS. Intended only for automation that already vets hook sources | | `-C, --cd <DIR>` | `-C` | path | Tell the agent to use the specified directory as its working root | | `--add-dir <DIR>` | — | path | Additional directories that should be writable alongside the primary workspace | | `--skip-git-repo-check` | — | boolean | Allow running wizard outside a Git repository | | `--ephemeral` | — | boolean | Run without persisting session files to disk | | `--ignore-user-config` | — | boolean | Do not load `$DBT_WIZARD_HOME/config.toml`; auth still uses `DBT_WIZARD_HOME` | | `--ignore-rules` | — | boolean | Do not load user or project execpolicy `.rules` files | | `--output-schema <FILE>` | — | path | Path to a JSON Schema file describing the model's final response shape | | `--color <COLOR>` | — | enum | Specifies color settings for use in the output | | `--json` | — | boolean | Print events to stdout as JSONL | | `--include-subagent-events` | — | boolean | When `--json` is set, capture and emit sub-agent events tagged with their thread ID. Each sub-agent event gets a `sub_agent_thread_id` field added to the JSONL line so consumers can distinguish it from the primary agent's events | | `--no-validation` | — | boolean | When set, the validation gate always responds with "declined" so the validation sub-agent is never spawned. Useful for benchmarking runs where sub-agent overhead is not desired | | `-o, --output-last-message <FILE>` | `-o` | path | Specifies file where the last message from the agent should be written | #### review Run a code review non-interactively **Arguments:** | Argument | Description | | ---------- | ----------------------------------------------------------- | | `[PROMPT]` | Custom review instructions. If `-` is used, read from stdin | | Flag | Short | Type | Description | | -------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--strict-config` | — | boolean | Error out when config.toml contains fields that are not recognized by this version of wizard | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--uncommitted` | — | boolean | Review staged, unstaged, and untracked changes | | `--base <BRANCH>` | — | string | Review changes against the given base branch | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | | `--commit <SHA>` | — | string | Review the changes introduced by a commit | | `--title <TITLE>` | — | string | Optional commit title to display in the review summary | #### login Manage login | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### logout Remove stored authentication credentials | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### mcp Manage external MCP servers for dbt Wizard | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### plugin Manage dbt Wizard plugins | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### providers Manage model providers | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### mcp-server Start dbt Wizard as an MCP server (stdio) | Flag | Short | Type | Description | | -------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--strict-config` | — | boolean | Error out when config.toml contains fields that are not recognized by this version of wizard | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### app-server \[experimental] Run the app server or related tooling | Flag | Short | Type | Description | | --------------------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | | `--strict-config` | — | boolean | Error out when config.toml contains fields that are not recognized by this version of wizard | | `--listen <URL>` | — | string | Transport endpoint URL. Supported values: `stdio://` (default), `unix://`, `unix://PATH`, `ws://IP:PORT`, `off` | | `--analytics-default-enabled` | — | boolean | Controls whether analytics are enabled by default. | | `--ws-auth <MODE>` | — | enum | Websocket auth mode for non-loopback listeners | | `--ws-token-file <PATH>` | — | path | Absolute path to the capability-token file | | `--ws-token-sha256 <HEX>` | — | string | Hex-encoded SHA-256 digest of the capability token | | `--ws-shared-secret-file <PATH>` | — | path | Absolute path to the shared secret file for signed JWT bearer tokens | | `--ws-issuer <ISSUER>` | — | string | Expected issuer for signed JWT bearer tokens | | `--ws-audience <AUDIENCE>` | — | string | Expected audience for signed JWT bearer tokens | | `--ws-max-clock-skew-seconds <SECONDS>` | — | string | Maximum clock skew when validating signed JWT bearer tokens | #### remote-control \[experimental] Manage the app-server daemon with remote control enabled | Flag | Short | Type | Description | | -------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--json` | — | boolean | Emit machine-readable JSON | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### completion Generate shell completion scripts **Arguments:** | Argument | Description | | --------- | --------------------------------- | | `[SHELL]` | Shell to generate completions for | | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### update Update dbt Wizard to the latest version | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### doctor Diagnose local dbt Wizard installation, config, auth, and runtime health | Flag | Short | Type | Description | | -------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--json` | — | boolean | Emit a redacted machine-readable report | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--summary` | — | boolean | Only show grouped check rows and the final count summary | | `--all` | — | boolean | Expand long lists in detailed human output | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | | `--no-color` | — | boolean | Disable ANSI color in human output | | `--ascii` | — | boolean | Use ASCII status labels and separators in human output | #### sandbox Run commands within a dbt Wizard-provided sandbox | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### debug Debugging tools | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### apply Apply the latest diff produced by dbt Wizard agent as a `git apply` to your local working tree **Arguments:** | Argument | Description | | ----------- | ----------- | | `<TASK_ID>` | | | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### resume Resume a previous interactive session (picker by default; use --last to continue the most recent) **Arguments:** | Argument | Description | | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | `[SESSION_ID]` | Conversation/session id (UUID) or thread name. UUIDs take precedence if it parses. If omitted, use --last to pick the most recent recorded session | | `[PROMPT]` | Optional user prompt to start the session | | Flag | Short | Type | Description | | -------------------------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--last` | — | boolean | Continue the most recent session without showing the picker | | `--all` | — | boolean | Show all sessions (disables cwd filtering and shows CWD column) | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | | `--include-non-interactive` | — | boolean | Include non-interactive sessions in the resume picker and --last selection | | `--remote <ADDR>` | — | string | Connect the TUI to a remote app server endpoint. | | `--remote-auth-token-env <ENV_VAR>` | — | string | Name of the environment variable containing the bearer token to send to a remote app server websocket | | `--strict-config` | — | boolean | Error out when config.toml contains fields that are not recognized by this version of wizard | | `-i, --image <FILE>...` | `-i` | path | Optional image(s) to attach to the initial prompt | | `-m, --model <MODEL>` | `-m` | enum | Model the agent should use | | `--oss` | — | boolean | Use open-source provider | | `--local-provider <OSS_PROVIDER>` | — | enum | Specify which local provider to use (lmstudio or ollama). If not specified with --oss, will use config default or show selection | | `-p, --profile <CONFIG_PROFILE>` | `-p` | path | Configuration profile from config.toml to specify default options | | `--profile-v2 <CONFIG_PROFILE_V2>` | — | path | Layer $DBT\_WIZARD\_HOME/\<name>.config.toml on top of the base user config | | `-s, --sandbox <SANDBOX_MODE>` | `-s` | enum | Select the sandbox policy to use when executing model-generated shell commands | | `--dangerously-bypass-approvals-and-sandbox` | — | boolean | Skip all confirmation prompts and execute commands without sandboxing. EXTREMELY DANGEROUS. Intended solely for running in environments that are externally sandboxed | | `--dangerously-bypass-hook-trust` | — | boolean | Run enabled hooks without requiring persisted hook trust for this invocation. DANGEROUS. Intended only for automation that already vets hook sources | | `-C, --cd <DIR>` | `-C` | path | Tell the agent to use the specified directory as its working root | | `--add-dir <DIR>` | — | path | Additional directories that should be writable alongside the primary workspace | | `-a, --ask-for-approval <APPROVAL_POLICY>` | `-a` | enum | Configure when the model requires human approval before executing a command | | `--search` | — | boolean | Enable live web search. When enabled, the native Responses `web_search` tool is available to the model (no per‑call approval) | | `--no-alt-screen` | — | boolean | Disable alternate screen mode | #### fork Fork a previous interactive session (picker by default; use --last to fork the most recent) **Arguments:** | Argument | Description | | -------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | `[SESSION_ID]` | Conversation/session id (UUID). When provided, forks this session. If omitted, use --last to pick the most recent recorded session | | `[PROMPT]` | Optional user prompt to start the session | | Flag | Short | Type | Description | | -------------------------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--last` | — | boolean | Fork the most recent session without showing the picker | | `--all` | — | boolean | Show all sessions (disables cwd filtering and shows CWD column) | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | | `--remote <ADDR>` | — | string | Connect the TUI to a remote app server endpoint. | | `--remote-auth-token-env <ENV_VAR>` | — | string | Name of the environment variable containing the bearer token to send to a remote app server websocket | | `--strict-config` | — | boolean | Error out when config.toml contains fields that are not recognized by this version of wizard | | `-i, --image <FILE>...` | `-i` | path | Optional image(s) to attach to the initial prompt | | `-m, --model <MODEL>` | `-m` | enum | Model the agent should use | | `--oss` | — | boolean | Use open-source provider | | `--local-provider <OSS_PROVIDER>` | — | enum | Specify which local provider to use (lmstudio or ollama). If not specified with --oss, will use config default or show selection | | `-p, --profile <CONFIG_PROFILE>` | `-p` | path | Configuration profile from config.toml to specify default options | | `--profile-v2 <CONFIG_PROFILE_V2>` | — | path | Layer $DBT\_WIZARD\_HOME/\<name>.config.toml on top of the base user config | | `-s, --sandbox <SANDBOX_MODE>` | `-s` | enum | Select the sandbox policy to use when executing model-generated shell commands | | `--dangerously-bypass-approvals-and-sandbox` | — | boolean | Skip all confirmation prompts and execute commands without sandboxing. EXTREMELY DANGEROUS. Intended solely for running in environments that are externally sandboxed | | `--dangerously-bypass-hook-trust` | — | boolean | Run enabled hooks without requiring persisted hook trust for this invocation. DANGEROUS. Intended only for automation that already vets hook sources | | `-C, --cd <DIR>` | `-C` | path | Tell the agent to use the specified directory as its working root | | `--add-dir <DIR>` | — | path | Additional directories that should be writable alongside the primary workspace | | `-a, --ask-for-approval <APPROVAL_POLICY>` | `-a` | enum | Configure when the model requires human approval before executing a command | | `--search` | — | boolean | Enable live web search. When enabled, the native Responses `web_search` tool is available to the model (no per‑call approval) | | `--no-alt-screen` | — | boolean | Disable alternate screen mode | #### cloud \[EXPERIMENTAL] Browse tasks from dbt Wizard in dbt platform and apply changes locally | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### exec-server \[EXPERIMENTAL] Run the standalone exec-server service | Flag | Short | Type | Description | | --------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--listen <URL>` | — | string | Transport endpoint URL. Supported values: `ws://IP:PORT` (default), `stdio`, `stdio://` | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--remote <URL>` | — | string | Register this exec-server as a remote environment using the given base URL | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | | `--environment-id <ID>` | — | string | Environment id to attach to when registering remotely | | `--name <NAME>` | — | string | Human-readable environment name | | `--use-agent-identity-auth` | — | boolean | Use Agent Identity auth from DBT\_WIZARD\_ACCESS\_TOKEN for remote registration | #### features Inspect feature flags | Flag | Short | Type | Description | | -------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --config <key=value>` | `-c` | string | Override a configuration value that would otherwise be loaded from `~/.dbt/wizard/config.toml`. Use a dotted path (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as TOML, the raw string is used as a literal. | | `--enable <FEATURE>` | — | string | Enable a feature (repeatable). Equivalent to `-c features.<name>=true` | | `--disable <FEATURE>` | — | string | Disable a feature (repeatable). Equivalent to `-c features.<name>=false` | #### Examples Here are some examples and commands that you might use. Replace the example prompts with your own: * **Run a task without the TUI** Use `wizard exec` for one-off tasks, scripts, and CI. It runs the agent, prints the result, and exits. ```shell # Run a task and exit wizard exec "explain what the orders model does" # Pipe a prompt in from stdin echo "summarize my schema.yml files" | wizard exec - # Emit machine-readable output for scripting wizard exec --json "list my models" > result.jsonl ``` * **Review your changes** `wizard review` runs a code review without starting an interactive session. ```shell # Review staged, unstaged, and untracked changes wizard review --uncommitted # Review your branch against main wizard review --base main ``` * **Resume a session** ```shell # Continue your most recent session wizard resume --last # Pick from a list of past sessions wizard resume ``` * **Override a config value** Use `-c` to override any value from `~/.dbt/wizard/config.toml` for a single run, without editing the file. ```shell # Set the model for this run only wizard exec -c model="dbt/gpt-5.5" "your prompt" ``` #### Related docs * [Slash command reference](https://docs.getdbt.com/docs/dbt-ai/wizard-slash-commands.md) for interactive TUI slash commands * [Headless mode](https://docs.getdbt.com/docs/dbt-ai/wizard-headless.md) * [Configuration reference](https://docs.getdbt.com/docs/dbt-ai/wizard-config.md) * [dbt command reference](https://docs.getdbt.com/reference/dbt-commands.md) for `dbt run`, `dbt build`, and other dbt commands --- ### dbt Wizard config reference Beta ### dbt Wizard config reference [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Local development dbt Wizard stores configuration in two TOML files under `~/.dbt/wizard/`, each controlling a different part of the product. #### The two config files See it in action and share your feedback Want to see dbt Wizard in action? Check out the [demo video](https://www.youtube.com/watch?v=-lIzh1xQWMA). We'd love to hear how dbt Wizard is working for you. Share your feedback by either running the `/feedback` slash command in your interactive terminal session or by going to the [#dbt-wizard](https://getdbt.slack.com/archives/C0B6KLW6T26) channel in the [dbt Community Slack](https://docs.getdbt.com/community/join?version=2.0). Thanks so much for your help in improving dbt Wizard and dbt data development! TOML (Tom's Obvious, Minimal Language) is a config file format. If you've written YAML or JSON before, it'll feel familiar — keys and values are separated by `=` (with spaces before and after the equals sign), and nested sections use `[section.name]` headers instead of indentation or curly braces. For example: \~/.dbt/wizard/config.toml ```toml model = "gpt-4o" [projects."/Users/you/jaffle-shop"] trust_level = "trusted" ``` This config sets the default AI model for all new sessions and applies to all future invocations across all projects. You can override the default AI model for a specific project by setting the `model` key in the `wizard_config.toml` file for that project or using the `/model` picker in the text-based user interface (TUI). | File | Controls | When to use | | ---------------------------------- | ---------------------------------------------------------------------------- | ------------------------------------------------------- | | `~/.dbt/wizard/config.toml` | Agent runtime — AI model, MCP servers, approval rules, trusted projects | Change agent-level behavior across all sessions | | `~/.dbt/wizard/wizard_config.toml` | Per-project settings — dbt executable path, deferral, saved model preference | Change how dbt Wizard works with a specific dbt project | Note that `dbt_project.yml` is separate from both and controls how dbt builds your project. #### Common tasks | Goal | Edit | Restart needed? | | ----------------------------------------- | ------------------------------------------------------------------ | --------------- | | Change the default AI model | `config.toml` → `model = "gpt-4o"` | Yes | | Point dbt Wizard at a specific dbt binary | `wizard_config.toml` → `path` | Yes | | Add the dbt MCP server | `config.toml` → `[mcp_servers.dbt]` | Yes | | Mark a repo as trusted | `config.toml` → `trust_level = "trusted"` under `[projects."..."]` | Yes | | Change approval or sandbox defaults | `config.toml` | Yes | #### Config precedence Settings resolve in this order (highest to lowest): 1. CLI flags (`-m`, `-c`) at invocation 2. In-session model picker (`/model`) 3. `~/.dbt/wizard/config.toml` 4. `~/.dbt/wizard/wizard_config.toml` 5. Built-in defaults #### config.toml Global agent settings for models, approvals, sandboxing, web search, and MCP servers. `~/.dbt/wizard/config.toml` controls how the agent session behaves. It's created automatically when dbt Wizard is installed. ##### Configuration keys This is the list of configuration keys that can be set in `config.toml`. * `model` * `approval_policy` * `sandbox_mode` * `web_search` You can view more by running `wizard config --help`. ###### AI model and API New dbt Wizard CLI installs use dbt Labs-managed models out of the box. Refer to [Configure BYOK](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) to use a different provider. | Key | Type | Default | Description | | ------- | ------ | ------- | ------------------------------------------------------------------------------------------------------------------------------------------- | | `model` | string | — | Default AI model for new sessions. Refer to [AI model ID format](#ai-model-id-format) below. Change interactively with `/model` in the TUI. | ###### Behavior | Key | Type | Default | Description | | ----------------- | ------ | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `approval_policy` | string | `on-request` | When Wizard asks before executing commands. Options: `untrusted` (ask for non-trusted commands), `on-request` (dbt Wizard decides), `never` (run without asking). | | `sandbox_mode` | string | `read-only` | Shell sandbox policy. Options: `read-only` (default), `workspace-write` (allow writes inside project dir), `danger-full-access` (no restrictions). | | `web_search` | string | `cached` | Web search mode. Options: `disabled`, `cached`, or `live`. | ###### Project trust Set `trust_level` per-project to allow dbt Wizard to use project-local config: ```toml [projects."/absolute/path/to/your-repo"] trust_level = "trusted" ``` ###### MCP servers ```toml [mcp_servers.dbt] command = "DBT_MCP_ENDPOINT" ``` ##### AI model ID format AI model IDs in `config.toml` use the public model ID, such as `gpt-4o`. To list all available AI model IDs: ```bash wizard debug models ``` ##### Environment variables Any key can be set as an environment variable using the `DBT_WIZARD_` prefix in `SCREAMING_SNAKE_CASE`: ```bash export DBT_WIZARD_MODEL=gpt-4o export DBT_WIZARD_APPROVAL_POLICY=never export OPENAI_API_KEY=sk-... # OpenAI (no prefix needed) export ANTHROPIC_API_KEY=sk-ant-... # Anthropic (no prefix needed) ``` `OPENAI_API_KEY` and `ANTHROPIC_API_KEY` are read directly without the `DBT_WIZARD_` prefix, following each provider's convention. ##### When it's read `config.toml` is read at session start when you run `wizard`. Changes take effect only after you exit and restart the CLI. Per-invocation `-c` flags and the in-session `/model` picker override this file for that session only. ##### Example ```toml model = "gpt-4o" [projects."/Users/you/jaffle-shop"] trust_level = "trusted" [mcp_servers.dbt] command = "DBT_MCP_ENDPOINT" ``` #### wizard\_config.toml Per-project settings Wizard writes during onboarding, including dbt path and deferral settings. `~/.dbt/wizard/wizard_config.toml` stores what dbt Wizard knows about each of your dbt projects. dbt Wizard writes to this file automatically during onboarding and when you change project settings. You can also edit it manually. ##### Configuration keys | Key | Description | | --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `version` | Schema version for this file (currently `1`). Don't change this manually. | | `global.terms_of_use_accepted_at` | Timestamp of when you accepted the dbt Wizard Terms of Use. Written once under the `[global]` section and applies across all projects. | | `global.dbt_compile_on_startup` | Whether to compile the development project at startup. Defaults to `true`. | | `global.prod_parse_on_startup` | Whether to parse the production environment and refresh deferral state at startup. Defaults to `true`. | | `global.prod_parse_args` | Arguments for the production parse. Defaults to `["parse", "--target", "prod"]`. | | `global.prod_state_ttl_hours` | Number of hours a production state snapshot remains fresh. Defaults to `24`. | | `global.prod_state_dir` | Directory for production state snapshots. Defaults to `target/prod-state` in the project. | | `global.compile_extra_args` | Extra arguments to append when dbt Wizard compiles any project without a project-specific override. | | `onboarded_at` | Timestamp of when you onboarded this project in dbt Wizard. | | `path` | Path to the dbt binary dbt Wizard should use for this project. | | `compile_extra_args` | Extra arguments to append when dbt Wizard compiles this project. | | `prod_parse_args` | Project-specific arguments for the production parse command. | | `deferral.mode` | Who handles [deferral](https://docs.getdbt.com/reference/node-selection/defer.md) for this project. Refer to [Deferral](#deferral) below. | | `deferral.target` | The [target](https://docs.getdbt.com/docs/local/profiles.yml.md) from your `profiles.yml` that dbt Wizard compiles and defers to when `deferral.mode` is `"wizard"` (for example, `"prod"`). | | `deferral.state` | Path to a state directory when `deferral.mode` is `"manual"`. | | `deferral.favor_state` | Whether deferred relations take precedence. Defaults to `true`. | | `profile_override.path` | Path to an alternative `profiles.yml` file for this project. | | `profile_override.profile` | Profile name to use from the selected profiles file. | | `profile_override.target` | Target to use from the selected profile. | Settings are stored per-project under `[projects."/absolute/path/to/repo"]` blocks: \~/.dbt/wizard/wizard\_config.toml ```toml version = 1 [global] terms_of_use_accepted_at = "2026-05-28T15:06:54Z" dbt_compile_on_startup = true prod_parse_on_startup = true prod_state_ttl_hours = 24 [projects."/Users/you/jaffle-shop"] onboarded_at = "2026-05-20T11:09:01.410346" path = "/Users/you/jaffle-shop/.venv/bin/dbt" [projects."/Users/you/jaffle-shop".deferral] mode = "wizard" target = "prod" favor_state = true ``` ###### Deferral [Deferral](https://docs.getdbt.com/reference/node-selection/defer.md) lets dbt Wizard reuse models that are already built elsewhere (for example, in production) instead of rebuilding everything when you're only working on part of a project, saving you time and warehouse cost. The `deferral.mode` setting in `wizard_config.toml` controls who handles deferral. It accepts six values, and dbt Wizard usually sets it for you during onboarding: | `deferral.mode` value | What it means | | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `"wizard"` | dbt Wizard handles deferral for you. You tell dbt Wizard which [target](https://docs.getdbt.com/docs/local/profiles.yml.md) from your `profiles.yml` to defer to (it tries to detect one automatically when you first set up the project). dbt Wizard then compiles that target and reuses its models for any upstream models you haven't built yourself. | | `"fusion_cloud"` | The dbt platform handles deferral against your connected environment, so dbt Wizard doesn't manage any local state. Set automatically when you're connected to the platform. | | `"cloud_cli"` | The dbt platform CLI manages credentials and deferral through the dbt platform, so dbt Wizard skips its production compile and deferral flag injection. | | `"dbt_state"` | dbt State or run cache handles deferral, so dbt Wizard skips its own production compile. | | `"manual"` | You provide the deferral manifest directory with `state`. | | `"disabled"` | Deferral is disabled for the project. | For more about dbt State, refer to [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md). Set `state` to a directory containing a compatible `manifest.json` when `mode = "manual"`: ```toml [projects."/Users/you/jaffle-shop".deferral] mode = "manual" state = "/Users/you/dbt-state/production" favor_state = true ``` **About [favor-state](https://docs.getdbt.com/reference/node-selection/defer.md?version=2.0#favor-state):** `favor_state` is configurable and defaults to `true`. When it is `true`, dbt Wizard passes `--favor-state` so deferred relations take precedence. Set it to `false` to use a relation that already exists in development and fall back to deferred state when it doesn't. You can also configure production snapshot and compile behavior with global `prod_parse_on_startup`, `prod_parse_args`, `prod_state_ttl_hours`, `prod_state_dir`, and `compile_extra_args` settings. Per-project `compile_extra_args` and `prod_parse_args` values let you override the global behavior for one project. Use a `profile_override` block to select a different profiles file, profile, or target for compilation. For configuration examples and a verification workflow, refer to [Developing with production deferral](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-6-production-deferral.md). #### Troubleshooting Here are some common config mistakes and how to fix them: | Symptom | Likely cause | Fix | | -------------------------------------------------------- | -------------------------------------------- | --------------------------------------------------- | | Still on old AI model after editing `wizard_config.toml` | `wizard_config.toml` doesn't store the model | Set `model` in `config.toml` and restart `wizard` | | AI model unchanged after editing `config.toml` | Old session still running | Press Ctrl+C, then run `wizard` again | | "Invalid model" error | Wrong AI model ID | Run `wizard debug models` to see valid AI model IDs | ##### Re-trigger onboarding flows If you need to re-run part of the setup — for example, after deleting config entries by mistake or switching to a new project — you can reset individual flows by deleting the relevant file. dbt Wizard will re-prompt you the next time it runs. To delete a file, run the command in your terminal (macOS/Linux). For example: ```bash rm ~/.dbt/wizard/auth.json ``` If you get a `No such file or directory` error, the file doesn't exist — which means that part of the setup hasn't run yet, or has already been reset. You can ignore the error. | What you want to redo | File to delete | | --------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | Re-run the trusted folder prompt | Remove your project's entry from `~/.dbt/wizard/config.toml` under `[projects."..."]` | | Re-authenticate with dbt platform | `~/.dbt/wizard/auth.json` | | Re-run dbt project setup | Remove your project's entry from `~/.dbt/wizard/wizard_config.toml` under `[projects."..."]`, or delete the file entirely to reset all projects | | Re-trigger BYOK key configuration | `~/.dbt/wizard/provider*` (this matches all provider files) | caution `config.toml` and `wizard_config.toml` are separate files that control different things. If you want to re-run project setup, edit `wizard_config.toml` — not `config.toml`. Editing the wrong file won't re-run the onboarding flow you expect. Refer to [The two config files](#the-two-config-files) at the top of this page for the difference. #### Related docs Links to BYOK setup, approval behavior, and CLI command reference. * [Configure BYOK](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) * [Approval and sandboxing](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md#approval-and-sandboxing) * [dbt Wizard command reference](https://docs.getdbt.com/docs/dbt-ai/wizard-cli-reference.md) * [How dbt Wizard works](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md) --- ### dbt Wizard in Studio IDE Preview ### dbt Wizard in Studio IDE [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt platform | Usage-based Use dbt Wizard in the Studio IDE to ship trusted dbt changes faster. It understands your project, answers context-grounded questions, generates models, tests, docs, and Semantic Layer definitions, and shows file diffs before changes are persisted. dbt Wizard supports the dbt development lifecycle from investigation to review. Use it to: * Ask project-aware questions using lineage, metadata, and catalog context. * Build or refactor models from natural-language prompts. * Generate and validate YAML for tests, documentation, semantic models, and metrics. * Make scoped edits to logic, names, materializations, tests, and related YAML. * Investigate job and run failures with dbt Agent Skills. The agent comes with the following out of the box, meaning no configuration needed: * [dbt Agent Skills](https://github.com/dbt-labs/dbt-agent-skills): dbt-recommended guidance and instructions, managed by dbt Labs. * [dbt MCP server Product docs toolset](https://docs.getdbt.com/docs/dbt-ai/mcp-available-tools.md#product-docs): Tools for searching and fetching content from dbt's official documentation. AI features are being enabled by default. They're already on for new accounts and are rolling out soon to existing accounts. If your organization opted out, they'll remain off. Admins can [turn AI off or back on and configure providers](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md) anytime. ##### Prerequisites * A [dbt platform account](https://www.getdbt.com/signup) and [Developer seat license](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md). * [Legacy Team plans](https://docs.getdbt.com/docs/platform/billing/plans-and-billing.md#legacy-plans) don't have access to dbt Wizard. Move to a [Starter, Enterprise, or Enterprise+ plan](https://www.getdbt.com/pricing) to use it. * A [development environment](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md#get-started-with-the-studio-ide) and credentials set up in the Studio IDE. * Use a supported AI provider. Refer to [Supported AI providers](https://docs.getdbt.com/docs/platform/wizard-platform.md#supported-ai-providers), or the [Model Provider Rate Table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) for the full model list and rates. If dbt Wizard stops responding, your account may have used up its usage credits. Refer to [Trial and billing](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/trial-and-billing.md).  Availability and considerations * **Where it runs:** Supported in the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) only, all [deployment types](https://docs.getdbt.com/docs/platform/about-platform/tenancy.md?version=2.0). Not supported in VS Code or the dbt platform CLI. * **Engines:** Works with dbt v2 and dbt v1. * **Conversations:** In the conversation list, open **More actions** menu (three dots) of the conversation you want to delete, then click **Delete** to remove one thread. Deleting the open thread clears the panel. * **Sessions:** Refreshing the same browser tab keeps your active session. A new tab, or returning after closing the tab, starts empty. * **Chat history:** Retained for 90 days only. Chat history isn't supported yet on single-tenant deployments, so save anything important before closing. * **Plan mode:** Not supported yet. The agent doesn't show a separate plan before applying changes, however you can use the **Ask for approval** mode to approve each file. * **New chat:** Click **Start new dbt Wizard chat** (top right of the dbt Wizard panel) to begin a new session. ##### Using dbt Wizard Use the dbt Wizard panel to generate resources with quick actions, or use the agent to build and refactor models end-to-end with natural language prompts. To use the dbt Wizard, follow these steps: 1. Open your dbt project in the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md), then click **dbt Wizard** in the command palette. 2. Start a prompt in several ways in the [dbt Wizard panel](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md?version=2#panel-controls): * **Quick actions**: Use [quick-action resource generation](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md#quick-action-resource-generation) at the top of the panel for quick action prompts. * **Plain text**: Type directly into the text field to describe what you want to build or change. * **Model context**: Type `@` to select a model as context. This scopes the agent's changes to that resource. 3. Select the [**Agent mode** button](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md#agent-modes) to specify the mode for the dbt Wizard. Available modes are **Explore only**, **Ask for approval** (default), and **Edit files automatically**. 4. Select the dbt managed model you'd like to work with from the [model picker](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/overview.md#choose-a-model) next to the **Agent mode** button. 5. [Review the agent's suggestions](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md#reviewing-agent-suggestions) and approve or reject the changes. You can also use the **Start new dbt Wizard chat** button to start a new chat session. 6. [Approve dbt commands](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md#granting-command-permissions) when the dbt Wizard requests to run commands like `dbt compile` or `dbt build`. 7. Repeat the process to build or change more models. 8. Commit the changes to your dbt project and open a pull request. The following images show how dbt Wizard displays its work and outcome: [![dbt Wizard refactoring a model and displaying the lineage inside the chat interface.](/img/docs/dbt-platform/wizard-ide-refactor-lineage.png?v=2 "dbt Wizard refactoring a model and displaying the lineage inside the chat interface.")](#)dbt Wizard refactoring a model and displaying the lineage inside the chat interface. [![Wizard final refactor result displayed as a diff](/img/docs/dbt-platform/wizard-ide-refactor-diff.png?v=2 "Wizard final refactor result displayed as a diff")](#)Wizard final refactor result displayed as a diff For more details on the dbt Wizard and how it works, expand the following sections to open additional information.  Panel controls The dbt Wizard panel contains: 1. **Quick actions** (center): Buttons at the top of the panel for common tasks like generating documentation, tests, semantic models, and metrics. When selected, the text field is pre-filled with a prompt. 2. **Agent mode button** (bottom left): Switch between **Explore only**, **Ask for approval**, and **Edit files automatically** mode. Click the button to change modes. 3. **Model picker** (bottom left): Select the dbt managed model to use for the session. Refer to [Choose a model](#choose-a-model) for the available models. 4. **dbt model context** (bottom left): Shows the currently open file. Use `@` in the text field to reference a different dbt model. Click **x** to remove the dbt model context. 5. **Text input field** (bottom left): Type your prompt in the text field to describe what you want to build or change. Type `@` to select a dbt model as context. This scopes the agent's changes to that resource. 6. **Start new dbt Wizard chat** (top right): Starts a new chat session. 7. **Stop** or **Enter** (bottom right): Press **Enter** to submit your prompt. Press **Stop** to stop the current session and agent processing. You cannot undo this action. [![The Wizard panel in the Studio IDE showing quick-action buttons, the agent mode button, the model picker, and the text input field.](/img/docs/dbt-platform/wizard-panel.png?v=2 "The Wizard panel in the Studio IDE showing quick-action buttons, the agent mode button, the model picker, and the text input field.")](#)The Wizard panel in the Studio IDE showing quick-action buttons, the agent mode button, the model picker, and the text input field. dbt Wizard also has a simplified wayfinder bar above the text input field. The wayfinder bar shows your current project and branch and guides you through Git tasks, such as committing files or creating a branch. []()  Agent modes The dbt Wizard operates in three modes: | Mode | Behavior | | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Explore only** | The agent queries and explains data but can't edit files or run builds. Best when you want to analyze data or validate a model's output without the agent proposing changes. Every answer comes with the SQL or metric definition behind it. Available for read-only users in the home tab. | | **Ask for approval** (default) | The agent drafts edits to files. You approve each file change before it is persisted. Best when you want tight control over what gets saved to your branch. | | **Edit files automatically** | The agent drafts and automatically saves file edits without per-file approval. Best for faster iteration when you're confident in the prompt. | Switch between modes anytime with the **Agent mode** button (bottom-left). The authoring modes keep the analytical tools available, so switching out of **Explore only** doesn't cost you anything. [![dbt Wizard in Explore only, Ask for approval, and Edit files automatically modes.](/img/docs/dbt-platform/wizard-modes.png?v=2 "dbt Wizard in Explore only, Ask for approval, and Edit files automatically modes.")](#)dbt Wizard in Explore only, Ask for approval, and Edit files automatically modes.  Reviewing agent suggestions When the dbt Wizard proposes code changes, you can review them before they are saved to your project: * **View the diff**: The agent displays a diff of the proposed changes. Click **Show all X lines** to expand and view the full suggestion. * **Line indicators**: Added and removed lines are highlighted with line number indicators so you can see exactly what changed. * **Copy or open in editor**: Use the options in the top-right corner of the diff view to copy the suggestion or open it directly in the editor.  Granting command permissions To validate or run models during a session, the agent may request to run dbt commands such as `dbt compile` or `dbt build`. You'll be prompted to approve each request before it executes. For example, the agent might request to run: ```bash dbt compile --select model_name ``` You can select one of the following options: | Option | Behavior | | ----------------------------------------------------- | ------------------------------------------------------------------------------------------------ | | **Yes, run once** | Grants permission to run this specific command one time. | | **Yes, and allow `dbt_command_name` for the session** | Grants permission to run dbt commands for the remainder of your session without prompting again. | | **No** | Denies the request. The agent will not run the command. | After you run a command, dbt Wizard adds an icon and a tooltip to the Studio IDE [**Commands** tab](https://docs.getdbt.com/docs/platform/studio-ide/ide-user-interface.md#console-section) results. This helps you distinguish agent-run commands from manually run commands in the run results and logs. ##### Bringing your own skills You can extend dbt Wizard with custom skills to encode your team's SQL conventions, naming rules, and modeling workflows — so you don't repeat them in every prompt. See [Skills](https://docs.getdbt.com/docs/dbt-ai/wizard-platform-skills.md) for the full reference, including how to create, structure, and invoke skills. ##### Debug job failures The dbt Wizard can investigate and troubleshoot dbt job and run failures directly from the Studio IDE. This capability is powered by the `troubleshooting-dbt-job-errors` [dbt Agent Skill](https://github.com/dbt-labs/dbt-agent-skills), which comes pre-configured with the agent — no setup required. You can ask the agent questions and issue commands like: * "What jobs have failed recently?" * "What is the root cause of the job failure?" * "How can I fix the recent job failure?" * "Fix the job failure." The agent notes when your local project state may differ from the job — for example, if you're on a different branch or have uncommitted changes — so you have full context before acting on any suggested fixes. ##### Timeout handling When a dbt command run by dbt Wizard runs for more than 5 minutes, the agent automatically attempts to stop the command on the server before returning control to you. Instead of hanging or showing a generic error, the agent returns a clear message that explains the command timed out and was aborted. The message also tells you whether the cancellation request succeeded. If cancellation fails, it's possible the command may still be running on the server. You can then choose whether to retry the command, narrow the request, or take another action. ##### dbt v2 migration workflow If you have access to [dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) with [AI features](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md) enabled, you can use the [dbt v2 migration workflow](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md#fusion-migration-workflow) skill. This skill can help you fix compatibility errors directly from the Studio IDE using dbt Wizard — no manual log investigation needed. It classifies every error, applies validated fixes automatically, and surfaces what's blocked. info The dbt v2 migration workflow is accessible through the dbt Wizard in the Studio IDE. If you're using VS Code or the dbt platform CLI, use the [autofix tool](https://docs.getdbt.com/guides/dbt-package-compat?step=4) instead. 1. From the job list, click the **Review job** button for a job with a successful run. * If you don't see the **Review job** button, enable the **Show dbt v2 eligibility** toggle in the job list. 2. In the **dbt v2 eligibility unknown for this job** pop-up, click **Debug in Studio with dbt Wizard**. 3. dbt redirects you to the Studio IDE and sets your personal development environment to dbt v2. 4. dbt Wizard opens and automatically triggers the dbt v2 migration skill with this prompt: ```text I need help fixing dbt v2 compatibility issues in this project. Please investigate and resolve any deprecation warnings or incompatibilities. Please use the migrating-dbt-core-to-fusion skill to guide this. ``` 5. Review and approve dbt Wizard's permission requests so it can run the commands it needs. 6. The dbt Wizard iteratively runs `dbt compile`, reads the results, and applies fixes until it reaches a successful compile or encounters an error it can't resolve. If it gets blocked, it exits cleanly, explains what it could not fix, and creates and links to a markdown file summarizing all changes made. 7. When the project compiles with no warnings or errors, commit and publish your changes. 8. After you merge the changes, wait for the job to run again or run it manually on dbt v2. [![The Developer Agent's v2 migration workflow triaging and fixing v2 compatibility errors in the Studio IDE.](/img/docs/dbt-platform/fusion-migration-workflow.gif?v=2 "The Developer Agent's v2 migration workflow triaging and fixing v2 compatibility errors in the Studio IDE.")](#)The Developer Agent's v2 migration workflow triaging and fixing v2 compatibility errors in the Studio IDE. For more on how to prepare your project for dbt v2 and what to do when you hit compatibility errors, see the [dbt v2 readiness checklist](https://docs.getdbt.com/docs/dbt/dbt-readiness.md) and the [Upgrade to dbt v2 guides](https://docs.getdbt.com/guides/prepare-v2-upgrade.md). ##### Writing effective prompts Good prompts include the *scope* (which models or area of the project), the *intent* (the transformation or business logic you want), and any *constraints* (naming conventions, materialization, tests). Here are a few examples: | Task | Example prompt | | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | Build a new model | "Create a model called `fct_daily_revenue` that joins `stg_orders` and `stg_payments`, aggregates revenue by day, and materializes as a table." | | Refactor an existing model | "Refactor `fct_orders` to use incremental materialization. Keep existing tests and follow our naming conventions." | | Generate tests and docs | "Add `not_null` and `unique` tests to the primary key of `dim_customers`, and generate documentation for all columns." | For detailed guidance, patterns, and more examples across SQL, documentation, tests, and semantic models, see the [Prompt cookbook](https://docs.getdbt.com/guides/prompt-cookbook.md). Best practices for using dbt Wizard For recommended workflows on real project tasks — understanding a project, validating changes, building Semantic Layer definitions, and more — refer to [How to use dbt Wizard in your dbt project](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-1-intro.md). Most of these prompts work the same in Studio IDE. #### Ask questions in Explore mode Explore mode in Studio IDE lets you ask questions of your production data in plain language. dbt Wizard in Explore mode answers and explains but never changes your project. This option is great for exploratory data analysis and getting a quick understanding of your data. Explore mode queries with your personal warehouse credentials. 1. Open dbt Wizard and set the mode picker (bottom-left) to **Explore only** if needed. 2. Type your question and press **Enter**. Make sure you're specific and include the time period, grouping, and filter you care about: * `what was total revenue in Q2 2026, by month?` * `how many new customers signed up in July?` * `which regions grew fastest this year?` 3. dbt Wizard gives a plain-language summary of what it did, then the result. Switch between **Chart**, **Table**, and **SQL** to see the data your way. 4. Validate the data by using the **SQL** view to review the query or governed metric behind the answer. Note that you can't edit the SQL query but you can copy it. 5. Keep going by asking a follow-up in the same conversation. For example, `now break that out by region` works after your first question. Explore mode uses dbt-managed inference, so questions draw from your account's dbt Wizard [consumption pool](https://docs.getdbt.com/docs/dbt-ai/wizard-billing-faqs.md) like any other dbt Wizard usage. In the Studio IDE, the answer and its visualization appear inline in the dbt Wizard panel alongside your project. [![Explore mode in the Studio IDE, showing a chart and Chart/Table/SQL toggles in the dbt Wizard panel.](/img/docs/dbt-platform/wizard-ide-explore-viz.png?v=2 "Explore mode in the Studio IDE, showing a chart and Chart/Table/SQL toggles in the dbt Wizard panel.")](#)Explore mode in the Studio IDE, showing a chart and Chart/Table/SQL toggles in the dbt Wizard panel. #### Choose a model With dbt managed inference, you can switch between the supported managed models at any using the model picker dropdown next to the **Agent mode** control (where you choose **Ask for approval** or **Edit files automatically**). * In [Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) and the [dbt Wizard home tab](https://docs.getdbt.com/docs/platform/wizard-home.md), open the model picker dropdown next to the **Agent mode** control in the dbt Wizard panel, then select a model. * The picker lists the managed models available to you. Refer to the [Model Provider Rate table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) for the available models and their token rates. * If you [bring your own key (BYOK)](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md), dbt Wizard uses the provider and model you configured with your key rather than the managed model picker. [![The model picker dropdown next to the Agent mode control in the Wizard panel.](/img/docs/dbt-platform/wizard-model-picker.png?v=2 "The model picker dropdown next to the Agent mode control in the Wizard panel.")](#)The model picker dropdown next to the Agent mode control in the Wizard panel. #### Related docs * [About dbt Wizard in the dbt platform](https://docs.getdbt.com/docs/platform/wizard-platform.md) * [dbt v2 readiness checklist](https://docs.getdbt.com/docs/dbt/dbt-readiness.md) * [Develop with dbt Wizard](https://docs.getdbt.com/docs/platform/studio-ide/develop-studio-ai.md) * [Prompt cookbook](https://docs.getdbt.com/guides/prompt-cookbook.md) * [Semantic models](https://docs.getdbt.com/docs/build/semantic-models.md) * [About dbt AI and intelligence](https://docs.getdbt.com/docs/dbt-ai/about-dbt-ai.md) --- ### dbt Wizard slash command reference Beta ### dbt Wizard slash command reference [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Local development Full reference for dbt Wizard interactive TUI slash commands. Auto-generated page This page is auto-generated. If you see any issues, please [file an issue](https://github.com/dbt-labs/docs.getdbt.com/issues) and we'll be happy to sort it out. This page covers the slash commands supported by dbt Wizard. These are `/commands` typed inside the interactive text based user interface (TUI) session to control dbt Wizard's behavior mid-session. See it in action and share your feedback Want to see dbt Wizard in action? Check out the [demo video](https://www.youtube.com/watch?v=-lIzh1xQWMA). We'd love to hear how dbt Wizard is working for you. Share your feedback by either running the `/feedback` slash command in your interactive terminal session or by going to the [#dbt-wizard](https://getdbt.slack.com/archives/C0B6KLW6T26) channel in the [dbt Community Slack](https://docs.getdbt.com/community/join?version=2.0). Thanks so much for your help in improving dbt Wizard and dbt data development! #### Slash commands Type `/` in the composer to open the command picker. Use arrow keys to navigate or keep typing to filter. Press **Tab** to queue a command while a task is running — it executes at the end of the current turn. ##### All slash commands | Command | Alias | Purpose | Available during task | | --------------------------------------------------- | -------- | ---------------------------------------------------------------------- | --------------------- | | [`/model`](#model-and-ai) | — | Choose what model and reasoning effort to use | ✗ | | [`/providers`](#model-and-ai) | — | Configure model providers and BYOK credentials | ✓ | | [`/ide`](#review-and-context) | — | Include current selection, open files, and other context from your IDE | ✓ | | [`/permissions`](#permissions-and-safety) | — | Choose what Wizard is allowed to do | ✗ | | [`/keymap`](#customization) | — | Remap TUI shortcuts | ✗ | | [`/vim`](#customization) | — | Toggle Vim mode for the composer | ✗ | | [`/setup-default-sandbox`](#permissions-and-safety) | — | Set up elevated agent sandbox | ✗ | | [`/sandbox-add-read-dir`](#permissions-and-safety) | — | Let sandbox read a directory: /sandbox-add-read-dir \<absolute\_path> | ✗ | | [`/experimental`](#customization) | — | Toggle experimental features | ✗ | | [`/approve`](#customization) | — | Approve one retry of a recent auto-review denial | ✓ | | [`/memories`](#memory) | — | Configure memory use and generation | ✗ | | [`/skills`](#skills-and-extensions) | — | Use skills to improve how Wizard performs specific tasks | ✓ | | [`/hooks`](#skills-and-extensions) | — | View and manage lifecycle hooks | ✓ | | [`/review`](#review-and-context) | — | Review my current changes and find issues | ✗ | | [`/rename`](#session-management) | — | Rename the current thread | ✓ | | [`/new`](#session-management) | — | Start a new chat during a conversation | ✗ | | [`/resume`](#session-management) | — | Resume a saved chat | ✗ | | [`/fork`](#session-management) | — | Fork the current chat | ✗ | | [`/init`](#session-management) | — | Create an AGENTS.md file with instructions for Wizard | ✗ | | [`/compact`](#session-management) | — | Summarize conversation to prevent hitting the context limit | ✗ | | [`/plan`](#long-running-tasks) | — | Switch to Plan mode | ✗ | | [`/goal`](#long-running-tasks) | — | Set or view the goal for a long-running task | ✓ | | [`/agent`](#long-running-tasks) | — | Switch the active agent thread | ✓ | | [`/side`](#long-running-tasks) | — | Start a side conversation in an ephemeral fork | ✓ | | [`/btw`](#long-running-tasks) | — | Start a side conversation in an ephemeral fork | ✓ | | [`/copy`](#review-and-context) | — | Copy last response as markdown | ✓ | | [`/raw`](#review-and-context) | — | Toggle raw scrollback mode for copy-friendly terminal selection | ✓ | | [`/diff`](#review-and-context) | — | Show git diff (including untracked files) | ✓ | | [`/mention`](#review-and-context) | — | Mention a file | ✓ | | [`/status`](#session-info) | — | Show current session configuration and token usage | ✓ | | [`/config`](#session-info) | — | View and manage configuration | ✓ | | [`/debug-config`](#session-info) | — | Show config layers and requirement sources for debugging | ✓ | | [`/title`](#customization) | — | Configure which items appear in the terminal title | ✓ | | [`/statusline`](#customization) | — | Configure which items appear in the status line | ✓ | | [`/theme`](#customization) | — | Choose a syntax highlighting theme | ✗ | | [`/pets`](#customization) | `/pet` | Choose or hide the terminal pet | ✗ | | [`/overview`](#review-and-context) | — | Render the dbt project overview card | ✓ | | [`/mcp`](#skills-and-extensions) | — | List configured MCP tools; use /mcp verbose for details | ✓ | | [`/apps`](#skills-and-extensions) | — | Manage apps | ✓ | | [`/plugins`](#skills-and-extensions) | — | Browse plugins | ✓ | | [`/quit`](#exit) | — | Exit Wizard | ✓ | | [`/exit`](#exit) | — | Exit Wizard | ✓ | | [`/feedback`](#session-info) | — | Send feedback to maintainers | ✓ | | [`/ps`](#background-terminals) | — | List background terminals | ✓ | | [`/stop`](#background-terminals) | `/clean` | Stop all background terminals | ✓ | | [`/clear`](#session-management) | — | Clear the terminal and start a new chat | ✗ | | [`/personality`](#model-and-ai) | — | Choose a communication style for Wizard | ✗ | | [`/realtime`](#realtime-experimental) | — | Toggle realtime voice mode (experimental) | ✓ | | [`/settings`](#realtime-experimental) | — | Configure realtime microphone/speaker | ✓ | | [`/subagents`](#long-running-tasks) | — | Switch the active agent thread | ✓ | *** ##### Model and AI Control the AI model, provider, speed, and response style. | Command | Inline args | Description | | -------------- | ----------- | ---------------------------------------------- | | `/model` | — | Choose what model and reasoning effort to use | | `/providers` | — | Configure model providers and BYOK credentials | | `/personality` | — | Choose a communication style for Wizard | *** ##### Session management Start, resume, branch, and clean up sessions. | Command | Inline args | Description | | ---------- | ----------- | ----------------------------------------------------------- | | `/new` | — | Start a new chat during a conversation | | `/clear` | — | Clear the terminal and start a new chat | | `/resume` | Yes | Resume a saved chat | | `/fork` | — | Fork the current chat | | `/compact` | — | Summarize conversation to prevent hitting the context limit | | `/rename` | Yes | Rename the current thread | | `/init` | — | Create an AGENTS.md file with instructions for Wizard | *** ##### Review and context Pull information into the session or trigger a code review. | Command | Inline args | Description | | ----------- | ----------- | ---------------------------------------------------------------------- | | `/review` | Yes | Review my current changes and find issues | | `/diff` | — | Show git diff (including untracked files) | | `/mention` | — | Mention a file | | `/copy` | — | Copy last response as markdown | | `/raw` | Yes | Toggle raw scrollback mode for copy-friendly terminal selection | | `/overview` | — | Render the dbt project overview card | | `/ide` | Yes | Include current selection, open files, and other context from your IDE | *** ##### Permissions and safety Control what Wizard is allowed to execute. | Command | Inline args | Description | | ------------------------ | ----------- | --------------------------------------------------------------------- | | `/permissions` | — | Choose what Wizard is allowed to do | | `/setup-default-sandbox` | — | Set up elevated agent sandbox | | `/sandbox-add-read-dir` | Yes | Let sandbox read a directory: /sandbox-add-read-dir \<absolute\_path> | *** ##### Customization Appearance, keybindings, and UI preferences. | Command | Inline args | Description | | --------------------- | ----------- | -------------------------------------------------- | | `/theme` | — | Choose a syntax highlighting theme | | `/keymap` | Yes | Remap TUI shortcuts | | `/vim` | — | Toggle Vim mode for the composer | | `/statusline` | — | Configure which items appear in the status line | | `/title` | — | Configure which items appear in the terminal title | | `/experimental` | — | Toggle experimental features | | `/approve` | — | Approve one retry of a recent auto-review denial | | `/pets` (alias: /pet) | Yes | Choose or hide the terminal pet | *** ##### Skills and extensions Manage capabilities Wizard can use during a session. | Command | Inline args | Description | | ---------- | ----------- | -------------------------------------------------------- | | `/skills` | — | Use skills to improve how Wizard performs specific tasks | | `/hooks` | — | View and manage lifecycle hooks | | `/mcp` | Yes | List configured MCP tools; use /mcp verbose for details | | `/apps` | — | Manage apps | | `/plugins` | — | Browse plugins | *** ##### Long-running tasks Manage multi-turn goals, parallel agents, and branched conversations. | Command | Inline args | Description | | ------------ | ----------- | ---------------------------------------------- | | `/plan` | Yes | Switch to Plan mode | | `/goal` | Yes | Set or view the goal for a long-running task | | `/agent` | — | Switch the active agent thread | | `/subagents` | — | Switch the active agent thread | | `/side` | Yes | Start a side conversation in an ephemeral fork | | `/btw` | Yes | Start a side conversation in an ephemeral fork | *** ##### Background terminals Inspect and control shell processes Wizard has running in the background. | Command | Inline args | Description | | ----------------------- | ----------- | ----------------------------- | | `/ps` | — | List background terminals | | `/stop` (alias: /clean) | — | Stop all background terminals | *** ##### Memory Control how Wizard stores and uses memory across sessions. | Command | Inline args | Description | | ----------- | ----------- | ----------------------------------- | | `/memories` | — | Configure memory use and generation | *** ##### Session info Inspect the current session state without changing anything. | Command | Inline args | Description | | --------------- | ----------- | -------------------------------------------------------- | | `/status` | — | Show current session configuration and token usage | | `/config` | — | View and manage configuration | | `/debug-config` | — | Show config layers and requirement sources for debugging | | `/feedback` | — | Send feedback to maintainers | *** ##### Exit | Command | Inline args | Description | | ------- | ----------- | ----------- | | `/quit` | — | Exit Wizard | | `/exit` | — | Exit Wizard | *** ##### Realtime (experimental) | Command | Inline args | Description | | ----------- | ----------- | ----------------------------------------- | | `/realtime` | — | Toggle realtime voice mode (experimental) | | `/settings` | — | Configure realtime microphone/speaker | *** #### Related docs * [Wizard CLI command reference](https://docs.getdbt.com/docs/dbt-ai/wizard-cli-reference.md) for flags and subcommands * [Use subagents](https://docs.getdbt.com/docs/dbt-ai/wizard-subagents.md) for the `/agent` and `/subagents` commands * [Headless mode](https://docs.getdbt.com/docs/dbt-ai/wizard-headless.md) for using `wizard exec` in scripts and CI * [Configuration reference](https://docs.getdbt.com/docs/dbt-ai/wizard-config.md) for `config.toml` keys and precedence * [dbt command reference](https://docs.getdbt.com/reference/dbt-commands.md) for `dbt run`, `dbt build`, and other dbt commands --- ### dbt Wizard use cases Common use cases for dbt Wizard, with example prompts and what to expect. dbt Wizard works best when you give it a clear scope (which dbt model or area), an intent (what you want to change or learn), and any constraints (naming conventions, materialization, tests). The following examples follow that pattern. * [Build a new model](#build-a-new-model) * [Refactor to incremental](#refactor-to-incremental) * [Add tests and docs](#add-tests-and-docs) * [Debug a job failure](#debug-a-job-failure) * [Assess source impact](#assess-source-impact) * [Rename a column project-wide](#rename-a-column-project-wide) * [Multi-file changes](#multi-file-changes) * [Validate before shipping](#validate-before-shipping) * [Add a semantic model](#add-a-semantic-model) This page assumes you're using dbt Wizard in the terminal with an active session or in dbt platform. For examples of using dbt Wizard in the Studio IDE, refer to the [Prompt cookbook](https://docs.getdbt.com/guides/prompt-cookbook.md). To use dbt Wizard in the CLI, use the `wizard` [command reference](https://docs.getdbt.com/docs/dbt-ai/wizard-cli-reference.md). Best practices for using dbt Wizard Once you're set up, refer to [How to use dbt Wizard in your dbt project](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-1-intro.md) for recommended workflows on real project tasks. #### Build a new model You have clean source data and want a new mart model without writing all the SQL by hand. **Example prompt:** ```text Create a model called `fct_monthly_revenue` that joins `stg_orders` and `stg_payments`, groups by `month` and `customer_id`, and materializes as a table. Add `not_null` tests to the primary key and a unique test on the grain. ``` **What dbt Wizard does:** 1. Reads `stg_orders` and `stg_payments` from your project index to understand the available columns 2. Generates `fct_monthly_revenue.sql` with the join and aggregation logic 3. Creates a matching YAML block with the tests you asked for 4. Shows a diff for both files — you approve before anything is saved **Tips:** * Reference your existing staging models by name so dbt Wizard uses the right columns * If your project has a team style guide in `.agents/skills/`, dbt Wizard picks it up automatically and applies naming and materialization conventions #### Refactor to incremental A full-refresh mart is getting slow. You want to switch it to incremental without breaking existing tests. **Example prompt:** ```text Refactor `fct_orders` to use incremental materialization with a `unique_key` on `order_id` and an updated_at filter. Keep all existing tests and don't change the output schema. ``` **What dbt Wizard does:** 1. Reads the current `fct_orders.sql` and its YAML 2. Rewrites the model with an `{% if is_incremental() %}` filter 3. Adds the `unique_key` and `materialized = 'incremental'` config 4. Flags if any existing tests might behave differently on incremental runs **Tips:** * Tell dbt Wizard which column to use as the high-watermark: it can infer a likely candidate from your schema but an explicit instruction is more reliable * Ask it to add a `full_refresh` note to the model description so future maintainers understand the intent #### Add tests and docs You've inherited a model with no tests or YAML. You want coverage without writing boilerplate. **Example prompt:** ```text Generate tests and documentation for dim_customers. Add not_null and unique on customer_id, not_null on email, and accepted_values on customer_status with values active, churned, and prospect. Write a column description for each. ``` **What dbt Wizard does:** 1. Reads `dim_customers.sql` and any existing YAML 2. Generates a complete `dim_customers.yml` with model description, column descriptions, and the tests you specified 3. Infers reasonable descriptions from column names — you review and edit before approving **Tips:** * You can ask dbt Wizard to write descriptions in a specific voice or format: "Write the descriptions in plain language, one sentence each" * To document a whole layer at once: "Generate documentation for all models in `models/staging/` that don't have a YAML file yet" For a workflow that finds coverage gaps and checks candidate assertions against warehouse data, refer to [Add data-informed tests with dbt Wizard](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-4-data-informed-tests.md). #### Debug a job failure A dbt platform job failed overnight. You want to understand why without digging through logs manually. **Example prompt:** ```text The nightly job failed. What's the root cause and how do I fix it? ``` **What dbt Wizard does:** 1. Uses the run evidence you provide, or connected dbt MCP tools, to retrieve and inspect job run details 2. Identifies the failing model, the error message, and the likely cause 3. Proposes a fix and shows the diff 4. Notes if your local branch differs from the job's branch so you have full context **Tips:** * You can be more specific: "What caused the failure in `fct_orders` in the last run of the Production job?" * Wizard won't apply a fix without your approval, which is especially useful when the failure is in a production model For the evidence-gathering, diagnosis, and validation procedure, refer to [Debug a failed dbt job with dbt Wizard](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-5-debug-failed-job.md). #### Assess source impact Before modifying a stg model, you want to know what other downstream models might be affected. **Example prompt:** ```text If I change the grain of stg_payments from one row per payment to one row per payment attempt, which downstream models break? ``` **What dbt Wizard does:** 1. Runs a lineage and impact query against your project to find all downstream dependents of `stg_payments` 2. Identifies which models join on payment-level keys and would be affected by a grain change 3. Lists the models by severity — marts and exposures first, then intermediate models 4. Suggests which models need to be updated as part of the change **Tips:** * You don't need to be precise about the change: even a "what depends on stg\_payments?" gives you the blast radius * Follow up with: "Write a migration plan for making this change safely" #### Rename a column project-wide A source column has been renamed. You need to update all references without missing anything. **Example prompt:** ```text The source column user_id in raw_customers has been renamed to customer_id. Update stg_customers and find any downstream models that reference user_id directly. ``` **What dbt Wizard does:** 1. Updates the column alias in `stg_customers.sql` 2. Searches downstream models for direct references to `user_id` 3. Proposes updates to any models that would break 4. Updates YAML column descriptions to match **Tips:** * Always run `dbt compile` after dbt Wizard's changes to catch any references it might have missed * Ask Wizard to "check for user\_id in any raw SQL strings or Jinja macros too" for thorough coverage #### Multi-file changes You need to make a change that touches multiple files at once — a model rename, a contract update, or a schema change — and have all the related files stay in sync. **Example prompt:** ```text Rename dim_users to dim_customers. Update the model file, its YAML, every downstream ref(), the tests, the documentation, and any exposures that point to it. ``` **What dbt Wizard does:** 1. Renames `dim_users.sql` to `dim_customers.sql` 2. Updates the model name in the corresponding YAML 3. Searches the project for `ref('dim_users')` and rewrites each occurrence 4. Updates any tests, docs blocks, and exposures that reference the old name 5. Compiles the project to confirm every reference resolves before showing you the diff 6. Flags anything it couldn't update automatically (for example, raw SQL strings or external dashboards) **Tips:** * Multi-file changes are coordinated as a single diff: review them together rather than file by file * For column type or schema changes, name the new type explicitly: "Change `order_amount` in `fct_orders` from `numeric` to `decimal(18,2)` and update downstream models and tests" #### Validate before shipping For changes where correctness matters more than speed, ask dbt Wizard to assess impact and validate the result against your project. **Example prompt:** ```text Add not_null and unique tests to the primary key of dim_customers. Use heavy validation, investigate any failures, and summarize skipped checks. ``` **What dbt Wizard does:** 1. Generates the YAML for the new tests 2. Assesses the affected resources and proposes a validation plan 3. Runs the approved compile, build, test, and comparison steps for the selected validation level 4. Reports failures, differences, unresolved risks, and skipped checks 5. Shows the proposed changes for you to review In dbt Wizard CLI, choose light, medium, heavy, or skipped validation based on the risk and cost of the change. Follow the [validation workflow](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-3-validate-changes.md) for the checks included at each level. **Tips:** * State the business behavior that must remain true, not only the commands to run #### Add a semantic model You have a mart model and want to expose it via the dbt Semantic Layer. **Example prompt:** ```text Create a semantic model for fct_orders. Include a revenue metric that sums order_total, a count of orders, and time dimensions on order_date at day, week, and month granularity. ``` **What dbt Wizard does:** 1. Reads `fct_orders.sql` and its YAML to understand available columns 2. Determines the dbt version and selects the compatible Semantic Layer YAML structure 3. Proposes entities, dimensions, and metrics based on the model grain and business request 4. Adds the definitions you approve and validates their references **Tips:** * If you're unsure what entities to use, ask first: "What would be good entities for a semantic model on fct\_orders?" * dbt Wizard follows the [dbt Semantic Layer documentation](https://docs.getdbt.com/docs/build/semantic-models.md): you can ask it to explain any generated field For version-specific examples and validation steps, refer to [Build Semantic Layer definitions with dbt Wizard](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-7-semantic-layer.md). #### Related docs * [Use dbt Wizard locally](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) * [Understand a dbt project](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-2-understand-project.md) * [Validate dbt changes](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-3-validate-changes.md) * [Add data-informed tests](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-4-data-informed-tests.md) * [Debug a failed job](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-5-debug-failed-job.md) * [Build Semantic Layer definitions](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-7-semantic-layer.md) * [dbt Wizard overview](https://docs.getdbt.com/docs/dbt-ai/wizard-cli.md) * [Configure BYOK](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) * [dbt Wizard in Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md): same agent, in the dbt platform * [Prompt cookbook](https://docs.getdbt.com/guides/prompt-cookbook.md): more prompt patterns for the dbt Wizard in Studio IDE (many apply to the CLI too) --- ### Deploy dbt dbt platform Use the dbt platform's capabilities to seamlessly run a dbt job in production or staging environments. Rather than run dbt commands manually from the command line, you can leverage the [dbt platform's in-app scheduling](https://docs.getdbt.com/docs/deploy/job-scheduler.md) to automate how and when you execute dbt. The dbt platform offers the easiest and most reliable way to run your dbt project in production. Effortlessly promote high quality code from development to production and build fresh data assets that your business intelligence tools and end users query to make business decisions. Deploying with dbt lets you: * Keep production data fresh on a timely basis * Ensure CI and production pipelines are efficient * Identify the root cause of failures in deployment environments * Maintain high-quality code and data in production * Gain visibility into the [health](https://docs.getdbt.com/docs/explore/data-tile.md) of deployment jobs, models, and tests * Uses [exports](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md) to write [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md) in your data platform for reliable and fast metric reporting * [Visualize](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md) and [orchestrate](https://docs.getdbt.com/docs/platform-integrations/orchestrate-exposures.md) downstream exposures to understand how models are used in downstream tools and proactively refresh the underlying data sources during scheduled dbt jobs. [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") * Use [dbt's Git repository caching](https://docs.getdbt.com/docs/platform/account-settings.md#git-repository-caching) to protect against third-party outages and improve job run reliability. [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") * Use [Hybrid projects](https://docs.getdbt.com/docs/deploy/hybrid-projects.md) to upload dbt artifacts into the dbt platform for central visibility, cross-project referencing, and easier collaboration. [Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Before continuing, make sure you understand dbt's approach to [deployment environments](https://docs.getdbt.com/docs/deploy/deploy-environments.md). Learn how to use dbt's features to help your team ship timely and quality production data more easily. #### Deploy with dbt [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/job-scheduler.md) ###### [Job scheduler](https://docs.getdbt.com/docs/deploy/job-scheduler.md) [The job scheduler is the backbone of running jobs in the dbt platform, bringing power and simplicity to building data pipelines in both continuous integration and production environments.](https://docs.getdbt.com/docs/deploy/job-scheduler.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/deploy-jobs.md) ###### [Deploy jobs](https://docs.getdbt.com/docs/deploy/deploy-jobs.md) [Create and schedule jobs for the job scheduler to run.](https://docs.getdbt.com/docs/deploy/deploy-jobs.md) <br /> <br /> [Runs on a schedule, by API, or after another job completes.](https://docs.getdbt.com/docs/deploy/deploy-jobs.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) ###### [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) [Intelligently determines which models to build by detecting changes in code or data at each job run.](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/continuous-integration.md) ###### [Continuous integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md) [Set up CI checks so you can build and test any modified code in a staging environment when you open PRs and push new commits to your dbt repository.](https://docs.getdbt.com/docs/deploy/continuous-integration.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/continuous-deployment.md) ###### [Continuous deployment](https://docs.getdbt.com/docs/deploy/continuous-deployment.md) [Set up merge jobs to ensure the latest code changes are always in production when pull requests are merged to your Git repository.](https://docs.getdbt.com/docs/deploy/continuous-deployment.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/job-commands.md) ###### [Job commands](https://docs.getdbt.com/docs/deploy/job-commands.md) [Configure which dbt commands to execute when running a dbt job.](https://docs.getdbt.com/docs/deploy/job-commands.md) <br /> #### Monitor jobs and alerts [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/orchestrate-exposures.md) ###### [Visualize and orchestrate exposures](https://docs.getdbt.com/docs/deploy/orchestrate-exposures.md) [Learn how to use dbt to automatically generate downstream exposures from dashboards and proactively refresh the underlying data sources during scheduled dbt jobs.](https://docs.getdbt.com/docs/deploy/orchestrate-exposures.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/artifacts.md) ###### [Artifacts](https://docs.getdbt.com/docs/deploy/artifacts.md) [dbt generates and saves artifacts for your project, which it uses to power features like creating docs for your project and reporting the freshness of your sources.](https://docs.getdbt.com/docs/deploy/artifacts.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/job-notifications.md) ###### [Job notifications](https://docs.getdbt.com/docs/deploy/job-notifications.md) [Receive email or Slack channel notifications when a job run succeeds, fails, or is canceled so you can respond quickly and begin remediation if necessary.](https://docs.getdbt.com/docs/deploy/job-notifications.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/model-notifications.md) ###### [Model notifications](https://docs.getdbt.com/docs/deploy/model-notifications.md) [Receive email notifications in real time about issues encountered by your models and tests while a job is running.](https://docs.getdbt.com/docs/deploy/model-notifications.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/run-visibility.md) ###### [Run visibility](https://docs.getdbt.com/docs/deploy/run-visibility.md) [View the history of your runs and the model timing dashboard to help identify where improvements can be made to the scheduled jobs.](https://docs.getdbt.com/docs/deploy/run-visibility.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/retry-jobs.md) ###### [Retry jobs](https://docs.getdbt.com/docs/deploy/retry-jobs.md) [Rerun your errored jobs from start or the failure point.](https://docs.getdbt.com/docs/deploy/retry-jobs.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/source-freshness.md) ###### [Source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) [Enable snapshots to capture the freshness of your data sources and configure how frequent these snapshots should be taken. This can help you determine whether your source data freshness is meeting your SLAs.](https://docs.getdbt.com/docs/deploy/source-freshness.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/webhooks.md) ###### [Webhooks](https://docs.getdbt.com/docs/deploy/webhooks.md) [Create outbound webhooks to send events about your dbt jobs' statuses to other systems in your organization.](https://docs.getdbt.com/docs/deploy/webhooks.md) <br /> #### Hybrid projects [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/deploy/hybrid-projects.md) ###### [Hybrid projects](https://docs.getdbt.com/docs/deploy/hybrid-projects.md) [Use Hybrid projects to upload dbt artifacts into the dbt platform for central visibility, cross-project referencing, and easier collaboration.](https://docs.getdbt.com/docs/deploy/hybrid-projects.md) <br /> #### Related docs * [Use exports to materialize saved queries](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md) * [Integrate with other orchestration tools](https://docs.getdbt.com/docs/deploy/deployment-tools.md) --- ### Deploy jobs dbt platform You can use deploy jobs to build production data assets. Deploy jobs make it easy to run dbt commands against a project in your cloud data platform, triggered either by schedule or events. Each job run in dbt will have an entry in the job's run history and a detailed run overview, which provides you with: * Job trigger type * Commit SHA * Environment name * Sources and documentation info, if applicable * Job run details, including run timing, [model timing data](https://docs.getdbt.com/docs/deploy/run-visibility.md#model-timing-tab), and [artifacts](https://docs.getdbt.com/docs/deploy/artifacts.md) * Detailed run steps with logs and their run step statuses. For dbt v2 runs, you can also download OpenTelemetry logs from individual steps. Refer to [Downloading logs](https://docs.getdbt.com/docs/deploy/run-visibility.md#access-logs). You can create a deploy job and configure it to run on [scheduled days and times](#schedule-days), enter a [custom cron schedule](#cron-schedule), or [trigger the job after another job completes](#trigger-on-job-completion). #### Prerequisites * You must have a [dbt account](https://www.getdbt.com/signup/) and [Developer seat license](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md). * For the [Trigger on job completion](#trigger-on-job-completion) feature, your dbt account must be on the [Starter or an Enterprise-tier](https://www.getdbt.com/pricing/) plan. * You must have a dbt project connected to a [data platform](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md). * You must have [access permission](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md) to view, create, modify, or run jobs. * You must set up a [deployment environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md). #### Create and schedule jobs info dbt uses [Coordinated Universal Time](https://en.wikipedia.org/wiki/Coordinated_Universal_Time) (UTC) for all jobs, including those configured with cron. It does not adjust for your local timezone or daylight saving time. For example: * 0 means 12am (midnight) UTC * 12 means 12pm (afternoon) UTC * 23 means 11pm UTC 1. On your deployment environment page, click **Create job** > **Deploy job** to create a new deploy job. 2. Options in the **Job settings** section: * **Job name** — Specify the name for the deploy job. For example, `Daily build`. * (Optional) **Description** — Provide a description of what the job does (for example, what the job consumes and what the job produces). * **Environment** — By default, it’s set to the deployment environment you created the deploy job from. 3. Options in the **Execution settings** section: * [**Commands**](https://docs.getdbt.com/docs/deploy/job-commands.md#built-in-commands) — By default, it includes the `dbt build` command. Click **Add command** to add more [commands](https://docs.getdbt.com/docs/deploy/job-commands.md) that you want to be invoked when the job runs. During a job run, [built-in commands](https://docs.getdbt.com/docs/deploy/job-commands.md#built-in-commands) are "chained" together and if one run step fails, the entire job fails with an "Error" status. * [**Generate docs on run**](https://docs.getdbt.com/docs/deploy/job-commands.md#checkbox-commands) (not applicable to dbt v2 jobs) — Enable this option if you want to [generate project docs](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) when this deploy job runs. If the step fails, the job can succeed if subsequent steps pass. * [**Run source freshness**](https://docs.getdbt.com/docs/deploy/job-commands.md#checkbox-commands) — Enable this option to invoke the `dbt source freshness` command before running the deploy job. If the step fails, the job can succeed if subsequent steps pass. Refer to [Source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) for more details. * [**Enable dbt State**](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") — dbt State reduces unnecessary model rebuilds by reusing nodes when neither the logic nor the data has changed. For more details, refer to [Setting up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) and [Enabling dbt State on individual jobs](https://docs.getdbt.com/docs/deploy/dbt-state-enable-jobs.md). 4. Options in the **Triggers** section: * **Run on schedule** — Run the deploy job on a set schedule. * **Timing** — Specify whether to [schedule](#schedule-days) the deploy job using **Intervals** that run the job every specified number of hours, **Specific hours** that run the job at specific times of day, or **Cron schedule** that run the job specified using [cron syntax](#cron-schedule). * **Days of the week** — By default, it’s set to every day when **Intervals** or **Specific hours** is chosen for **Timing**. Using `state:modified` on a scheduled job Using a [`state:modified`](https://docs.getdbt.com/reference/node-selection/methods.md#state) selector on a scheduled job can result in the job completing successfully with zero models built when no changes are detected since the last deferred run. Refer to [Scheduled jobs and state:modified](#scheduled-jobs-and-statemodified) for details and recommendations. * **Run when another job finishes** — Run the deploy job when another *upstream* deploy [job completes](#trigger-on-job-completion). * **Project** — Specify the parent project that has that upstream deploy job. * **Job** — Specify the upstream deploy job. * **Completes on** — Select the job run status(es) that will [enqueue](https://docs.getdbt.com/docs/deploy/job-scheduler.md#scheduler-queue) the deploy job. 5. (Optional) Options in the **Advanced settings** section: * **Environment variables** — Define [environment variables](https://docs.getdbt.com/docs/build/environment-variables.md) to customize the behavior of your project when the deploy job runs. * **Target name** — Define the [target name](https://docs.getdbt.com/docs/build/custom-target-names.md) to customize the behavior of your project when the deploy job runs. Environment variables and target names are often used interchangeably. * **Run timeout** — Cancel the deploy job if the run time exceeds the timeout value. * **Compare changes against** — By default, it’s set to **No deferral**. Select either **Environment** or **This Job** to let dbt know what it should compare the changes against. info Older versions of dbt only allow you to defer to a specific job instead of an environment. Deferral to a job compares state against the project code that was run in the deferred job's last successful run. While deferral to an environment is more efficient as dbt will compare against the project representation (which is stored in the `manifest.json`) of the last successful deploy job run that executed in the deferred environment. By considering *all* deploy jobs that run in the deferred environment, dbt will get a more accurate, latest project representation state. * **dbt version** — By default, it’s set to inherit the [dbt version](https://docs.getdbt.com/docs/dbt-versions.md) from the environment. dbt Labs strongly recommends that you don't change the default setting. This option to change the version at the job level is useful only when you upgrade a project to the next dbt version; otherwise, mismatched versions between the environment and job can lead to confusing behavior. * **Threads** — By default, it’s set to 4 [threads](https://docs.getdbt.com/docs/local/profiles.yml.md#understanding-threads). Increase the thread count to increase model execution concurrency. [![Example of Advanced Settings on the Deploy Job page](/img/docs/dbt-platform/using-dbt-platform/deploy-job-adv-settings.png?v=2 "Example of Advanced Settings on the Deploy Job page")](#)Example of Advanced Settings on the Deploy Job page ##### Schedule days To set your job's schedule, use the **Run on schedule** option to choose specific days of the week, and select customized hours or intervals. Under **Timing**, you can either use regular intervals for jobs that need to run frequently throughout the day or customizable hours for jobs that need to run at specific times: * **Intervals** — Use this option to set how often your job runs, in hours. For example, if you choose **Every 2 hours**, the job will run every 2 hours from midnight UTC. This doesn't mean that it will run at exactly midnight UTC. However, subsequent runs will always be run with the same amount of time between them. For example, if the previous scheduled pipeline ran at 00:04 UTC, the next run will be at 02:04 UTC. This option is useful if you need to run jobs multiple times per day at regular intervals. * **Specific hours** — Use this option to set specific times when your job should run. You can enter a comma-separated list of hours (in UTC) when you want the job to run. For example, if you set it to `0,12,23,` the job will run at midnight, noon, and 11 PM UTC. Job runs will always be consistent between both hours and days, so if your job runs at 00:05, 12:05, and 23:05 UTC, it will run at these same hours each day. This option is useful if you want your jobs to run at specific times of day and don't need them to run more frequently than once a day. ##### Cron schedule To fully customize the scheduling of your job, choose the **Cron schedule** option and use cron syntax. With this syntax, you can specify the minute, hour, day of the month, month, and day of the week, allowing you to set up complex schedules like running a job on the first Monday of each month. **Note:** Cron schedules in dbt use UTC and don't convert to your local timezone or adjust for daylight saving time. **Cron frequency** To enhance performance, job scheduling frequencies vary by dbt plan: * Developer plans: dbt sets a minimum interval of every 10 minutes for scheduling jobs. This means scheduling jobs to run more frequently, or at less than 10 minute intervals, is not supported. * Starter, Enterprise, and Enterprise+ plans: No restrictions on job execution frequency. **Examples** Use tools such as [crontab.guru](https://crontab.guru/) to generate the correct cron syntax. This tool allows you to input cron snippets and return their plain English translations. The dbt job scheduler supports using `L` to schedule jobs on the last day of the month. Examples of cron job schedules: * `0 * * * *`: Every hour, at minute 0. * `*/5 * * * *`: Every 5 minutes. (Not available on Developer plans) * `5 4 * * *`: At exactly 4:05 AM UTC. * `30 */4 * * *`: At minute 30 past every 4th hour (such as 4:30 AM, 8:30 AM, 12:30 PM, and so on, all UTC). * `0 0 */2 * *`: At 12:00 AM (midnight) UTC every other day. * `0 0 * * 1`: At midnight UTC every Monday. * `0 0 L * *`: At 12:00 AM (midnight), on the last day of the month. * `0 0 L 1,2,3,4,5,6,8,9,10,11,12 *`: At 12:00 AM, on the last day of the month, only in January, February, March, April, May, June, August, September, October, November, and December. * `0 0 L 7 *`: At 12:00 AM, on the last day of the month, only in July. * `0 0 L * FRI,SAT`: At 12:00 AM, on the last day of the month, and on Friday and Saturday. * `0 12 L * *`: At 12:00 PM (afternoon), on the last day of the month. * `0 7 L * 5`: At 07:00 AM, on the last day of the month, and on Friday. * `30 14 L * *`: At 02:30 PM, on the last day of the month. * `0 4 * * MON#1`: At 4:00 AM on the first Monday of every month. ##### Scheduled jobs and `state:modified` [`state:modified`](https://docs.getdbt.com/reference/node-selection/methods.md#state) and `state:modified+` only detect changes to your project's code, configuration, or manifest-relevant metadata — not changes to the data itself, like new rows landing in a source table. When dbt detects no code or configuration changes since the deferred manifest was last produced, the job succeeds without building any models. This is expected behavior. If your job needs to build models on every scheduled run regardless of code changes, remove the `state:modified` selector from that job. Reserve it for [CI](https://docs.getdbt.com/docs/deploy/ci-jobs.md) or [merge jobs](https://docs.getdbt.com/docs/deploy/merge-jobs.md), where the intent is to build only what changed in a given pull request or merge. ##### Trigger on job completion [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") To *chain* deploy jobs together: 1. In the **Triggers** section, enable the **Run when another job finishes** option. 2. Select the project that has the deploy job you want to run after completion. 3. Specify the upstream (parent) job that, when completed, will trigger your job. * You can also use the [Create Job API](https://docs.getdbt.com/dbt-cloud/api-v2#/operations/Create%20Job) to do this. 4. In the **Completes on** option, select the job run status(es) that will [enqueue](https://docs.getdbt.com/docs/deploy/job-scheduler.md#scheduler-queue) the deploy job. [![Example of Trigger on job completion on the Deploy job page](/img/docs/deploy/deploy-job-completion.png?v=2 "Example of Trigger on job completion on the Deploy job page")](#)Example of Trigger on job completion on the Deploy job page 5. You can set up a configuration where an upstream job triggers multiple downstream (child) jobs and jobs in other projects. You must have proper [permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#project-role-permissions) to the project and job to configure the trigger. If another job triggers your job to run, you can find a link to the upstream job in the [run details section](https://docs.getdbt.com/docs/deploy/run-visibility.md#job-run-details). #### Delete a job To delete a job or multiple jobs in dbt: 1. Click **Deploy** on the navigation header. 2. Click **Jobs** and select the job you want to delete. 3. Click **Settings** on the top right of the page and then click **Edit**. 4. Scroll to the bottom of the page and click **Delete job** to delete the job. <br /> [![Delete a job](/img/docs/dbt-platform/platform-configuring-dbt-platform/delete-job.png?v=2 "Delete a job")](#)Delete a job 5. Confirm your action in the pop-up by clicking **Confirm delete** in the bottom right to delete the job immediately. This action cannot be undone. However, you can create a new job with the same information if the deletion was made in error. 6. Refresh the page, and the deleted job should now be gone. If you want to delete multiple jobs, you'll need to perform these steps for each job. If you're having any issues, feel free to [contact us](mailto:support@getdbt.com) for additional help. #### Job monitoring On the **Environments** page, there are two sections that provide an overview of the jobs for that environment: * **In progress** — Lists the currently in progress jobs with information on when the run started * **Top jobs by models built** — Ranks jobs by the number of models built over a specific time [![In progress jobs and Top jobs by models built](/img/docs/deploy/in-progress-top-jobs.png?v=2 "In progress jobs and Top jobs by models built")](#)In progress jobs and Top jobs by models built #### Job settings history You can view historical job settings changes over the last 90 days. To view the change history: 1. Navigate to **Orchestration** from the main menu and click **Jobs**. 2. Click a **job name**. 3. Click **Settings**. 4. Click **History**. [![Example of the job settings history.](/img/docs/deploy/job-history.png?v=2 "Example of the job settings history.")](#)Example of the job settings history. #### Related docs * [Run visibility](https://docs.getdbt.com/docs/deploy/run-visibility.md) * [Artifacts](https://docs.getdbt.com/docs/deploy/artifacts.md) * [Continuous integration (CI) jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md) * [Webhooks](https://docs.getdbt.com/docs/deploy/webhooks.md) --- ### Deploy your metrics dbt platform | Starter, Enterprise, Enterprise+ This section explains how you can perform a job run in your deployment environment in dbt to materialize and deploy your metrics. Currently, the deployment environment is only supported. 1. Once you’ve [defined your semantic models and metrics](https://docs.getdbt.com/guides/sl-qs.md?step=6), commit and merge your metric changes in your dbt project. 2. In dbt, create a new [deployment environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md#create-a-deployment-environment) or use an existing environment on dbt 1.6 or higher. * Note: Deployment environment is currently supported (*development experience coming soon*) 3. To create a new environment, navigate to **Orchestration** in the navigation menu, select **Environments**, and then select **Create new environment**. 4. Fill in your deployment credentials with the username and password for your data platform. You can name the schema anything you want. Click **Save** to create your new production environment. 5. [Create a new deploy job](https://docs.getdbt.com/docs/deploy/deploy-jobs.md#create-and-schedule-jobs) that runs in the environment you just created. Go back to the **Orchestration** menu, select **Jobs**, select **Create job**, and click **Deploy job**. 6. Set the job to run a `dbt parse` job to parse your projects and generate a [`semantic_manifest.json` artifact](https://docs.getdbt.com/reference/artifacts/sl-manifest.md) file(Applies to dbt v1.12 and later) and an [`osi_document.json` artifact](https://docs.getdbt.com/reference/artifacts/sl-manifest.md#apache-ossie-document). Although running `dbt build` isn't required, you can choose to do so if needed. note If you are on dbt v2, add the `dbt docs generate` command to your job to successfully deploy your metrics. 7. Run the job by clicking the **Run now** button. Monitor the job's progress in real-time through the **Run summary** tab. Once the job completes successfully, your dbt project, including the generated documentation, will be fully deployed and available for use in your production environment. If any issues arise, review the logs to diagnose and address any errors. What’s happening internally? * Merging the code into your main branch allows dbt to pull those changes and build the definition in the manifest produced by the run. <br /> * Re-running the job in the deployment environment helps materialize the models, which the metrics depend on, in the data platform. It also makes sure that the manifest is up to date. <br /> * The Semantic Layer APIs pull in the most recent manifest and enables your integration to extract metadata from it. #### Next steps After you've executed a job and deployed your Semantic Layer: * [Set up your Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md) in dbt. * Discover the [available integrations](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md), such as Tableau, Google Sheets, Microsoft Excel, and more. * Start querying your metrics with the [API query syntax](https://docs.getdbt.com/docs/dbt-apis/sl-jdbc.md#querying-the-api-for-metric-metadata). #### Related docs * [Optimize querying performance](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-cache.md) using declarative caching. * [Validate semantic nodes in CI](https://docs.getdbt.com/docs/deploy/ci-jobs.md#semantic-validations-in-ci) to ensure code changes made to dbt models don't break these metrics. * If you haven't already, learn how to [build your metrics and semantic models](https://docs.getdbt.com/docs/build/build-metrics-intro.md) in your development tool of choice. --- ### Deployment environments dbt platform Deployment environments in dbt are crucial for deploying dbt jobs in production and using features or integrations that depend on dbt metadata or results. To execute dbt, environments determine the settings used during job runs, including: * The version of dbt that will be used to run your project * The warehouse connection information (including the target database/schema settings) * The [connection profile](https://docs.getdbt.com/docs/platform/about-profiles.md) (the credentials dbt uses to connect) * The version of your code to execute A dbt project can have multiple deployment environments, providing you the flexibility and customization to tailor the execution of dbt jobs. You can use deployment environments to [create and schedule jobs](https://docs.getdbt.com/docs/deploy/deploy-jobs.md#create-and-schedule-jobs), [enable continuous integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md), or more based on your specific needs or requirements. Learn how to manage dbt environments To learn different approaches to managing dbt environments and recommendations for your organization's unique needs, read [dbt environment best practices](https://docs.getdbt.com/guides/set-up-ci.md). Learn more about development vs. deployment environments in [dbt Environments](https://docs.getdbt.com/docs/dbt-platform-environments.md). There are three types of deployment environments: * **Production**: Environment for transforming data and building pipelines for production use. * **Staging**: Environment for working with production tools while limiting access to production data. * **General**: General use environment for deployment development. We highly recommend using the `Production` environment type for the final, source of truth deployment data. There can be only one environment marked for final production workflows and we don't recommend using a `General` environment for this purpose. #### Create a deployment environment To create a new dbt deployment environment, navigate to **Orchestration** > **Environments** and then click **Create Environment**. Select **Deployment** as the environment type. The option will be greyed out if you already have a development environment. [![Navigate to Orchestration > Environments to create a deployment environment](/img/docs/dbt-platform/platform-configuring-dbt-platform/create-deploy-env.png?v=2 "Navigate to Orchestration > Environments to create a deployment environment")](#)Navigate to Orchestration > Environments to create a deployment environment ##### Set as production environment In dbt, each project can have one designated deployment environment, which serves as its production environment. This production environment is *essential* for using features like Catalog and cross-project references. It acts as the source of truth for the project's production state in dbt. [![Set your production environment as the default environment in your Environment Settings](/img/docs/dbt-platform/using-dbt-platform/prod-settings-1.png?v=2 "Set your production environment as the default environment in your Environment Settings")](#)Set your production environment as the default environment in your Environment Settings ##### Semantic Layer For customers using the Semantic Layer, the next section of environment settings is the Semantic Layer configurations. [The Semantic Layer setup guide](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md) has the most up-to-date setup instructions. You can also leverage the dbt Job scheduler to [validate your semantic nodes in a CI job](https://docs.getdbt.com/docs/deploy/ci-jobs.md#semantic-validations-in-ci) to ensure code changes made to dbt models don't break these metrics. #### Staging environment Use a staging environment to grant developers access to deployment workflows and tools while controlling access to production data. Staging environments enable you to achieve more granular control over permissions, data warehouse connections, and data isolation — within the purview of a single project in dbt. ##### Git workflow You can approach this in a couple of ways, but the most straightforward is configuring staging with a long-living branch (for example, `staging`) similar to, but separate from the primary branch (for example, `main`). In this scenario, the workflows would ideally move upstream from the Development environment -> Staging environment -> Production environment with developer branches feeding into the `staging` branch, then ultimately merging into `main`. In many cases, the `main` and `staging` branches will be identical after a merge and remain until the next batch of changes from the `development` branches are ready to be elevated. We recommend setting branch protection rules on `staging` similar to `main`. Some customers prefer to connect Development and Staging to their `main` branch and then cut release branches on a regular cadence (daily or weekly), which feeds into Production. ##### Why use a staging environment These are the primary motivations for using a staging environment: 1. An additional validation layer before changes are deployed into production. You can deploy, test, and explore your dbt models in staging. 2. Clear isolation between development workflows and production data. It enables developers to work in metadata-powered ways, using features like deferral and cross-project references, without accessing data in production deployments. 3. Provide developers with the ability to create, edit, and trigger ad hoc jobs in the staging environment, while keeping the production environment locked down using [environment-level permissions](https://docs.getdbt.com/docs/platform/manage-access/environment-permissions.md). **Conditional configuration of sources** enables you to point to "prod" or "non-prod" source data, depending on the environment you're running in. For example, this source will point to `<DATABASE>.sensitive_source.table_with_pii`, where `<DATABASE>` is dynamically resolved based on an environment variable. models/sources.yml ```yaml sources: - name: sensitive_source database: "{{ env_var('SENSITIVE_SOURCE_DATABASE') }}" tables: - name: table_with_pii ``` There is exactly one source (`sensitive_source`), and all downstream dbt models select from it as `{{ source('sensitive_source', 'table_with_pii') }}`. The code in your project and the shape of the DAG remain consistent across environments. By setting it up in this way, rather than duplicating sources, you get some important benefits. **Cross-project references in dbt Mesh:** Let's say you have `Project B` downstream of `Project A` with cross-project refs configured in the models. When developers work in the IDE for `Project B`, cross-project refs will resolve to the staging environment of `Project A`, rather than production. You'll get the same results with those refs when jobs are run in the staging environment. Only the production environment will reference the production data, keeping the data and access isolated without needing separate projects. **Faster development enabled by deferral:** If `Project B` also has a staging deployment, then references to unbuilt upstream models(Applies to dbt v1.11 and later) and [user-defined functions (UDFs)](https://docs.getdbt.com/docs/build/udfs.md) within `Project B` will resolve to that environment using [deferral](https://docs.getdbt.com/docs/platform/about-defer.md), rather than resolving to the models(Applies to dbt v1.11 and later) and functions in production. This saves developers time and warehouse spend, while preserving clear separation of environments. Finally, the staging environment has its own view in [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md), giving you a full view of your prod and pre-prod data. ##### Create a Staging environment [![Create a staging environment](/img/docs/dbt-platform/platform-configuring-dbt-platform/create-staging-environment.png?v=2 "Create a staging environment")](#)Create a staging environment Follow the steps outlined in [connection profiles](https://docs.getdbt.com/docs/platform/about-profiles.md) to complete the remainder of the environment setup. We recommend that the data warehouse credentials be for a dedicated user or service principal. #### Deployment connection A deployment environment needs two settings to run jobs: | Setting | Scope | Controls | | ------------------------- | ------------------------------------------- | ---------------------------------------------------------------------------- | | **Deployment connection** | Environment-level | Where dbt builds objects in your warehouse (database, schema, and warehouse) | | **Connection profile** | Project-level (assigned to the environment) | How dbt authenticates, and the credentials it uses to connect | You need to complete both settings as a deployment connection on its own isn't enough for jobs to run. You also need a [connection profile](#connection-profiles) assigned to the environment. Deployment connections Deployment connections are created and managed at the account-level for dbt accounts and assigned to an environment. To change warehouse type, we recommend creating a new environment. Each project can have multiple connections (Snowflake account, Redshift host, Bigquery project, Databricks host, and so on.) of the same warehouse type. Some details of that connection (databases/schemas/and so on.) can be overridden within this section of the dbt environment settings. This section determines the exact location in your warehouse dbt should target when building warehouse objects! This section will look a bit different depending on your warehouse provider. For all warehouses, use [extended attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) to override missing or inactive (grayed-out) settings. ##### Postgres This section will not appear if you are using Postgres, as all values are inferred from the project's connection. Use [extended attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) to override these values. ##### Redshift This section will not appear if you are using Redshift, as all values are inferred from the project's connection. Use [extended attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) to override these values. ##### Snowflake [![Snowflake Deployment Connection Settings](/img/docs/collaborate/snowflake-deploy-env-deploy-connection.png?v=2 "Snowflake Deployment Connection Settings")](#)Snowflake Deployment Connection Settings ###### Editable fields * **Role**: Snowflake role * **Database**: Target database * **Warehouse**: Snowflake warehouse ##### Bigquery This section will not appear if you are using Bigquery, as all values are inferred from the project's connection. Use [extended attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) to override these values. ##### Spark This section will not appear if you are using Spark, as all values are inferred from the project's connection. Use [extended attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) to override these values. ##### Databricks [![Databricks Deployment Connection Settings](/img/docs/collaborate/databricks-deploy-env-deploy-connection.png?v=2 "Databricks Deployment Connection Settings")](#)Databricks Deployment Connection Settings ###### Editable fields * **Catalog** (optional): [Unity Catalog namespace](https://docs.getdbt.com/docs/local/connect-data-platform/databricks-setup.md) #### Connection profiles The deployment connection sets *where* dbt builds. The connection profile sets *how* dbt authenticates. Deployment credentials are managed through connection profiles, which are created at the project level and assigned to deployment environments. Profiles define the credentials and attributes dbt uses to connect to your warehouse. To configure credentials for this environment, refer to [About dbt platform profiles](https://docs.getdbt.com/docs/platform/about-profiles.md). Jobs need a profile assigned to the environment, not only a deployment connection. #### Delete an environment Deleting an environment automatically deletes its associated job(s). If you want to keep those jobs, move them to a different environment first. Follow these steps to delete an environment in dbt: 1. Navigate to **Orchestration** > **Environments**. 2. Select the environment you want to delete. 3. Click **Settings** on the top right of the page and then click **Edit**. 4. Scroll to the bottom of the page and click **Delete** to delete the environment. [![Delete an environment](/img/docs/dbt-platform/platform-configuring-dbt-platform/delete-environment.png?v=2 "Delete an environment")](#)Delete an environment 5. Confirm your action in the pop-up by clicking **Confirm delete** in the bottom right to delete the environment immediately. This action cannot be undone. However, you can create a new environment with the same information if the deletion was made in error. 6. Refresh your page and the deleted environment should now be gone. To delete multiple environments, you'll need to perform these steps to delete each one. If you're having any issues, feel free to [contact us](mailto:support@getdbt.com) for additional help. #### Related docs * [dbt environment best practices](https://docs.getdbt.com/guides/set-up-ci.md) * [About dbt platform profiles](https://docs.getdbt.com/docs/platform/about-profiles.md) * [Deploy jobs](https://docs.getdbt.com/docs/deploy/deploy-jobs.md) * [CI jobs](https://docs.getdbt.com/docs/deploy/continuous-integration.md) * [Delete a job or environment in dbt](https://docs.getdbt.com/faqs/Environments/delete-environment-job.md) --- ### Derived metrics In MetricFlow, derived metrics are metrics created by defining an expression using other metrics. They enable you to perform calculations with existing metrics. This is helpful for combining metrics and doing math functions on aggregated columns, like creating a profit metric. The parameters, description, and type for derived metrics are: (Applies to dbt v1.12 and later) | Parameter | Description | Required | Type | | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------- | ------ | | `name` | The name of the metric. | Required | String | | `description` | A human-readable summary of the metric. | Optional | String | | `type` | The metric type (`simple`, `cumulative`, `ratio`, `derived`, or `conversion`). | Required | String | | `label` | Display name for downstream tools. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | String | | `expr` | The expression that combines other metrics. Validation warns if it is missing or references undefined metrics. | Required | String | | `input_metrics` | Defines aliases, filters, or offsets for metrics referenced in the expression. Needed only when you customize those attributes. | Optional | List | | `input_metrics.name` | The name of the referenced metric defined elsewhere in the project. | Required when `metric_aliases` provided | String | | `input_metrics.alias` | Alternate name you can reference in `expr`. | Optional | String | | `input_metrics.filter` | Filter to apply to the referenced metric. | Optional | String | | `input_metrics.offset_window` | Offset applied to the referenced metric (for example, `1 week`). Allowed only for derived metrics. | Optional | String | The following displays the complete specification for derived metrics, along with an example. (Applies to dbt v1.12 and later) models/model.yml ```yaml metrics: - name: my_derived_metric description: cool derived metric # Optional label: my derived metric label # Optional type: derived # Required expr: my_simple_metric - my_simple_metric_a_week_ago # Required for derived input_metrics: # Required for derived if using aliases / filters / offset_window for portions of the expression - name: my_simple_metric alias: my_simple_metric_a_week_ago filter: "{{ Dimension('my_primary_entity__my_categorical_dimension_column') }} > 10" offset_window: 1 week # Allowed for derived metrics ``` For advanced data modeling, you can use `fill_nulls_with` and `join_to_timespine` to [set null metric values to zero](https://docs.getdbt.com/docs/build/fill-nulls-advanced.md), ensuring numeric values for every data row. #### Derived metrics example (Applies to dbt v1.12 and later) models/model.yml ```yaml models: - name: fct_orders semantic_model: enabled: true name: order ... rest of config ... metrics: - name: order_gross_profit description: "Gross profit from each order." label: Order gross profit type: derived expr: revenue - cost input_metrics: - name: order_total alias: revenue - name: order_cost alias: cost - name: food_order_gross_profit label: Food order gross profit description: "The gross profit for each food order." type: derived expr: revenue - cost input_metrics: - name: order_total alias: revenue filter: | {{ Dimension('order__is_food_order') }} = True - name: order_cost alias: cost filter: | {{ Dimension('order__is_food_order') }} = True - name: order_total_growth_mom description: "Percentage growth of orders total compared to 1 month ago" label: Order total growth % M/M type: derived expr: (order_total - order_total_prev_month) * 100 / order_total_prev_month input_metrics: - name: order_total - name: order_total alias: order_total_prev_month offset_window: 1 month ``` #### Derived metric offset To perform calculations using a metric's value from a previous time period, you can add an offset parameter to a derived metric. For example, if you want to calculate period-over-period growth or track user retention, you can use this metric offset. **Note:** You must include the [`metric_time` dimension](https://docs.getdbt.com/docs/build/dimensions.md#time) when querying a derived metric with an offset window. The following example displays how you can calculate monthly revenue growth using a 1-month offset window: (Applies to dbt v1.12 and later) models/model.yml ```yaml models: - name: customers semantic_model: enabled: true name: customers_semantic_model metrics: - name: customer_retention description: Percentage of customers that are active now and those active 1 month ago label: customer_retention type: derived expr: current_active_customers / active_customers_prev_month input_metrics: - name: active_customers alias: current_active_customers - name: active_customers alias: active_customers_prev_month offset_window: 1 month ``` ##### Offset windows and granularity You can query any granularity and offset window combination. The following example queries a metric with a 7-day offset and a monthly grain: (Applies to dbt v1.12 and later) models/model.yml ```yaml models: - name: customers semantic_model: enabled: true name: customers_semantic_model ... rest of config ... metrics: - name: d7_booking_change description: Difference between bookings now and 7 days ago type: derived label: d7 bookings change expr: current_bookings - bookings_7_days_ago input_metrics: - name: bookings alias: current_bookings - name: bookings offset_window: 7 days alias: bookings_7_days_ago ``` When you run the query `dbt sl query --metrics d7_booking_change --group-by metric_time__month` for the metric, here's how it's calculated. For dbt v1, you can use the `mf query` prefix. 1. Retrieve the raw, unaggregated dataset with the specified (Applies to dbt v1.12 and later) simple metric and dimensions at the smallest level of detail, which is currently 'day'. 2. Then, perform an offset join on the daily dataset, followed by performing a date trunc and aggregation to the requested granularity. For example, to calculate `d7_booking_change` for July 2017: * First, sum up all the booking values for each day in July to calculate the bookings metric. * The following table displays the range of days that make up this monthly aggregation. | | Orders | Metric\_time | | ----- | ------ | ------------------------ | | | 330 | 2017-07-31 | | | 7030 | 2017-07-30 to 2017-07-02 | | | 78 | 2017-07-01 | | Total | 7438 | 2017-07-01 | 3. Calculate July's bookings with a 7-day offset. The following table displays the range of days that make up this monthly aggregation. Note that the month begins 7 days later (offset by 7 days) on 2017-07-24. | | Orders | Metric\_time | | ----- | ------ | ------------------------ | | | 329 | 2017-07-24 | | | 6840 | 2017-07-23 to 2017-06-30 | | | 83 | 2017-06-24 | | Total | 7252 | 2017-07-01 | 4. Lastly, calculate the derived metric and return the final result set: ```bash bookings - bookings_7_days_ago would be compile as 7438 - 7252 = 186. ``` | d7\_booking\_change | metric\_time\_\_month | | ------------------- | --------------------- | | 186 | 2017-07-01 | #### Related docs * [Fill null values for simple, derived, or ratio metrics](https://docs.getdbt.com/docs/build/fill-nulls-advanced.md) --- ### Dimensions (Applies to dbt v1.12 and later) Dimensions represent the non-aggregatable columns in your data set, which are the attributes, features, or characteristics that describe or categorize data. In the context of the Semantic Layer, dimensions are part of a larger structure called a semantic model. They are created along with other elements like [entities](https://docs.getdbt.com/docs/build/entities.md) and [simple metrics](https://docs.getdbt.com/docs/build/simple.md) and used to add more details to your data. In SQL, dimensions are typically included in the `group by` clause of your SQL query. All dimensions require a `name`, `type`, and can optionally include an `expr` parameter. The `name` for your Dimension must be unique within the same semantic model. (Applies to dbt v1.12 and later) | Parameter | Description | Required | Type | | -------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ---------- | | `name` | The name of the dimension that will be visible to the user in downstream tools. It can also serve as an alias for derived dimensions<br /><br />Dimension names should be unique within a semantic model, but they can be non-unique across different models as MetricFlow uses [joins](https://docs.getdbt.com/docs/build/join-logic.md) to identify the right dimension. | Required | String | | `type` | Specifies the type of group created in the semantic model. There are two types:<br /><br />- **Categorical**: Describe attributes or features like geography or sales region.<br />- **Time**: Time-based dimensions like timestamps or dates. | Required | String | | `description` | A clear description of the dimension. | Optional | String | | `expr` | Defines the underlying column or SQL query for a dimension. If no `expr` is specified, MetricFlow will use the column with the same name as the group. You can use the column name itself to input a SQL expression. | Optional | String | | `label` | Defines the display value in downstream tools. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | String | | [`meta`](https://docs.getdbt.com/reference/resource-configs/meta.md) | Set metadata for a resource and organize resources. Accepts plain text, spaces, and quotes. | Optional | Dictionary | Refer to the following for the complete specification for dimensions: ```yaml models: - name: Model name # Required semantic_model: enabled: true # bool. Required columns: # Any column can have either an entity or a dimension, but not both - name: my_dimension_column # Required description: Column description # Optional dimension: name: my_dimension # Optional, defaults to column name type: categorical # Required. Accepted values: categorical | time label: Recommended adding a string that defines the display value in downstream tools # Optional description: Same as always # Optional, defaults to the column description if not otherwise specified ``` Refer to the following example to see how dimensions are used in a semantic model: (Applies to dbt v1.12 and later) ```yaml models: - name: fact_transactions semantic_model: enabled: true name: transactions agg_time_dimension: order_date columns: # --- entities --- - name: transaction_column entity: type: primary name: transaction # --- dimensions tied 1:1 to columns --- - name: another_transaction_column granularity: day dimension: type: time name: order_date label: "Date of transaction" description: "A record for every transaction that takes place. Carts are considered multiple transactions for each SKU." - name: type dimension: type: categorical name: type ``` #### `derived_semantics` in `dimensions` Use the `derived_semantics` key in the model YAML entry when you need to derive a dimension definition that is not a direct 1:1 mapping to a single physical column. The `expr` field is required when using `derived_semantics`. ```yaml models: - name: my_model semantic_model: enabled: true ... derived_semantics: dimensions: - name: is_bulk type: categorical expr: "case when quantity > 10 then true else false end" # Required ``` (Applies to dbt v1.12 and later) Dimensions are bound to the primary entity of the semantic model they are defined in. MetricFlow requires that all semantic models have a primary entity. This is to guarantee unique dimension names. If your data source doesn't have a primary entity, you need to assign the entity a name using the `entity` key. It doesn't necessarily have to map to a column in that table and assigning the name doesn't affect query generation. We recommend making these "virtual primary entities" unique across your semantic model. See the following example on how to define a primary entity: ```yaml models: - name: bookings_monthly_source semantic_model: enabled: true agg_time_dimension: ds columns: # Primary entity - name: booking_id entity: type: primary name: booking_id - name: order_date granularity: day dimension: type: time label: "Date" metrics: - name: bookings_monthly type: simple agg: sum ``` If your table doesn't have a physical primary key column, you can still declare a primary entity. Set the model’s grain by declaring a `primary_entity`. The new YAML spec supports this and treats the model as being at that entity’s grain. ```yaml models: - name: model_without_pk semantic_model: enabled: true primary_entity: order # "virtual primary entity" columns: - name: customer_id entity: foreign ``` #### Dimensions types This section further explains the dimension definitions, along with examples. Dimensions have the following types: * [`derived_semantics` in `dimensions`](#derived_semantics-in-dimensions) * [Dimensions types](#dimensions-types) * [Categorical](#categorical) * [Time](#time) * [SCD Type II](#scd-type-ii) * [Basic structure](#basic-structure) * [Semantic model parameters and keys](#semantic-model-parameters-and-keys) * [Implementation](#implementation) * [SCD examples](#scd-examples) #### Categorical Categorical dimensions are used to group metrics by different attributes, features, or characteristics such as product type. They can refer to existing columns in your dbt model or be calculated using a SQL expression with the `expr` parameter. An example of a categorical dimension is `is_bulk_transaction`, which is a group created by applying a case statement to the underlying column `quantity`. This allows users to group or filter the data based on bulk transactions. (Applies to dbt v1.9 and later) ```yaml dimensions: - name: is_bulk_transaction type: categorical expr: case when quantity > 10 then true else false end config: meta: usage: "Filter to identify bulk transactions, like where quantity > 10." ``` #### Time (Applies to dbt v1.12 and later) Time dimensions no longer use `type_params`. * For dimensions defined on a column entry, add the column’s `granularity` at the column level. * For derived dimensions, add `granularity` in the dimension configuration. A semantic model’s default aggregation time dimension is set with the `agg_time_dimension` property at the model's top level. A metric can override this with its own `agg_time_dimension`. For more information, see [Migrate to the latest YAML spec](https://docs.getdbt.com/docs/build/latest-metrics-spec.md). You can use multiple time groups in separate metrics. For example, the `users_created` metric uses `created_at`, and the `users_deleted` metric uses `deleted_at`: ```bash # dbt users dbt sl query --metrics users_created,users_deleted --group-by metric_time__year --order-by metric_time__year # dbt v1 users mf query --metrics users_created,users_deleted --group-by metric_time__year --order-by metric_time__year ``` You can set `is_partition` for time to define specific time spans. ##### is\_partition Use `is_partition: True` to show that a dimension exists over a specific time window. For example, a date-partitioned dimensional table. When you query metrics from different tables, the Semantic Layer uses this parameter to ensure that the correct dimensional values are joined to measures. (Applies to dbt v1.12 and later) ```yaml models: - name: orders semantic_model: enabled: true agg_time_dimension: created_at columns: - name: ts_created granularity: day dimension: type: time name: created_at label: "Date of creation" config: meta: notes: "Only valid for orders from 2022 onward" is_partition: true - name: ts_deleted granularity: day dimension: type: time name: deleted_at label: "Date of deletion" is_partition: true metrics: - name: users_deleted type: simple agg: sum expr: 1 agg_time_dimension: deleted_at - name: users_created type: simple agg: sum expr: 1 ``` ##### time\_granularity (Applies to dbt v1.12 and later) `granularity` specifies the grain of a time dimension. MetricFlow will transform the underlying column to the specified granularity. For example, if you add hourly granularity to a time dimension column, MetricFlow will run a `date_trunc` function to convert the timestamp to hourly. You can easily change the time grain at query time and aggregate it to a coarser grain, for example, from hourly to monthly. However, you can't go from a coarser grain to a finer grain (monthly to hourly). Our supported granularities are: * nanosecond (Snowflake only) * microsecond * millisecond * second * minute * hour * day * week * month * quarter * year Aggregation between metrics with different granularities is possible, with the Semantic Layer returning results at the coarsest granularity by default. For example, when querying two metrics with daily and monthly granularity, the resulting aggregation will be at the monthly level. ```yaml models: - name: your_model_name semantic_model: enabled: true agg_time_dimension: created_at columns: - name: ts_created granularity: hour dimension: type: time name: created_at label: "Date of creation" is_partition: true - name: ts_deleted granularity: day dimension: type: time name: deleted_at label: "Date of deletion" is_partition: true metrics: - name: users_deleted type: simple agg: sum expr: 1 agg_time_dimension: deleted_at - name: users_created type: simple agg: sum expr: 1 ``` ##### SCD Type II (Applies to dbt v1.12 and later) caution Currently, semantic models with SCD Type II dimensions cannot contain simple metrics. MetricFlow supports joins against dimensions values in a semantic model built on top of a slowly changing dimension (SCD) Type II table. This is useful when you need a particular metric sliced by a group that changes over time, such as the historical trends of sales by a customer's country. ###### Basic structure SCD Type II are groups that change values at a coarser time granularity. SCD Type II tables typically have two time columns that indicate the validity period of a dimension: `valid_from` (or `tier_start`) and `valid_to` (or `tier_end`). This creates a range of valid rows with different dimension values for a metric. MetricFlow associates the metric with the earliest available dimension value within a coarser time window, such as a month. By default, it uses the group valid at the start of this time granularity. MetricFlow supports the following basic structure of an SCD Type II data platform table: | entity\_key | dimensions\_1 | dimensions\_2 | ... | dimensions\_x | valid\_from | valid\_to | | ----------- | ------------- | ------------- | --- | ------------- | ----------- | ---------- | | 123 | value\_a | value\_x | ... | value\_n | 2024-01-01 | 2024-06-30 | | 123 | value\_b | value\_y | ... | value\_m | 2024-07-01 | 2024-12-31 | * `entity_key` (required): A unique identifier for each row in the table, such as a primary key or another unique identifier specific to the entity. * `valid_from` (required): Start date timestamp for when the dimension is valid. Use `validity_params: is_start: True` in the semantic model to specify this. * `valid_to` (required): End date timestamp for when the dimension is valid. Use `validity_params: is_end: True` in the semantic model to specify this. ###### Semantic model parameters and keys When configuring an SCD Type II table in a semantic model, use `validity_params` to specify the start (`valid_from`) and end (`valid_to`) of the validity window for each dimension. * `validity_params`: Parameters that define the validity window. * `is_start: True`: Indicates the start of the validity period. Displayed as `valid_from` in the SCD table. * `is_end: True`: Indicates the end of the validity period. Displayed as `valid_to` in the SCD table. Here’s an example configuration: (Applies to dbt v1.12 and later) ```yaml models: - name: tiers semantic_model: enabled: true agg_time_dimension: tier_start columns: - name: start_date granularity: day dimension: type: time # The type of dimension name: tier_start # The name of the dimension. label: "Start date of tier" # A readable label for the dimension validity_params: # Defines the validity window is_start: true # Indicates the start of the validity period - name: end_date granularity: day dimension: type: time name: tier_end label: "End date of tier" validity_params: is_end: true # Indicates the end of the validity period ``` SCD Type II tables have a specific dimension with a start and end date. To join tables: * Set the additional [entity `type`](https://docs.getdbt.com/docs/build/entities.md#entity-types) parameter to the `natural` key. * Use a `natural` key as an [entity `type`](https://docs.getdbt.com/docs/build/entities.md#entity-types), which means you don't need a `primary` key. * In most instances, SCD tables don't have a logically usable `primary` key because `natural` keys map to multiple rows. ###### Implementation Here are some guidelines to follow when implementing SCD Type II tables: * The SCD table must have `valid_to` and `valid_from` time dimensions, which are logical constructs. * The `valid_from` and `valid_to` properties must be specified exactly once per SCD table configuration. * The `valid_from` and `valid_to` properties shouldn't be used or specified on the same time dimension. * The `valid_from` and `valid_to` time dimensions must cover a non-overlapping period where one row matches each natural key value (meaning they must not overlap and should be distinct). * We recommend defining the underlying dbt model with [dbt snapshots](https://docs.getdbt.com/docs/build/snapshots.md). This supports the SCD Type II table layout and ensures that the table is updated with the latest data. This is an example of SQL code that shows how a sample metric called `num_events` is joined with versioned dimensions data (stored in a table called `scd_dimensions`) using a primary key made up of the `entity_key` and `timestamp` columns. ```sql select metric_time, dimensions_1, sum(1) as num_events from events a left outer join scd_dimensions b on a.entity_key = b.entity_key and a.metric_time >= b.valid_from and (a.metric_time < b. valid_to or b.valid_to is null) group by 1, 2 ``` ###### SCD examples The following are examples of how to use SCD Type II tables in a semantic model:  SCD dimensions for sales tiers and the time length of that tier. This example shows how to create slowly changing dimensions (SCD) using a semantic model. The SCD table contains information about salespersons' tier and the time length of that tier. Suppose you have the underlying SCD table: | sales\_person\_id | tier | start\_date | end\_date | | ----------------- | ---- | ----------- | ---------- | | 111 | 1 | 2019-02-03 | 2020-01-05 | | 111 | 2 | 2020-01-05 | 2048-01-01 | | 222 | 2 | 2020-03-05 | 2048-01-01 | | 333 | 2 | 2020-08-19 | 2021-10-22 | | 333 | 3 | 2021-10-22 | 2048-01-01 | As mentioned earlier, the `validity_params` include two important arguments that specify the columns in the SCD table that mark the start and end dates (or timestamps) for each tier or dimension: * `is_start` * `is_end` Additionally, the entity is tagged as `natural` to differentiate it from a `primary` entity. In a `primary` entity, each entity value has one row. In contrast, a `natural` entity has one row for each combination of entity value and its validity period. (Applies to dbt v1.12 and later) ```yaml models: - name: sales_person_tiers semantic_model: enabled: true agg_time_dimension: tier_start # You can use a virtual primary entity name primary_entity: sales_person columns: - name: start_date granularity: day dimension: type: time name: tier_start label: "Start date of tier" validity_params: is_start: true - name: end_date granularity: day dimension: type: time name: tier_end label: "End date of tier" validity_params: is_end: true - name: tier dimension: type: categorical name: tier - name: sales_person_id entity: type: natural name: sales_person ``` The following code represents a separate semantic model that holds a fact table for `transactions`: (Applies to dbt v1.12 and later) ```yaml models: - name: fact_transactions semantic_model: enabled: true agg_time_dimension: metric_time columns: - name: transaction_id entity: type: primary name: transaction_id - name: customer_id entity: type: foreign name: customer - name: product_id entity: type: foreign name: product - name: sales_person_id entity: type: foreign name: sales_person - name: metric_time granularity: day dimension: type: time name: metric_time label: "Date of transaction" is_partition: true - name: sales_geo dimension: type: categorical name: sales_geo metrics: - name: transactions type: simple agg: sum expr: 1 - name: gross_sales type: simple agg: sum expr: sales_price - name: sales_persons_with_a_sale type: simple agg: count_distinct expr: sales_person_id ``` You can now access the metrics in the `transactions` semantic model organized by the slowly changing dimension of `tier`. In the sales tier example, For instance, if a salesperson was Tier 1 from 2022-03-01 to 2022-03-12, and gets promoted to Tier 2 from 2022-03-12 onwards, all transactions from March would be categorized under Tier 1 since the dimensions value of Tier 1 comes earlier (and is the default starting point), even though the salesperson was promoted to Tier 2 on 2022-03-12.  SCD dimensions with sales tiers and group transactions by month when tiers are missing This example shows how to create slowly changing dimensions (SCD) using a semantic model. The SCD table contains information about salespersons' tier and the time length of that tier. Suppose you have the underlying SCD table: | sales\_person\_id | tier | start\_date | end\_date | | ----------------- | ---- | ----------- | ---------- | | 111 | 1 | 2019-02-03 | 2020-01-05 | | 111 | 2 | 2020-01-05 | 2048-01-01 | | 222 | 2 | 2020-03-05 | 2048-01-01 | | 333 | 2 | 2020-08-19 | 2021-10-22 | | 333 | 3 | 2021-10-22 | 2048-01-01 | In the sales tier example, if sales\_person\_id 456 is Tier 2 from 2022-03-08 onwards, but there is no associated tier level dimension for this person from 2022-03-01 to 2022-03-08, then all transactions associated with sales\_person\_id 456 for the month of March will be grouped under 'NA' since no tier is present prior to Tier 2. The following command or code represents how to return the count of transactions generated by each sales tier per month: ```bash # dbt platform users dbt sl query --metrics transactions --group-by metric_time__month,sales_person__tier --order-by metric_time__month,sales_person__tier # dbt v1 users mf query --metrics transactions --group-by metric_time__month,sales_person__tier --order-by metric_time__month,sales_person__tier ``` --- ### Discover data with Catalog dbt platform | Starter, Enterprise, Enterprise+ With Catalog, you can view your project's [resources](https://docs.getdbt.com/docs/build/projects.md) (such as models, tests, and metrics), their lineage, and [model consumption](https://docs.getdbt.com/docs/explore/view-downstream-exposures.md) to gain a better understanding of its latest production state. Use Catalog to navigate and manage your projects within dbt to help you and other data developers, analysts, and consumers discover and leverage your dbt resources. Catalog integrates with the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md), [dbt Insights](https://docs.getdbt.com/docs/explore/dbt-insights.md), [Orchestrator](https://docs.getdbt.com/docs/deploy/deployments.md), and [Canvas](https://docs.getdbt.com/docs/platform/canvas.md) to help you develop or view your dbt resources. #### Prerequisites * You have a dbt account on the [Starter, Enterprise, or Enterprise+ plan](https://www.getdbt.com/pricing/). * You have set up a [production](https://docs.getdbt.com/docs/deploy/deploy-environments.md#set-as-production-environment) or [staging](https://docs.getdbt.com/docs/deploy/deploy-environments.md#create-a-staging-environment) deployment environment for each project you want to explore. * You have at least one successful job run in the deployment environment. Note that [CI jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md) do not update Catalog. * You are on the Catalog page. To do this, select **Catalog** from the top-level navigation in dbt. #### Generate metadata Catalog uses the metadata provided by the [Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md) to display the details about [the state of your dbt project](https://docs.getdbt.com/docs/dbt-apis/project-state.md). The metadata that's available depends on the [deployment environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md) you've designated as *production* or *staging* in your dbt project. #### dbt metadata If you're using a [hybrid project setup](https://docs.getdbt.com/docs/deploy/hybrid-setup.md) and uploading artifacts from dbt v1, make sure to follow the [setup instructions](https://docs.getdbt.com/docs/deploy/hybrid-setup.md#connect-project-in-dbt-cloud) to connect your project in dbt. This enables Catalog to access and display your metadata correctly. * To ensure all metadata is available in Catalog, run `dbt build` and `dbt docs generate` as part of your job in your production or staging environment. Running these two commands ensures all relevant metadata (like lineage, test results, documentation, and more) is available in Catalog. * Catalog automatically retrieves the metadata updates after each job run in the production or staging deployment environment so it always has the latest results for your project. This includes deploy and merge jobs. * Note that CI jobs don't update Catalog. This is because they don't reflect the production state and don't provide the necessary metadata updates. * To view a resource and its metadata, you must define the resource in your project and run a job in the production or staging environment. * The resulting metadata depends on the [commands](https://docs.getdbt.com/docs/deploy/job-commands.md) executed by the jobs. ##### When dbt creates model metadata dbt populates a model's metadata in Catalog when both of the following conditions are met: * The model is defined in your dbt project (it exists in the manifest). * The model appears in the `run_results` of a [`dbt build`](https://docs.getdbt.com/reference/commands/build.md), [`dbt run`](https://docs.getdbt.com/reference/commands/run.md), or [`dbt clone`](https://docs.getdbt.com/reference/commands/clone.md) command, regardless of the run's success or failure status. Note that `dbt docs generate` alone does not create model entries in Catalog. It provides supplementary metadata like column details and descriptions for models that already exist. ##### When dbt removes model metadata dbt removes a model's metadata from Catalog in these two cases: * **Model removed from project**: If a model is deleted from your dbt project (and therefore no longer exists in the manifest), its metadata is removed after a subsequent job run in which the model is no longer included. * **Environment inactivity**: If an environment has had no job runs in the past 3 months, all metadata for that environment is purged. To prevent this, schedule jobs to run at least once every 3 months. | To view in Catalog | You must successfully run | | --------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | All metadata | [dbt build](https://docs.getdbt.com/reference/commands/build.md), [dbt docs generate](https://docs.getdbt.com/reference/commands/cmd-docs.md), and [dbt source freshness](https://docs.getdbt.com/reference/commands/source.md#dbt-source-freshness) together as part of the same job in the environment | | Model lineage, details, or results | [dbt run](https://docs.getdbt.com/reference/commands/run.md) or [dbt build](https://docs.getdbt.com/reference/commands/build.md) on a given model within a job in the environment | | Columns and statistics for models, sources, and snapshots | [dbt docs generate](https://docs.getdbt.com/reference/commands/cmd-docs.md) within [a job](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) in the environment | | Data test results | [dbt test](https://docs.getdbt.com/reference/commands/test.md) or [dbt build](https://docs.getdbt.com/reference/commands/build.md) within a job in the environment | | Unit test results | [dbt test](https://docs.getdbt.com/reference/commands/test.md) or [dbt build](https://docs.getdbt.com/reference/commands/build.md) within a job in the environment. Unit tests are typically run in development or CI environments, so their results rarely appear in production Catalog. | | Source freshness results | [dbt source freshness](https://docs.getdbt.com/reference/commands/source.md#dbt-source-freshness) within a job in the environment | | Snapshot details | [dbt snapshot](https://docs.getdbt.com/reference/commands/snapshot.md) or [dbt build](https://docs.getdbt.com/reference/commands/build.md) within a job in the environment | | Seed details | [dbt seed](https://docs.getdbt.com/reference/commands/seed.md) or [dbt build](https://docs.getdbt.com/reference/commands/build.md) within a job in the environment | tip If your organization works in both dbt v1 and Cloud, you can unify these workflows by automatically uploading dbt v1 artifacts into dbt Cloud and viewing them in Catalog for a more connected dbt experience. To learn more, visit [hybrid projects](https://docs.getdbt.com/docs/deploy/hybrid-projects.md). #### Catalog overview [Global navigation](https://docs.getdbt.com/docs/explore/global-navigation.md) [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") Catalog introduces the ability to widen your search by including dbt resources (models, seeds, snapshots, sources, exposures, and more) across your entire account. This broadens the results returned and gives you greater insight into all the assets across your dbt projects. Learn more in [Global navigation](https://docs.getdbt.com/docs/explore/global-navigation.md) or in our [video overview](https://www.loom.com/share/ae93b3d241cd439fbe5f98f5e6872113?). Navigate the Catalog overview page to access your project's resources and metadata. The page includes the following sections: * **Search bar** — [Search](#search-resources) for resources in your project by keyword. You can also use filters to refine your search results. * **Sidebar** — Use the left sidebar to browse your project's [resources, file tree, and database](#browse-with-the-sidebar) in the lower section of the sidebar. You can also browse model [performance](https://docs.getdbt.com/docs/explore/model-performance.md) and [project recommendations](https://docs.getdbt.com/docs/explore/project-recommendations.md) depending on your plan. Refer to [Availability by plan](#availability-by-plan) for more info. * Find your project recommendations within your project's landing page.\* * **Lineage graph** — Explore your project's or account's [lineage graph](#project-lineage) to visualize the relationships between resources. * **Latest updates** — View the latest changes or issues related to your project's resources, including the most recent job runs, changed properties, lineage, and issues. * **Marts and public models** — View the [marts](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md#guide-structure-overview) and [public models](https://docs.getdbt.com/docs/mesh/govern/model-access.md#access-modifiers) in your project. You can also navigate to all public models in your account through this view. * **Model query history** — Use [model query history](https://docs.getdbt.com/docs/explore/model-query-history.md) to track consumption queries on your models for deeper insights. * **Visualize downstream exposures** — [Set up](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md) and [visualize downstream exposures](https://docs.getdbt.com/docs/explore/view-downstream-exposures.md) to automatically expose relevant data models from Tableau to enhance visibility. * **Data health signals** — View the [data-health-signals](https://docs.getdbt.com/docs/explore/data-health-signals.md) for each resource to understand its health and performance. ##### Catalog permissions When using global navigation and searching across your projects, the following permissions apply. * Your project access permissions determine which dbt projects appear in the left-hand menu of the global navigation. * In Catalog searches, we use soft access controls, you'll see all matching resources in search results, with clear indicators for items you don't have access to. ##### Availability by plan Catalog is available on all Starter, Enterprise, and Enterprise+ plans. However, certain features are only available on Enterprise and Enterprise+ plans: | Feature | Starter | Enterprise | Enterprise+ | | ------------------------------------------------------------------------------------------ | ------- | ---------- | ----------- | | dbt lineage & resource browsing | ✅ | ✅ | ✅ | | [Global navigation](https://docs.getdbt.com/docs/explore/global-navigation.md) | ✅ | ✅ | ✅ | | [Data health signals](https://docs.getdbt.com/docs/explore/data-health-signals.md) | ✅ | ✅ | ✅ | | [Model performance](https://docs.getdbt.com/docs/explore/model-performance.md) | ❌ | ✅ | ✅ | | [Project recommendations](https://docs.getdbt.com/docs/explore/project-recommendations.md) | ❌ | ✅ | ✅ | | [Column-level lineage](https://docs.getdbt.com/docs/explore/column-level-lineage.md) | ❌ | ✅ | ✅ | | [Multi-project lineage](https://docs.getdbt.com/docs/explore/explore-multiple-projects.md) | ❌ | ✅ | ✅ | | [Model query history](https://docs.getdbt.com/docs/explore/model-query-history.md) | ❌ | ✅ | ✅ | | [Downstream exposures](https://docs.getdbt.com/docs/explore/view-downstream-exposures.md) | ❌ | ✅ | ✅ | | [Data health tile](https://docs.getdbt.com/docs/explore/data-tile.md) | ❌ | ✅ | ✅ | On-demand learning If you enjoy video courses, check out our [dbt Catalog on-demand course](https://learn.getdbt.com/courses/dbt-catalog) and learn how to best explore your dbt project(s)! #### Explore your project's lineage graph Catalog provides a visualization of your project's DAG that you can interact with. To access the project's full lineage graph, select **Overview** in the left sidebar and click the **Explore Lineage** button on the main (center) section of the page. If you don't see the project lineage graph immediately, click **Render Lineage**. It can take some time for the graph to render depending on the size of your project and your computer's available memory. The graph of very large projects might not render so you can select a subset of nodes by using selectors, instead. The nodes in the lineage graph represent the project's resources and the edges represent the relationships between the nodes. Nodes are color-coded and include iconography according to their resource type. By default, Catalog shows the project's [applied state](https://docs.getdbt.com/docs/dbt-apis/project-state.md#definition-logical-vs-applied-state-of-dbt-nodes) lineage. That is, it shows models that have been successfully built and are available to query, not just the models defined in the project. To explore the lineage graphs of tests and macros, view [their resource details pages](#view-resource-details). By default, Catalog excludes these resources from the full lineage graph unless a search query returns them as results.  How can I interact with the full lineage graph? * Hover over any item in the graph to display the resource's name and type. * Zoom in and out on the graph by mouse-scrolling. * Grab and move the graph and the nodes. * Right-click on a node (context menu) to: * Refocus on the node, including its upstream and downstream nodes * Refocus on the node and its downstream nodes only * Refocus on the node and it upstream nodes only * View the node's [resource details](#view-resource-details) page * Select a resource to highlight its relationship with other resources in your project. A panel opens on the graph's right-hand side that displays a high-level summary of the resource's details. The side panel includes a **General** tab for information like description, materialized type, and other details. In the side panel's upper right corner: * Click the View Resource icon to [view the resource details](#view-resource-details). * Click the [Open in IDE](#open-in-ide) icon to examine the resource using the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md). * Click the Copy Link to Page icon to copy the page's link to your clipboard. * Use [selectors](https://docs.getdbt.com/reference/node-selection/methods.md) (in the search bar) to select specific resources or a subset of the DAG. This can help narrow the focus on the resources that interest you. All selectors are available for use, except those requiring a state comparison (result, source status, and state). You can also use the `--exclude` and the `--select` flag (which is optional). Examples: * `resource_type:model [RESOURCE_NAME]` — Returns all models matching the name search * `resource_type:metric,tag:nightly` — Returns metrics with the tag `nightly` * Use [graph operators](https://docs.getdbt.com/reference/node-selection/graph-operators.md) (in the search bar) to select specific resources or a subset of the DAG. This can help narrow the focus on the resources that interest you. Examples: * `+orders` — Returns all the upstream nodes of `orders` * `+dim_customers,resource_type:source` — Returns all sources that are upstream of `dim_customers` * Use [set operators](https://docs.getdbt.com/reference/node-selection/set-operators.md) (in the search bar) to select specific resources or a subset of the DAG. This can help narrow the focus on the resources that interest you. For example: * `+snowplow_sessions +fct_orders` — Use space-delineated arguments for a union operation. Returns resources that are upstream nodes of either `snowplow_sessions` or `fct_orders`. * [View resource details](#view-resource-details) by selecting a node (double-clicking) in the graph. * Click **Lenses** (lower right corner of the graph) to use Catalog [lenses](#lenses) feature. ##### Example of full lineage graph To explore a model in your project's lineage graph in Catalog: 1. Select your project in the left sidebar. 2. Click the **View lineage** button. 3. Click a model to view the description. [![Example of full lineage graph](/img/docs/collaborate/dbt-explorer/example-project-lineage-graph.png?v=2 "Example of full lineage graph")](#)Example of full lineage graph #### Lenses The **Lenses** feature is available from your [project's lineage graph](#project-lineage) (lower right corner). Lenses are like map layers for your DAG. Lenses make it easier to understand your project's contextual metadata at scale, especially to distinguish a particular model or a subset of models. When you apply a lens, tags become visible on the nodes in the lineage graph, indicating the layer value along with coloration based on that value. If you're significantly zoomed out, only the tags and their colors are visible in the graph. Lenses are helpful to analyze a subset of the DAG if you're zoomed in, or to find models/issues from a larger vantage point.  List of available lenses A resource in your project is characterized by resource type, materialization type, or model layer, as well as its latest run or latest test status. Lenses are available for the following metadata: * **Resource type**: Organizes resources by resource type, such as models, tests, seeds, saved query, and [more](https://docs.getdbt.com/docs/build/projects.md). Resource type uses the `resource_type` selector. * **Materialization type**: Identifies the strategy for building the dbt models in your data platform. * **Latest status**: The status from the latest execution of the resource in the current environment. For example, diagnosing a failed DAG region. * **Model layer**: The modeling layer that the model belongs to according to [best practices guide](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md#guide-structure-overview). For example, discovering marts models to analyze. * **Marts** — A model with the prefix `fct_` or `dim_` or a model that lives in the `/marts/` subdirectory. * **Intermediate** — A model with the prefix `int_`. Or, a model that lives in the `/int/` or `/intermediate/` subdirectory. * **Staging** — A model with the prefix `stg_`. Or, a model that lives in the `/staging/` subdirectory. * **Test status**: The status from the latest execution of the tests that ran again this resource. In the case that a model has multiple tests with different results, the lens reflects the 'worst case' status. * **Consumption query history**: The number of queries against this resource over a given time period. ##### Example of lenses Example of applying the **Materialization type** *lens* on a zoomed-out lineage graph. In the bottom-left corner, open **Lenses** and select **Materialization type**. Each node displays a colored badge for its related materialization type: * Table * View * Incremental * Ephemeral * Materialized View. [![Example of the Materialization type lens](/img/docs/collaborate/dbt-explorer/example-materialization-type.png?v=2 "Example of the Materialization type lens")](#)Example of the Materialization type lens Example of applying the **Test status** *lens* on a lineage graph filtered to models. In the bottom-left corner, open **Lenses** and select **Test status**. Each node displays a colored badge for its latest test status: * Pass * Error * Fail * Warn * Skipped * Reused. [![Example of the Test Status lens](/img/docs/collaborate/dbt-explorer/example-test-status.png?v=2 "Example of the Test Status lens")](#)Example of the Test Status lens #### Keyword search With Catalog, global navigation provides a search experience allowing you to find dbt resources across all your projects, as well as non-dbt resources in Snowflake. You can locate resources in your project by performing a keyword search in the search bar. All resource names, column names, resource descriptions, warehouse relations, and code matching your search criteria will be displayed as a list on the main (center) section of the page. When searching for an exact column name, the results show all relational nodes containing that column in their schemas. If there's a match, a notice in the search result indicates the resource contains the specified column. Also, you can apply filters to further refine your search results.  Search features * **Partial keyword search** — Also referred to as fuzzy search. Catalog uses a "contains" logic to improve your search results. This means you can search for partial terms without knowing the exact root word of your search term. * **Exclude keywords** — Prepend a minus sign (-) to the keyword you want to exclude from search results. For example, `-user` will exclude all matches of that keyword from search results. * **Boolean operators** — Use Boolean operators to enhance your keyword search. For example, the search results for `users OR github` will include matches for either keyword. * **Phrase search** — Surround a string of keywords with double quotation marks to search for that exact phrase (for example, `"stg users"`). To learn more, refer to [Phrase search](https://en.wikipedia.org/wiki/Phrase_search) on Wikipedia. * **SQL keyword search** — Use SQL keywords in your search. For example, the search results `int github users joined` will include matches that contain that specific string of keywords (similar to phrase searching).  Filters side panel The **Filters** side panel becomes available after you perform a keyword search. Use this panel to further refine the results from your keyword search. By default, Catalog searches across all resources in the project. You can filter on: * [Resource type](https://docs.getdbt.com/docs/build/projects.md) (like models, sources, and so on) * [Model access](https://docs.getdbt.com/docs/mesh/govern/model-access.md) (like public, private) * [Model layer](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md) (like marts, staging) * [Model materialization](https://docs.getdbt.com/docs/build/materializations.md) (like view, table) * [Tags](https://docs.getdbt.com/reference/resource-configs/tags.md) (supports multi-select) Under the **Models** option, you can filter on model properties (access or materialization type). Also available are **Advanced** options, where you can limit the search results to column name, model code, and more.  Global navigation Catalog builds on the functionality of the old navigation and introduces exciting new capabilities to enhance your experience. For more information, refer to [Global navigation](https://docs.getdbt.com/docs/explore/global-navigation.md). ##### Example of keyword search Example of results from searching on the keyword `customers` and applying the filters models, description, and code. [Data health signals](https://docs.getdbt.com/docs/explore/data-health-signals.md) are visible to the right of the model name in the search results. #### Browse with the sidebar From the sidebar, you can browse your project's resources, its file tree, and the database. * **Resources** tab — All resources in the project organized by type. Select any resource type in the list and all those resources in the project will display as a table in the main section of the page. For a description on the different resource types (like models, metrics, and so on), refer to [About dbt projects](https://docs.getdbt.com/docs/build/projects.md). * [Data health signals](https://docs.getdbt.com/docs/explore/data-health-signals.md) are visible to the right of the resource name under the **Health** column. * **File Tree** tab — All resources in the project organized by the file in which they are defined. This mirrors the file tree in your dbt project repository. * **Database** tab — All resources in the project organized by the database and schema in which they are built. This mirrors your data platform's structure that represents the [applied state](https://docs.getdbt.com/docs/dbt-apis/project-state.md) of your project. #### Integrated tool access Users with a [developer license](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#license-based-access-control) or an analyst seat\* can open a resource directly from the Catalog in the Studio IDE to view its model files, in Insights to query it, or in Canvas for visual editing. \* The [Analyst license type](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md?version=1.12#licenses) is not available for new purchase. #### View model versions If models in the project are versioned, you can see which [version of the model](https://docs.getdbt.com/docs/mesh/govern/model-versions.md) is being applied — `prerelease`, `latest`, and `old` — in the title of the model's details page and in the model list from the sidebar. #### View resource details You can view the definition and latest run results of any resource in your project. To find a resource and view its details, you can interact with the lineage graph, use search, or browse the Catalog. The details (metadata) available to you depends on the resource's type, its definition, and the [commands](https://docs.getdbt.com/docs/deploy/job-commands.md) that run within jobs in the production environment. In the upper right corner of the resource details page, you can: * Click the [Open in Studio IDE](#open-in-ide) icon to examine the resource using the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md). * Click the Share icon to copy the page's link to your clipboard.  What details are available for a model? * **Data health signals** — [Data health signals](https://docs.getdbt.com/docs/explore/data-health-signals.md) offer a quick, at-a-glance view of data health. These icons indicate whether a model is Healthy, Caution, Degraded, or Unknown. Hover over an icon to view detailed information about the model's health. * **Status bar** (below the page title) — Information on the last time the model ran, whether the run was successful, how the data is materialized, number of rows, and the size of the model. * **General** tab includes: * **Lineage** graph — The model's lineage graph that you can interact with. The graph includes one upstream node and one downstream node from the model. Click the Expand icon in the graph's upper right corner to view the model in full lineage graph mode. * **Description** section — A [description of the model](https://docs.getdbt.com/docs/build/documentation.md#adding-descriptions-to-your-project). * **Recent** section — Information on the last time the model ran, how long it ran for, whether the run was successful, the job ID, and the run ID. * **Tests** section — [Data tests](https://docs.getdbt.com/docs/build/data-tests.md) for the model, including a status indicator for the latest test status. A ✅ denotes a passing test. * **Details** section — Key properties like the model's relation name (for example, how it's represented and how you can query it in the data platform: `database.schema.identifier`); model governance attributes like access, group, and if contracted; and more. * **Relationships** section — The nodes the model **Depends On**, is **Referenced by**, and (if applicable) is **Used by** for projects that have declared the models' project as a dependency. * **Code** tab — The source code and compiled code for the model. * **Columns** tab — The available columns in the model, including [tags](https://docs.getdbt.com/reference/resource-configs/tags.md) defined on columns in your dbt YAML. Column tags appear as clickable badges that filter the lineage view by that tag. This tab also shows test results (if any) that you can select to view the test's details page. A ✅ denotes a passing test. To filter the columns in the resource (including by tag name), use the search bar at the top of the columns view.  What details are available for an exposure? * **Status bar** (below the page title) — Information on the last time the exposure was updated. * **Data health signals** — [Data health signals](https://docs.getdbt.com/docs/explore/data-health-signals.md) offer a quick, at-a-glance view of data health. These icons indicate whether a resource is Healthy, Caution, or Degraded. Hover over an icon to view detailed information about the exposure's health. * **General** tab includes: * **Data health** — The status on data freshness and data quality. * **Status** section — The status on data freshness and data quality. * **Lineage** graph — The exposure's lineage graph. Click the **Expand** icon in the graph's upper right corner to view the exposure in full lineage graph mode. Integrates natively with Tableau and auto-generates downstream lineage. * **Description** section — A description of the exposure. * **Details** section — Details like exposure type, maturity, owner information, and more. * **Relationships** section — The nodes the exposure **Depends On**.  What details are available for a test? * **Status bar** (below the page title) — Information on the last time the test ran, whether the test passed, test name, test target, and column name. Defaults to all if not specified. * **Test Type** (next to the Status bar) — Information on the different test types available: Unit test or Data test. Defaults to all if not specified. When you select a test, the following details are available: * **General** tab includes: * **Lineage** graph — The test's lineage graph that you can interact with. The graph includes one upstream node and one downstream node from the test resource. Click the Expand icon in the graph's upper right corner to view the test in full lineage graph mode. * **Description** section — A description of the test. * **Recent** section — Information on the last time the test ran, how long it ran for, whether the test passed, the job ID, and the run ID. * **Details** section — Details like schema, severity, package, and more. * **Relationships** section — The nodes the test **Depends On**. * **Code** tab — The source code and compiled code for the test. Example of the Tests view:  What details are available for each source table within a source collection? * **Status bar** (below the page title) — Information on the last time the source was updated and the number of tables the source uses. * **Data health signals** — [Data health signals](https://docs.getdbt.com/docs/explore/data-health-signals.md) offer a quick, at-a-glance view of data health. These icons indicate whether a resource is Healthy, Caution, or Degraded. Hover over an icon to view detailed information about the source's health. * **General** tab includes: * **Lineage** graph — The source's lineage graph that you can interact with. The graph includes one upstream node and one downstream node from the source. Click the Expand icon in the graph's upper right corner to view the source in full lineage graph mode. * **Description** section — A description of the source. * **Source freshness** section — Information on whether refreshing the data was successful, the last time the source was loaded, the timestamp of when a run generated data, and the run ID. * **Details** section — Details like database, schema, and more. * **Relationships** section — A table that lists all the sources used with their freshness status, the timestamp of when freshness was last checked, and the timestamp of when the source was last loaded. * **Columns** tab — The available columns in the source, including [tags](https://docs.getdbt.com/reference/resource-configs/tags.md) defined on columns in your dbt YAML. Column tags appear as clickable badges that filter the lineage view by that tag. This tab also shows test results (if any) that you can select to view the test's details page. A ✅ denotes a passing test. ##### Example of model details To view a model's details page in Catalog: 1. Select your project in the sidebar. 2. Click **Models**. 3. Under **Models**, select a model (for example, `customers`). On the **General** tab, you can review the description, latest status, and lineage graph. Select **Full Lineage** to open the full graph view. [![Example of resource details](/img/docs/collaborate/dbt-explorer/example-model-details.png?v=2 "Example of resource details")](#)Example of resource details [![Example of downstream exposure details for Tableau.](/img/docs/platform-integrations/auto-exposures/explorer-lineage2.png?v=2 "Example of downstream exposure details for Tableau.")](#)Example of downstream exposure details for Tableau. #### Staging environment Catalog supports views for [staging deployment environments](https://docs.getdbt.com/docs/deploy/deploy-environments.md#staging-environment), in addition to the production environment. This gives you a unique view into your pre-production data workflows, with the same tools available in production, while providing an extra layer of scrutiny. You can explore the metadata from your production or staging environment to inform your data development lifecycle. Just [set a single environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md) per dbt project as "production" or "staging," and ensure the proper metadata has been generated then you'll be able to view it in Catalog. Refer to [Generating metadata](https://docs.getdbt.com/docs/explore/explore-projects.md#generate-metadata) for more details. #### Related content * [Enterprise permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) * [About model governance](https://docs.getdbt.com/docs/mesh/govern/about-model-governance.md) * Blog on [What is data mesh?](https://www.getdbt.com/blog/what-is-data-mesh-the-definition-and-importance-of-data-mesh) --- ### DuckDB and Apache Iceberg Beta ### DuckDB and Apache Iceberg [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt v2 only DuckDB support for `catalogs.yml` requires [dbt v2](https://docs.getdbt.com/docs/introduction.md) with the `use_catalogs_v2` behavior flag enabled. It isn't available in the legacy Python `dbt-duckdb` adapter for dbt v1. dbt\_project.yml ```yaml flags: use_catalogs_v2: true ``` Unlike Snowflake, Databricks, and BigQuery, DuckDB doesn't ship with a single built-in "managed" Iceberg catalog. This means there's no `table_format='iceberg'`-only shortcut for DuckDB — every Iceberg model requires a `catalog_name` that points to an entry in [`catalogs.yml`](https://docs.getdbt.com/docs/build/iceberg/catalogs-yml.md). dbt supports creating Iceberg tables for two DuckDB materializations: * [Table](https://docs.getdbt.com/docs/build/materializations.md#table) * [Incremental](https://docs.getdbt.com/docs/build/materializations.md#incremental) #### How DuckDB attaches catalogs When you configure a catalog with a `duckdb` block in `catalogs.yml`, dbt generates and runs the appropriate DuckDB [`ATTACH`](https://duckdb.org/docs/sql/statements/attach.html) statement on your behalf — you don't need to write `ATTACH` SQL yourself. dbt then resolves any model with a matching `catalog_name` to that attached database. #### Iceberg REST catalogs DuckDB can attach to any catalog that implements the Iceberg REST protocol, including self-hosted catalogs (such as [Lakekeeper](https://github.com/lakekeeper/lakekeeper) or [Nessie](https://github.com/projectnessie/nessie)), AWS Glue, and AWS S3 Tables. catalogs.yml ```yaml catalogs: - name: rest_catalog type: iceberg_rest table_format: iceberg config: duckdb: endpoint: "https://my-iceberg-rest.example.com" secret: my_iceberg_secret ``` models/my\_iceberg\_model.sql ```sql {{ config( materialized = 'table', catalog_name = 'rest_catalog' ) }} select * from {{ ref('jaffle_shop_customers') }} ``` Run the model with `dbt run -s my_iceberg_model`. Instead of `endpoint`, you can use `endpoint_type: GLUE` or `endpoint_type: S3_TABLES` to attach one of these well-known AWS-managed Iceberg REST endpoints without specifying a URL: catalogs.yml ```yaml catalogs: - name: s3_tables_catalog type: iceberg_rest table_format: iceberg config: duckdb: endpoint_type: S3_TABLES warehouse: "arn:aws:s3tables:us-east-1:123456789012:bucket/example" ``` `endpoint` and `endpoint_type` are mutually exclusive. #### Cross-platform Mesh: reading catalogs managed by other platforms Because a single catalog entry in `catalogs.yml` can carry configuration for multiple platforms at once, you can point DuckDB at the same physical catalog that Snowflake or Databricks writes to — enabling [cross-platform Mesh](https://docs.getdbt.com/docs/mesh/cross-platform-mesh.md) without copying data. ##### Snowflake Horizon [Snowflake Horizon](https://docs.getdbt.com/docs/build/iceberg/adapters/snowflake-iceberg-support.md) is Snowflake's managed Iceberg catalog. Add a `duckdb` block alongside the `snowflake` block to let DuckDB attach to the same catalog: catalogs.yml ```yaml catalogs: - name: horizon_catalog type: horizon table_format: iceberg config: snowflake: external_volume: my_external_volume duckdb: warehouse: horizon_wh endpoint: "https://horizon.example.com/catalog" secret: horizon_secret default_schema: demo ``` ##### Databricks Unity Catalog Similarly, for [Databricks Unity Catalog](https://docs.getdbt.com/docs/build/iceberg/adapters/databricks-iceberg-support.md): catalogs.yml ```yaml catalogs: - name: unity_catalog type: unity table_format: iceberg config: databricks: file_format: delta use_uniform: true duckdb: warehouse: unity_wh endpoint: "https://dbc-example.cloud.databricks.com/api/2.1/unity-catalog/iceberg" default_schema: demo ``` ##### Read-only vs. read-write By default, dbt attaches Horizon and Unity catalogs read-write (`read_only: false`) and applies write-compat `ATTACH` defaults for each (for example, disabling multi-table commits on Unity). Writing to these catalogs from DuckDB requires DuckDB 1.5.4+ and [duckdb-iceberg#1017](https://github.com/duckdb/duckdb-iceberg/issues/1017). If you only need to *read* Iceberg tables that another platform wrote, set `read_only: true`: ```yaml duckdb: warehouse: horizon_wh endpoint: "https://horizon.example.com/catalog" read_only: true ``` #### DuckLake [DuckLake](https://ducklake.select/) is a separate open table format (not Apache Iceberg) built for DuckDB, but you configure it the same way, through `catalogs.yml`. Because DuckLake isn't Iceberg, its catalog entries use `table_format: default`. catalogs.yml ```yaml catalogs: - name: local_lake type: ducklake table_format: default config: duckdb: metadata_path: "metadata.ducklake" data_path: "s3://my-bucket/lake" # optional ``` models/my\_ducklake\_model.sql ```sql {{ config( materialized = 'table', catalog_name = 'local_lake' ) }} select * from {{ ref('jaffle_shop_customers') }} ``` dbt installs the DuckLake extension and attaches the catalog before running your model: ```sql INSTALL ducklake ATTACH IF NOT EXISTS 'ducklake:metadata.ducklake' AS local_lake (DATA_PATH 's3://my-bucket/lake') ``` #### Secrets The `secret` field in a `duckdb` catalog block references a named secret defined in `profiles.yml`, which dbt turns into a DuckDB [`CREATE SECRET`](https://duckdb.org/docs/configuration/secrets_manager.html) statement: profiles.yml ```yaml my_profile: target: dev outputs: dev: type: duckdb path: ':memory:' secrets: - type: iceberg name: my_iceberg_secret # additional key-value pairs become CREATE SECRET parameters # (for example, token, client_id, client_secret) -- see DuckDB's # iceberg extension docs for the parameters your catalog needs. ``` #### DuckDB-specific configs for Iceberg catalogs You can supply these configs, nested under `config.duckdb`, for `horizon`, `unity`, and `iceberg_rest` catalogs: | Field | Required | Description | | ------------------------------------ | -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | | `endpoint` | One of `endpoint`/`endpoint_type` | Full Iceberg REST catalog URL. | | `endpoint_type` | One of `endpoint`/`endpoint_type` | `GLUE` or `S3_TABLES`, for well-known AWS-managed endpoints. | | `warehouse` | Required for `horizon`; required when `endpoint_type` is `S3_TABLES` | Warehouse identifier passed as the `ATTACH` source. | | `secret` | Optional | Name of a DuckDB secret from `profiles.yml` to use for authentication. | | `attach_as` | Optional | Overrides the DuckDB attach alias. Defaults to the catalog's `name`. | | `default_region` | Optional | AWS region, when applicable. | | `default_schema` | Optional | Default schema/namespace within the catalog. | | `max_table_staleness` | Optional | How long DuckDB may serve cached metadata before refreshing. | | `authorization_type` | Optional | `OAUTH2`, `SIGV4`, or `NONE`. Can't be combined with `endpoint_type`. | | `access_delegation_mode` | Optional | `VENDED_CREDENTIALS` or `NONE`. | | `read_only` | Optional | Attach the catalog read-only. Defaults to `false` (read-write). | | `support_nested_namespaces` | Optional | Whether the catalog supports nested namespaces. | | `stage_create_tables` | Optional | Write-compat: stage `CREATE TABLE AS SELECT` writes. Requires DuckDB 1.5.4+. | | `disable_multi_table_commit` | Optional | Write-compat: disable multi-table commits. Requires DuckDB 1.5.4+. | | `skip_create_table_metadata_updates` | Optional | Write-compat: skip metadata updates on `CREATE TABLE`. Requires DuckDB 1.5.4+. | | `remove_files_on_delete` | Optional | Write-compat: remove underlying data files when a table is dropped. Requires DuckDB 1.5.4+. | | `purge_requested` | Optional | Purge underlying files when supported by the catalog. | | `encode_entire_prefix` | Optional | Percent-encode the entire object key prefix. | For `ducklake` catalogs, `config.duckdb` accepts: | Field | Required | Description | | ------------------------- | -------- | ---------------------------------------------------------------------------------------------------------- | | `metadata_path` | Required | Path to the DuckLake metadata store, for example `metadata.ducklake` or a database connection string. | | `data_path` | Optional | Where DuckLake writes data files. | | `attach_as` | Optional | Overrides the DuckDB attach alias. Defaults to the catalog's `name`. | | `metadata_schema` | Optional | Schema within the metadata store to use. | | `metadata_catalog` | Optional | Catalog/database name within the metadata store. | | `data_inlining_row_limit` | Optional | Inline row groups smaller than this many rows into the metadata catalog instead of writing a Parquet file. | | `create_if_not_exists` | Optional | Create the DuckLake catalog if it doesn't already exist. | | `read_only` | Optional | Attach read-only. | | `encrypted` | Optional | Encrypt the DuckLake catalog. | | `automatic_migration` | Optional | Automatically migrate the catalog's DuckLake format version on attach. | | `override_data_path` | Optional | Allow attaching with a `data_path` that differs from the one recorded in an existing catalog. | --- ### Enabling dbt State in Studio Preview ### Enabling dbt State in Studio [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Login required | Usage-based When you enable dbt State in the Studio IDE, it runs automatically on every `dbt run` or `dbt build` during development — skipping unchanged models and reusing production results so your runs are *faster*. You can [turn it on for your development environment](#enabling-dbt-state-on-a-development-environment) so it's the default for everyone, or you can [override that setting just for your own account](#overriding-dbt-state-setting-per-user). **Prerequisite**: An account admin must [enable dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) before you can use it. #### Enabling dbt State on a development environment Enabling dbt State on your development environment turns it on for everyone using the Studio IDE, unless they override it for their own account. 1. Go to **Orchestration** > **Environments** and select your development environment. 2. Click **Settings** > **Edit**. 3. In the **dbt State** section, select **Enable dbt State**. 4. Click **Save**. 5. In the pop-up box, click **Continue** if you want to go ahead with the changes and restart all IDE sessions for this project. #### Overriding dbt State setting per user You can override the development environment's dbt State setting for your own account without affecting other users. Because the user-level setting takes precedence over the environment-level setting, you can turn dbt State on for yourself before enabling it for your whole team, or turn it off when it's enabled at the environment level. 1. Click your account name in the lower-left corner and select **Account settings**. 2. Under **Your profile**, go to **Credentials**. 3. Select the project you want to enable dbt State for. 4. Click **Edit** and go to the **User development settings** section. 5. Under **dbt State**, select one of the following options: * **Enabled**: Enables dbt State for your user regardless of the development environment setting. * **Disabled**: Disables dbt State for your user regardless of the development environment setting. * **Reset (inherit from development)**: Only appears after you've saved an **Enabled** or **Disabled** override. Clears your override and falls back to the dbt State setting configured on your development environment. 6. Click **Save**. #### Related docs * [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) * [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) * [Enable dbt State on individual jobs](https://docs.getdbt.com/docs/deploy/dbt-state-enable-jobs.md) --- ### Enabling dbt State on individual jobs Preview ### Enabling dbt State on individual jobs [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Login required | Usage-based dbt State is available on all job types: deploy, continuous integration (CI), and merge jobs. To enable dbt State on any job — whether already existing or newly created in an environment that doesn't have dbt State enabled: 1. Go to **Orchestration** > **Jobs**. 2. Select the job you want dbt State enabled for. 3. Click **Settings** > **Edit**. 4. In the **Execution settings** section of the job, select **Enable dbt State**. 5. Click **Save**. #### Related docs * [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) * [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) * [Enable dbt State in Studio](https://docs.getdbt.com/docs/deploy/dbt-state-enable-studio.md) --- ### Enhance your code [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/build/environment-variables.md) ###### [Environment variables](https://docs.getdbt.com/docs/build/environment-variables.md) [Learn how you can use environment variables to customize the behavior of a dbt project.](https://docs.getdbt.com/docs/build/environment-variables.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/build/hooks-operations.md) ###### [Hooks and operations](https://docs.getdbt.com/docs/build/hooks-operations.md) [Learn how to use hooks to trigger actions and operations to invoke macros.](https://docs.getdbt.com/docs/build/hooks-operations.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/build/packages.md) ###### [Packages](https://docs.getdbt.com/docs/build/packages.md) [Learn how you can leverage code reuse through packages (libraries).](https://docs.getdbt.com/docs/build/packages.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/build/project-variables.md) ###### [Project variables](https://docs.getdbt.com/docs/build/project-variables.md) [Learn how to use project variables to provide data to models for compilation.](https://docs.getdbt.com/docs/build/project-variables.md) --- ### Enhance your models [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/build/materializations.md) ###### [Materializations](https://docs.getdbt.com/docs/build/materializations.md) [Learn how to use materializations to make dbt models persist in a data platform.](https://docs.getdbt.com/docs/build/materializations.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/build/incremental-models.md) ###### [Incremental models](https://docs.getdbt.com/docs/build/incremental-models.md) [Learn how to use incremental models so you can limit the amount of data that needs to be transformed.](https://docs.getdbt.com/docs/build/incremental-models.md) <br /> --- ### Entities Entities are real-world concepts in a business, such as customers, transactions, and ad campaigns. We often focus our analyses on specific entities, such as customer churn or annual recurring revenue modeling. In our Semantic Layer models, these entities serve as a join key across semantic models. (Applies to dbt v1.12 and later) Within a semantic graph, the required parameters for an entity are `name` and `type`. The `name` refers to either the key column name from the underlying data table, or it may serve as an alias with the column name referenced in the `expr` parameter. The `name` for your entity must be unique to the semantic model and can not be the same as an existing simple metric or `dimension` within that same model. Entities can be specified with a single column or multiple columns. Entities (join keys) in a semantic model are identified by their name. Each entity name must be unique within a semantic model, but it doesn't have to be unique across different semantic models. There are four entity types: * [Primary](#primary) — Has only one record for each row in the table and includes every record in the data platform. This key uniquely identifies each record in the table. * [Unique](#unique) — Contains only one record per row in the table and allows for null values. May have a subset of records in the data warehouse. * [Foreign](#foreign) — A field (or a set of fields) in one table that uniquely identifies a row in another table. This key establishes a link between tables. * [Natural](#natural) — Columns or combinations of columns in a table that uniquely identify a record based on real-world data. This key is derived from actual data attributes. Use entities as dimensions You can also use entities as dimensions, which allows you to aggregate a metric to the granularity of that entity. #### Entity types MetricFlow's join logic depends on the entity `type` you use and determines how to join semantic models. Refer to [Joins](https://docs.getdbt.com/docs/build/join-logic.md) for more info on how to construct joins. ##### Primary A primary key has *only one* record for each row in the table and includes every record in the data platform. It must contain unique values and can't contain null values. Use the primary key to ensure that each record in the table is distinct and identifiable.  Primary key example For example, consider a table of employees with the following columns: ```sql employee_id (primary key) first_name last_name ``` In this case, `employee_id` is the primary key. Each `employee_id` is unique and represents one specific employee. There can be no duplicate `employee_id` and can't be null. ##### Unique A unique key contains *only one* record per row in the table but may have a subset of records in the data warehouse. However, unlike the primary key, a unique key allows for null values. The unique key ensures that the column's values are distinct, except for null values.  Unique key example For example, consider a table of students with the following columns: ```sql student_id (primary key) email (unique key) first_name last_name ``` In this example, `email` is defined as a unique key. Each email address must be unique; however, multiple students can have null email addresses. This is because the unique key constraint allows for one or more null values, but non-null values must be unique. This then creates a set of records with unique emails (non-null) that could be a subset of the entire table, which includes all students. ##### Foreign A foreign key is a field (or a set of fields) in one table that uniquely identifies a row in another table. The foreign key establishes a link between the data in two tables. It can include zero, one, or multiple instances of the same record. It can also contain null values.  Foreign key example For example, consider you have two tables, `customers` and `orders`: customers table: ```sql customer_id (primary key) customer_name ``` orders table: ```sql order_id (primary key) order_date customer_id (foreign key) ``` In this example, the `customer_id` in the `orders` table is a foreign key that references the `customer_id` in the `customers` table. This link means each order is associated with a specific customer. However, not every order must have a customer; the `customer_id` in the orders table can be null or have the same `customer_id` for multiple orders. ##### Natural Natural keys are columns or combinations of columns in a table that uniquely identify a record based on real-world data. For instance, if you have a `sales_person_department` dimension table, the `sales_person_id` can serve as a natural key. You can only use natural keys for [SCD type II dimensions](https://docs.getdbt.com/docs/build/dimensions.md#scd-type-ii). #### Entities configuration The following is the complete spec for entities: (Applies to dbt v1.12 and later) ```yaml models: - name: model_name semantic_model: enabled: true name: semantic_model_name # ...rest of the semantic model config (labels, groups, etc.) columns: - name: transaction_id entity: type: primary # Required; primary | natural | foreign | unique name: entity_name # Optional; defaults to column name description: "A description of the field or role the entity takes in this table" # Additional configuration config: # Optional meta: owner: "Team Name" ``` Here's an example of how to define entities in a semantic model: (Applies to dbt v1.12 and later) ```yaml models: - name: your_model_name semantic_model: enabled: true columns: - name: id_transaction entity: type: primary name: transaction - name: id_order entity: type: foreign name: order ``` #### `derived_semantics` in `entities` Use `derived_semantics` when you need an entity that is not a direct 1:1 mapping to a single physical column. The `expr` field is required when using `derived_semantics`. For example: ```yaml # Derived entities (expression-based) derived_semantics: entities: - name: user type: foreign expr: "substring(id_order from 2)" # Required ``` #### Combine columns with a key If a table doesn't have any key (like a primary key), use *surrogate combination* to form a key that will help you identify a record by combining two columns. This applies to any [entity type](https://docs.getdbt.com/docs/build/entities.md#entity-types). For example, you can combine `date_key` and `brand_code` from the `raw_brand_target_weekly` table to form a *surrogate key*. The following example creates a surrogate key by joining `date_key` and `brand_code` using a pipe (`|`) as a separator. (Applies to dbt v1.12 and later) ```yaml models: - name: your_model_name semantic_model: enabled: true derived_semantics: entities: - name: brand_target_key type: foreign expr: "date_key || '|' || brand_code" ``` #### Examples As mentioned, entities serve as our join keys, using the unique entity name. Therefore, we can join a single `unique` key to multiple `foreign` keys. Consider a `date_categories` table with the following columns: ```sql date_id (primary key) date_day (unique key) fiscal_year_name ``` And an `orders` table with the following columns: ```sql order_id (primary key) ordered_at delivered_at order_total ``` How might we define our Semantic Layer YAML so that we can query `order_total` by `ordered_at` `fiscal_year_name`, and `delivered_at` `fiscal_year_name`? (Applies to dbt v1.12 and later) First, we need to define two `unique` entities in the `date_categories` with the expression set to `date_day`: ```yaml models: - name: date_categories semantic_model: enabled: true agg_time_dimension: date_day columns: - name: date_id entity: type: primary name: date_id - name: date_day granularity: day dimension: type: time name: date_day - name: fiscal_year_name dimension: type: categorical name: fiscal_year_name description: "Formatted fiscal year string (e.g. 'FY2025')" derived_semantics: # Unique entities entities: - name: ordered_at_entity type: unique expr: date_day - name: delivered_at_entity type: unique expr: date_day ``` (Applies to dbt v1.12 and later) Then, we need to add these same entities as `foreign` keys to our `orders` model, with the expression set to `ordered_at` and `delivered_at`: ```yaml models: - name: orders semantic_model: enabled: true agg_time_dimension: ordered_at columns: - name: order_id entity: type: primary name: order_id - name: ordered_at granularity: day dimension: type: time name: ordered_at derived_semantics: # Foreign keys entities: - name: ordered_at_entity type: foreign expr: ordered_at - name: delivered_at_entity type: foreign expr: delivered_at metrics: - name: order_total type: simple agg: sum description: "Total amount for each order including taxes." ``` With this configuration, our semantic models can join on `ordered_at = date_day` via the `ordered_at_entity`, and on `delivered_at = date_day` via the `delivered_at_entity`. To validate our output, we can run: * `dbt sl query --metrics order_total --group-by ordered_at_entity__fiscal_year_name` or * `dbt sl query --metrics order_total --group-by delivered_at_entity__fiscal_year_name` --- ### Environment variables Environment variables can be used to customize the behavior of a dbt project depending on where the project is running. Refer to the docs on [env\_var](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md) for more information on how to call the Jinja function `{{env_var('DBT_KEY','OPTIONAL_DEFAULT')}}` in your project code. Environment Variable Naming and Prefixing Environment variables in dbt must be prefixed with either `DBT_`, `DBT_ENV_SECRET_`, or `DBT_ENV_CUSTOM_ENV_`. Environment variable keys are uppercased and case sensitive. When referencing `{{env_var('DBT_KEY')}}` in your project's code, the key must match exactly the variable defined in dbt's UI. ##### Setting and overriding environment variables This section explains how to set and override environment variables in dbt. * [Order of precedence](#order-of-precedence) * [Setting environment variables](#setting-environment-variables) * [Overriding environment variables at the job level](#overriding-environment-variables-at-the-job-level) * [Overriding environment variables at the personal level](#overriding-environment-variables-at-the-personal-level) * [Local environment variables](#local-environment-variables) ###### Order of precedence Environment variable values can be set in multiple places within dbt. As a result, dbt will interpret environment variables according to the following order of precedence (lowest to highest): [![Environment variables order of precedence](</img/docs/dbt-platform/using-dbt-platform/Environment Variables/env-var-precdence.png?v=2> "Environment variables order of precedence")](#)Environment variables order of precedence There are four levels of environment variables: 1. The optional default argument supplied to the `env_var` Jinja function in code, which can be overridden at (*lowest precedence*) 2. The project-wide level by its default value, which can be overridden at 3. The environment level, which can in turn be overridden again at 4. The job level (job override) or in the Studio IDE for an individual dev (personal override). (*highest precedence*) ###### Setting environment variables To set environment variables at the project and environment level, click **Orchestration** in the left-side menu, then select **Environments**. Click **Environment variables** to add and update your environment variables. [![Environment variables tab](</img/docs/dbt-platform/using-dbt-platform/Environment Variables/navigate-to-env-vars.png?v=2> "Environment variables tab")](#)Environment variables tab You'll notice there is a **Project default** column. This is a great place to set a value that will persist across your whole project, independent of where the code is run. We recommend setting this value when you want to supply a catch-all default or add a project-wide token or secret. To the right of the **Project default** column, you can see all your environments. Values set at the environment level take priority over the project-level default value. This is where you can tell dbt to interpret an environment value differently in your staging versus production environment, as an example. [![Setting project level and environment level values](</img/docs/dbt-platform/using-dbt-platform/Environment Variables/project-environment-view.png?v=2> "Setting project level and environment level values")](#)Setting project level and environment level values ###### Overriding environment variables at the job level You may have multiple jobs that run in the same environment, and you'd like the environment variable to be interpreted differently depending on the job. When setting up or editing a job, you will see a section where you can override environment variable values defined at the environment or project level. [![Navigating to environment variables job override settings](</img/docs/dbt-platform/using-dbt-platform/Environment Variables/job-override.gif?v=2> "Navigating to environment variables job override settings")](#)Navigating to environment variables job override settings Every job runs in a specific deployment environment, and by default, a job will inherit the values set at the environment level (or the highest precedence level set) for the environment in which it runs. If you'd like to set a different value at the job level, edit the value to override it. [![Setting a job override value](</img/docs/dbt-platform/using-dbt-platform/Environment Variables/job-override.png?v=2> "Setting a job override value")](#)Setting a job override value ###### Overriding environment variables at the personal level You can also set a personal value override for an environment variable when you develop in the dbt-integrated developer environment (Studio IDE). By default, dbt uses environment variable values set in the project's development environment. To see and override these values, from dbt: 1. Click your account name in the left side menu and select **Account settings**. 2. Under the **Your profile** section, click **Credentials** and then select your project. 3. Scroll to the **Environment variables** section and click **Edit** to make the necessary changes. To supply an override, developers can edit and specify a different value to use. These values will be respected in the Studio IDE both for the Results and Compiled SQL tabs. [![Setting a personal override value](</img/docs/dbt-platform/using-dbt-platform/Environment Variables/personal-override.png?v=2> "Setting a personal override value")](#)Setting a personal override value Appropriate coverage If you have not set a project level default value for every environment variable, it may be possible that dbt does not know how to interpret the value of an environment variable in all contexts. In such cases, dbt will throw a compilation error: "Env var required but not provided". Changing environment variables mid-session in the Studio IDE If you change the value of an environment variable mid-session while using the Studio IDE, you may have to refresh the Studio IDE for the change to take effect. To refresh the Studio IDE mid-development, click on either the green 'ready' signal or the red 'compilation error' message at the bottom right corner of the Studio IDE. A new modal will pop up, and you should select the **Restart IDE** button. This loads your environment variable values into your development environment. [![Refreshing IDE mid-session](</img/docs/dbt-platform/using-dbt-platform/Environment Variables/refresh-ide.png?v=2> "Refreshing IDE mid-session")](#)Refreshing IDE mid-session There are some known issues with partial parsing of a project and changing environment variables mid-session in the IDE. If you find that your dbt project is not compiling to the values you've set, try deleting the `target/partial_parse.msgpack` file in your dbt project which will force dbt to re-compile your whole project. ###### Local environment variables If you are using the dbt VS Code extension, you can set environment variables locally in your shell profile (`~/.zshrc` or `~/.bashrc`) or in a `.env` file at the root level of your dbt project. For more information, refer to [Configure the dbt VS Code extension](https://docs.getdbt.com/docs/configure-dbt-extension.md#set-environment-variables-locally). ##### Handling secrets While all environment variables are encrypted at rest in dbt, dbt has additional capabilities for managing environment variables with secret or otherwise sensitive values. If you want a particular environment variable to be scrubbed from all logs and error messages, in addition to obfuscating the value in dbt, you can prefix the key with `DBT_ENV_SECRET_`. Environment variables prefixed with `DBT_ENV_SECRET_` are protected with additional security controls. They are encrypted at rest using an encryption key (for example, AWS KMS when your deployment is hosted on AWS) and can only be accessed by decrypting them with that key. Decryption is restricted to specific flows where the value is required, such as when a job runs. Secret keys are never written to logs or error messages and are obfuscated in dbt, so they are not exposed in the UI or artifacts, and are only available to dbt at runtime as needed. [![DBT\_ENV\_SECRET prefix obfuscation](</img/docs/dbt-platform/using-dbt-platform/Environment Variables/DBT_ENV_SECRET.png?v=2> "DBT_ENV_SECRET prefix obfuscation")](#)DBT\_ENV\_SECRET prefix obfuscation **Note**: An environment variable can be used to store a [git token for repo cloning](https://docs.getdbt.com/docs/build/environment-variables.md#clone-private-packages). We recommend you make the git token's permissions read only and consider using a machine account or service user's PAT with limited repo access in order to practice good security hygiene. ##### Special environment variables dbt has a number of pre-defined variables built in. Variables are set automatically and cannot be changed. This means that the order of precedence for overriding environment variables doesn't apply to these pre-defined variables at the project, environment, or job level. ###### Studio IDE details The following environment variable is set automatically for the Studio IDE: * `DBT_CLOUD_GIT_BRANCH` — Provides the development Git branch name in the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md). * The variable changes when the branch is changed. * Doesn't require restarting the Studio IDE after a branch change. * Currently not available in the [dbt CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md). Use case — This is useful in cases where you want to dynamically use the Git branch name as a prefix for a [development schema](https://docs.getdbt.com/docs/build/custom-schemas.md) (`{{ env_var('DBT_CLOUD_GIT_BRANCH') }}`). ###### dbt platform context The following environment variables are set automatically: * `DBT_ENV` — This key is reserved for the dbt application and will always resolve to 'prod'. For deployment runs only. * `DBT_CLOUD_ENVIRONMENT_NAME` — The name of the dbt environment in which `dbt` is running. * `DBT_CLOUD_ENVIRONMENT_TYPE` — The type of dbt environment in which `dbt` is running. The valid values are `dev`, `staging`, or `prod`. The value will be empty for [General deployment environments](https://docs.getdbt.com/docs/dbt-platform-environments.md#types-of-environments), so use a default like `{{ env_var('DBT_CLOUD_ENVIRONMENT_TYPE', '') }}`. * `DBT_CLOUD_INVOCATION_CONTEXT` — The context type in which `dbt` is invoked. The values are `dev`, `staging`, `prod`, or `ci`. * Additionally, use `DBT_CLOUD_INVOCATION_CONTEXT` in the `generate_schema_name()` macro to define explicit guidelines to use the default schema only (with the `dbt_cloud_pr prefix`) in CI job runs, even if those CI jobs run in the same environment as production jobs. ###### Run details * `DBT_CLOUD_PROJECT_ID` — The ID of the dbt Project for this run * `DBT_CLOUD_JOB_ID` — The ID of the dbt Job for this run * `DBT_CLOUD_RUN_ID` — The ID of this particular run * `DBT_CLOUD_RUN_REASON_CATEGORY` — The "category" of the trigger for this run (one of: `scheduled`, `github_pull_request`, `gitlab_merge_request`, `azure_pull_request`, `other`) * `DBT_CLOUD_RUN_REASON` — The specific trigger for this run (for example, `Scheduled`, `Kicked off by <email>`, or a custom value set through the API) * `DBT_CLOUD_ENVIRONMENT_ID` — The ID of the environment for this run * `DBT_CLOUD_ACCOUNT_ID` — The ID of the dbt account for this run ###### Git details *The following variables are currently only available for GitHub, GitLab, and Azure DevOps PR builds triggered using a webhook* * `DBT_CLOUD_PR_ID` — The Pull Request ID in the connected version control system * `DBT_CLOUD_GIT_SHA` — The git commit SHA which is being run for this Pull Request build ##### Example usage Environment variables can be used in many ways, and they give you the power and flexibility to do what you want to do more easily in dbt.  Clone private packages Now that you can set secrets as environment variables, you can pass git tokens into your package HTTPS URLs to allow for on-the-fly cloning of private repositories. Read more about enabling [private package cloning](https://docs.getdbt.com/docs/build/packages.md#private-packages).  Dynamically set your warehouse in your Snowflake connection Environment variables make it possible to dynamically change the Snowflake virtual warehouse size depending on the job. Instead of calling the warehouse name directly in your project connection, you can reference an environment variable which will get set to a specific virtual warehouse at runtime. For example, suppose you'd like to run a full-refresh job in an XL warehouse, but your incremental job only needs to run in a medium-sized warehouse. Both jobs are configured in the same dbt environment. In your connection configuration, you can use an environment variable to set the warehouse name to `{{env_var('DBT_WAREHOUSE')}}`. Then in the job settings, you can set a different value for the `DBT_WAREHOUSE` environment variable depending on the job's workload. Currently, it's not possible to dynamically set environment variables across models within a single run. This is because each env\_var can only have a single set value for the entire duration of the run. **Note** — You can also use this method with Databricks SQL Warehouse. [![Adding environment variables to your connection credentials](</img/docs/dbt-platform/using-dbt-platform/Environment Variables/warehouse-override.png?v=2> "Adding environment variables to your connection credentials")](#)Adding environment variables to your connection credentials Environment variables and Snowflake OAuth limitations Env vars work well with username/password and keypair, including scheduled jobs, because dbt v1 consumes the Jinja inserted into the autogenerated [`profiles.yml`](https://docs.getdbt.com/docs/local/profiles.yml.md) and resolves it to do an `env_var` lookup. However, there are some limitations when using env vars with Snowflake OAuth Connection settings: * You can't use them in the account/host field, but they can be used for database, warehouse, and role. For these fields, [use extended attributes](https://docs.getdbt.com/docs/deploy/deploy-environments.md#deployment-connection). Something to note, if you supply an environment variable in the account/host field, Snowflake OAuth Connection will **fail** to connect. This happens because the field doesn't pass through Jinja rendering, so dbt simply passes the literal `env_var` code into a URL string like `{{ env_var("DBT_ACCOUNT_HOST_NAME") }}.snowflakecomputing.com`, which is an invalid hostname. Use [extended attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) instead.  Audit your run metadata Here's another motivating example that uses the dbt run ID, which is set automatically at each run. This additional data field can be used for auditing and debugging: ```sql {{ config(materialized='incremental', unique_key='user_id') }} with users_aggregated as ( select user_id, min(event_time) as first_event_time, max(event_time) as last_event_time, count(*) as count_total_events from {{ ref('users') }} group by 1 ) select *, -- Inject the run id if present, otherwise use "manual" '{{ env_var("DBT_CLOUD_RUN_ID", "manual") }}' as _audit_run_id from users_aggregated ```  Configure Semantic Layer credentials Use [Extended Attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) and [Environment Variables](https://docs.getdbt.com/docs/build/environment-variables.md) when connecting to the Semantic Layer. If you set a value directly in the Semantic Layer Credentials, it will have a higher priority than Extended Attributes. When using environment variables, the default value for the environment will be used. For example, set the warehouse by using `{{env_var('DBT_WAREHOUSE')}}` in your Semantic Layer credentials. Similarly, if you set the account value using `{{env_var('DBT_ACCOUNT')}}` in Extended Attributes, dbt will check both the Extended Attributes and the environment variable. --- ### Example usage for dbt State Preview ### Example usage for dbt State [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Login required | Usage-based These examples use the Jaffle Shop project to show side-by-side comparisons of CLI output with and without dbt State enabled. To enable dbt State, follow the steps in [Setting up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md). The following examples use this [Jaffle Shop project](https://github.com/dbt-labs/jaffle-shop) DAG as a reference. You can refer to it to understand the model lineage for each scenario. [![The Jaffle Shop DAG](/img/docs/dbt-state/dbt_state_dag.png?v=2 "The Jaffle Shop DAG")](#)The Jaffle Shop DAG Each of the following scenarios shows how a run differs between dbt alone and dbt with dbt State, using the same command and project. | Scenario | Command | What dbt State changes | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | | [Initial run in empty schema](#initial-run-in-empty-schema) | `dbt run --target prod` | Same result | | [Second run](#second-run) | `dbt run --target prod` | Reuses unchanged table models and most views; may rebuild views with `select *` directly on a `ref()` or `source()` | | [Selecting a model in a fresh dev environment after changing the customers model](#selecting-a-model-in-a-fresh-dev-environment-after-changing-the-customers-model) | `dbt run --target dev --select "customers"` | Defers to prod for upstream models | | [Selecting a model in a new dev schema with no model changes](#selecting-a-model-in-a-new-dev-schema-with-no-model-changes) | `dbt run --target dev --select "customers"` | Defers and clones unchanged models | <br /> Every skipped model is a model you didn't pay to rebuild. dbt State tracks what's changed and skips the rest — reducing run time and warehouse costs. #### Initial run in empty schema ```shell dbt run --target prod ``` With no prior state to compare against, dbt builds every model from scratch. dbt State captures metadata from this run for future comparisons. ##### Without dbt State ```shell Running with dbt=1.12.0-b2 Registered adapter: snowflake=1.11.5 Found 12 models, 6 seeds, 27 data tests, 6 sources, 644 macros, 3 unit tests Concurrency: 1 threads (target='prod') 1 of 12 START sql view model jaffle_analytics.stg_customers .................... [RUN] 1 of 12 OK created sql view model jaffle_analytics.stg_customers ............... [SUCCESS 1 in 0.54s] 2 of 12 START sql view model jaffle_analytics.stg_locations .................... [RUN] 2 of 12 OK created sql view model jaffle_analytics.stg_locations ............... [SUCCESS 1 in 0.48s] 3 of 12 START sql view model jaffle_analytics.stg_order_items .................. [RUN] 3 of 12 OK created sql view model jaffle_analytics.stg_order_items ............. [SUCCESS 1 in 0.51s] 4 of 12 START sql view model jaffle_analytics.stg_orders ....................... [RUN] 4 of 12 OK created sql view model jaffle_analytics.stg_orders .................. [SUCCESS 1 in 0.77s] 5 of 12 START sql view model jaffle_analytics.stg_products ..................... [RUN] 5 of 12 OK created sql view model jaffle_analytics.stg_products ................ [SUCCESS 1 in 0.58s] 6 of 12 START sql view model jaffle_analytics.stg_supplies ..................... [RUN] 6 of 12 OK created sql view model jaffle_analytics.stg_supplies ................ [SUCCESS 1 in 0.63s] 7 of 12 START sql table model jaffle_analytics.locations ....................... [RUN] 7 of 12 OK created sql table model jaffle_analytics.locations .................. [SUCCESS 1 in 1.81s] 8 of 12 START sql table model jaffle_analytics.products ........................ [RUN] 8 of 12 OK created sql table model jaffle_analytics.products ................... [SUCCESS 1 in 1.33s] 9 of 12 START sql table model jaffle_analytics.order_items ..................... [RUN] 9 of 12 OK created sql table model jaffle_analytics.order_items ................ [SUCCESS 1 in 2.09s] 10 of 12 START sql table model jaffle_analytics.supplies ....................... [RUN] 10 of 12 OK created sql table model jaffle_analytics.supplies .................. [SUCCESS 1 in 1.26s] 11 of 12 START sql table model jaffle_analytics.orders ......................... [RUN] 11 of 12 OK created sql table model jaffle_analytics.orders .................... [SUCCESS 1 in 1.70s] 12 of 12 START sql table model jaffle_analytics.customers ...................... [RUN] 12 of 12 OK created sql table model jaffle_analytics.customers ................. [SUCCESS 1 in 1.77s] Finished running 6 table models, 6 view models in 0 hours 0 minutes and 17.77 seconds (17.77s). Completed successfully Done. PASS=12 WARN=0 ERROR=0 SKIP=0 NO-OP=0 TOTAL=12 ``` ##### With dbt State ```shell Running with dbt=1.12.0-b2 State adapter: dbt-state v2.43.1 is enabled Registered adapter: snowflake=1.11.5 Found 12 models, 6 seeds, 27 data tests, 6 sources, 644 macros, 3 unit tests Concurrency: 1 threads (target='prod') 1 of 12 START sql view model jaffle_analytics.stg_customers .................... [RUN] State adapter: Fetching freshness metadata 1 of 12 OK created sql view model jaffle_analytics.stg_customers ............... [SUCCESS 1 in 0.54s] 2 of 12 START sql view model jaffle_analytics.stg_locations .................... [RUN] 2 of 12 OK created sql view model jaffle_analytics.stg_locations ............... [SUCCESS 1 in 0.48s] 3 of 12 START sql view model jaffle_analytics.stg_order_items .................. [RUN] 3 of 12 OK created sql view model jaffle_analytics.stg_order_items ............. [SUCCESS 1 in 0.51s] 4 of 12 START sql view model jaffle_analytics.stg_orders ....................... [RUN] 4 of 12 OK created sql view model jaffle_analytics.stg_orders .................. [SUCCESS 1 in 0.77s] 5 of 12 START sql view model jaffle_analytics.stg_products ..................... [RUN] 5 of 12 OK created sql view model jaffle_analytics.stg_products ................ [SUCCESS 1 in 0.58s] 6 of 12 START sql view model jaffle_analytics.stg_supplies ..................... [RUN] 6 of 12 OK created sql view model jaffle_analytics.stg_supplies ................ [SUCCESS 1 in 0.63s] 7 of 12 START sql table model jaffle_analytics.locations ....................... [RUN] 7 of 12 OK created sql table model jaffle_analytics.locations .................. [SUCCESS 1 in 1.81s] 8 of 12 START sql table model jaffle_analytics.products ........................ [RUN] 8 of 12 OK created sql table model jaffle_analytics.products ................... [SUCCESS 1 in 1.33s] 9 of 12 START sql table model jaffle_analytics.order_items ..................... [RUN] 9 of 12 OK created sql table model jaffle_analytics.order_items ................ [SUCCESS 1 in 2.09s] 10 of 12 START sql table model jaffle_analytics.supplies ....................... [RUN] 10 of 12 OK created sql table model jaffle_analytics.supplies .................. [SUCCESS 1 in 1.26s] 11 of 12 START sql table model jaffle_analytics.orders ......................... [RUN] 11 of 12 OK created sql table model jaffle_analytics.orders .................... [SUCCESS 1 in 1.70s] 12 of 12 START sql table model jaffle_analytics.customers ...................... [RUN] 12 of 12 OK created sql table model jaffle_analytics.customers ................. [SUCCESS 1 in 1.77s] Finished running 6 table models, 6 view models in 0 hours 0 minutes and 17.77 seconds (17.77s). Completed successfully. Total cache hits: 0. Estimated time saved: 0.00s. Freshness tolerance: 45m. Done. PASS=12 WARN=0 ERROR=0 SKIP=0 NO-OP=0 REUSED=0 TOTAL=12 ``` #### Second run ```shell dbt run --target prod ``` For each model, dbt State compares the current logic and upstream data against the previous run. If nothing has changed, dbt State skips the build or clones the result from another environment. With dbt State enabled, all twelve models are reused — nothing changed, so there's nothing to rebuild. dbt State reuses views when their SQL logic is unchanged, even if new data has arrived upstream. Because views don't store data, they always read directly from the underlying tables when queried. The staging views shown here use `select *` on a CTE, which dbt State can reuse. For details on `select *` patterns that can force a rebuild, refer to [Views with `select *`](https://docs.getdbt.com/faqs/State/views-rebuilt.md#views-with-select). ##### Without dbt State ```shell Running with dbt=1.12.0-b2 Registered adapter: snowflake=1.11.5 Found 12 models, 6 seeds, 27 data tests, 6 sources, 644 macros, 3 unit tests Concurrency: 1 threads (target='prod') 1 of 12 START sql view model jaffle_analytics.stg_customers .................... [RUN] 1 of 12 OK created sql view model jaffle_analytics.stg_customers ............... [SUCCESS 1 in 0.54s] 2 of 12 START sql view model jaffle_analytics.stg_locations .................... [RUN] 2 of 12 OK created sql view model jaffle_analytics.stg_locations ............... [SUCCESS 1 in 0.48s] 3 of 12 START sql view model jaffle_analytics.stg_order_items .................. [RUN] 3 of 12 OK created sql view model jaffle_analytics.stg_order_items ............. [SUCCESS 1 in 0.51s] 4 of 12 START sql view model jaffle_analytics.stg_orders ....................... [RUN] 4 of 12 OK created sql view model jaffle_analytics.stg_orders .................. [SUCCESS 1 in 0.77s] 5 of 12 START sql view model jaffle_analytics.stg_products ..................... [RUN] 5 of 12 OK created sql view model jaffle_analytics.stg_products ................ [SUCCESS 1 in 0.58s] 6 of 12 START sql view model jaffle_analytics.stg_supplies ..................... [RUN] 6 of 12 OK created sql view model jaffle_analytics.stg_supplies ................ [SUCCESS 1 in 0.63s] 7 of 12 START sql table model jaffle_analytics.locations ....................... [RUN] 7 of 12 OK created sql table model jaffle_analytics.locations .................. [SUCCESS 1 in 1.81s] 8 of 12 START sql table model jaffle_analytics.products ........................ [RUN] 8 of 12 OK created sql table model jaffle_analytics.products ................... [SUCCESS 1 in 1.33s] 9 of 12 START sql table model jaffle_analytics.order_items ..................... [RUN] 9 of 12 OK created sql table model jaffle_analytics.order_items ................ [SUCCESS 1 in 2.09s] 10 of 12 START sql table model jaffle_analytics.supplies ....................... [RUN] 10 of 12 OK created sql table model jaffle_analytics.supplies .................. [SUCCESS 1 in 1.26s] 11 of 12 START sql table model jaffle_analytics.orders ......................... [RUN] 11 of 12 OK created sql table model jaffle_analytics.orders .................... [SUCCESS 1 in 1.70s] 12 of 12 START sql table model jaffle_analytics.customers ...................... [RUN] 12 of 12 OK created sql table model jaffle_analytics.customers ................. [SUCCESS 1 in 1.77s] Finished running 6 table models, 6 view models in 0 hours 0 minutes and 17.77 seconds (17.77s). Completed successfully Done. PASS=12 WARN=0 ERROR=0 SKIP=0 NO-OP=0 TOTAL=12 ``` ##### With dbt State ```shell Running with dbt=1.12.0-b2 State adapter: dbt-state v2.43.1 is enabled Registered adapter: snowflake=1.11.5 Unable to do partial parsing because of a version mismatch Found 12 models, 6 seeds, 27 data tests, 6 sources, 658 macros, 3 unit tests Concurrency: 1 threads (target='prod') 1 of 12 START sql view model jaffle_analytics.stg_customers .................... [RUN] State adapter: Fetching freshness metadata 1 of 12 OK created sql view model jaffle_analytics.stg_customers ............... [No new changes in 2.73s] 2 of 12 START sql view model jaffle_analytics.stg_locations .................... [RUN] 2 of 12 OK created sql view model jaffle_analytics.stg_locations ............... [No new changes in 0.87s] 3 of 12 START sql view model jaffle_analytics.stg_order_items .................. [RUN] 3 of 12 OK created sql view model jaffle_analytics.stg_order_items ............. [No new changes in 1.04s] 4 of 12 START sql view model jaffle_analytics.stg_orders ....................... [RUN] 4 of 12 OK created sql view model jaffle_analytics.stg_orders .................. [No new changes in 0.74s] 5 of 12 START sql view model jaffle_analytics.stg_products ..................... [RUN] 5 of 12 OK created sql view model jaffle_analytics.stg_products ................ [No new changes in 1.47s] 6 of 12 START sql view model jaffle_analytics.stg_supplies ..................... [RUN] 6 of 12 OK created sql view model jaffle_analytics.stg_supplies ................ [No new changes in 0.92s] 7 of 12 START sql table model jaffle_analytics.locations ....................... [RUN] 7 of 12 OK created sql table model jaffle_analytics.locations .................. [No new changes in 1.84s] 8 of 12 START sql table model jaffle_analytics.products ........................ [RUN] 8 of 12 OK created sql table model jaffle_analytics.products ................... [No new changes in 1.99s] 9 of 12 START sql table model jaffle_analytics.order_items ..................... [RUN] 9 of 12 OK created sql table model jaffle_analytics.order_items ................ [No new changes in 2.35s] 10 of 12 START sql table model jaffle_analytics.supplies ....................... [RUN] 10 of 12 OK created sql table model jaffle_analytics.supplies .................. [No new changes in 0.24s] 11 of 12 START sql table model jaffle_analytics.orders ......................... [RUN] 11 of 12 OK created sql table model jaffle_analytics.orders .................... [No new changes in 1.07s] 12 of 12 START sql table model jaffle_analytics.customers ...................... [RUN] 12 of 12 OK created sql table model jaffle_analytics.customers ................. [No new changes in 2.19s] Finished running 6 table models, 6 view models in 0 hours 0 minutes and 19.79 seconds (19.79s). Completed successfully. Total cache hits: 12. Estimated time saved: 17.77s. Freshness tolerance: 45m. Done. PASS=0 WARN=0 ERROR=0 SKIP=0 NO-OP=0 REUSED=12 TOTAL=12 ``` #### Selecting a model in a fresh dev environment after changing the customers model ```shell dbt run --target dev --select "customers" ``` Imagine you've made a small change to the `customers` model and run it in a fresh developer schema. Without dbt State, dbt v1 fails because upstream relations are missing. With dbt State, dbt [defers](https://docs.getdbt.com/docs/deploy/dbt-state-deferral.md) to prod for upstream models and runs only the updated `customers` model. ##### Without dbt State ```shell Running with dbt=1.12.0-b2 Registered adapter: snowflake=1.11.5 Found 12 models, 6 seeds, 27 data tests, 6 sources, 658 macros, 3 unit tests Concurrency: 1 threads (target='dev') 1 of 1 START sql table model dbt_schema.customers .............................. [RUN] 1 of 1 ERROR creating sql table model dbt_schema.customers ..................... [ERROR in 0.36s] Finished running 1 table model in 0 hours 0 minutes and 5.11 seconds (5.11s). Completed with 1 error, 0 partial successes, and 0 warnings: [ERROR]: in model customers (models/marts/customers.sql) Database Error in model customers (models/marts/customers.sql) 002003 (42S02): SQL compilation error: Object 'ANALYTICS.DBT_SCHEMA.STG_CUSTOMERS' does not exist or not authorized. compiled code at target/run/jaffle_shop/models/marts/customers.sql compiled code at target/compiled/jaffle_shop/models/marts/customers.sql Done. PASS=0 WARN=0 ERROR=1 SKIP=0 NO-OP=0 REUSED=0 TOTAL=1 ``` ##### With dbt State ```shell Running with dbt=1.12.0-b2 State adapter: dbt-state v2.43.1 is enabled Registered adapter: snowflake=1.11.5 Found 12 models, 6 seeds, 27 data tests, 6 sources, 658 macros, 3 unit tests Concurrency: 1 threads (target='dev') 1 of 1 START sql table model dbt_schema.customers .............................. [RUN] State adapter: Fetching freshness metadata 1 of 1 OK created sql table model dbt_schema.customers ......................... [SUCCESS 128 in 52.46s] Finished running 1 table model in 0 hours 0 minutes and 58.06 seconds (58.06s). Completed successfully Done. PASS=1 WARN=0 ERROR=0 SKIP=0 NO-OP=0 REUSED=0 TOTAL=1 ``` #### Selecting a model in a new dev schema with no model changes ```shell dbt run --target dev --select "customers" ``` Suppose you create a fresh dev schema and run only the `customers` model. Without dbt State, dbt v1 fails because there is no data in the schema. With dbt State, dbt knows `customers` just ran in another schema: it defers to prod for upstream models and clones `customers` because the outcome is unchanged. ##### Without dbt State ```shell Running with dbt=1.12.0-b2 Registered adapter: snowflake=1.11.5 Found 12 models, 6 seeds, 27 data tests, 6 sources, 658 macros, 3 unit tests Concurrency: 1 threads (target='dev') 1 of 1 START sql table model dbt_schema.customers .............................. [RUN] 1 of 1 ERROR creating sql table model dbt_schema.customers ..................... [ERROR in 0.36s] Finished running 1 table model in 0 hours 0 minutes and 5.11 seconds (5.11s). Completed with 1 error, 0 partial successes, and 0 warnings: [ERROR]: in model customers (models/marts/customers.sql) Database Error in model customers (models/marts/customers.sql) 002003 (42S02): SQL compilation error: Object 'ANALYTICS.DBT_SCHEMA.STG_CUSTOMERS' does not exist or not authorized. compiled code at target/run/jaffle_shop/models/marts/customers.sql compiled code at target/compiled/jaffle_shop/models/marts/customers.sql Done. PASS=0 WARN=0 ERROR=1 SKIP=0 NO-OP=0 REUSED=0 TOTAL=1 ``` ##### With dbt State ```shell Running with dbt=1.12.0-b2 State adapter: dbt-state v2.43.1 is enabled Registered adapter: snowflake=1.11.5 Found 12 models, 6 seeds, 27 data tests, 6 sources, 658 macros, 3 unit tests Concurrency: 1 threads (target='dev') 1 of 1 START sql table model dbt_schema.customers .............................. [RUN] State adapter: Fetching freshness metadata 1 of 1 OK created sql table model dbt_schema.customers ......................... [Cloned from other environment in 57.19s] Finished running 1 table model in 0 hours 1 minutes and 3.50 seconds (63.50s). Completed successfully. Total cache hits: 1. Estimated time saved: 2.20s. Freshness tolerance: 45m. Done. PASS=0 WARN=0 ERROR=0 SKIP=0 NO-OP=0 REUSED=1 TOTAL=1 ``` #### Related docs * [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) * [Setting up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) * [Non-interactive environment setup for dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-cicd.md) * [Configuring deferral in dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-deferral.md) * [Monitoring dbt State activity in dbt platform](https://docs.getdbt.com/docs/deploy/dbt-state-interface.md) * [Migrating from state-aware orchestration to dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md) * [dbt State configs](https://docs.getdbt.com/reference/resource-configs/dbt-state-configs.md) --- ### Explore cost data dbt platform | Enterprise, Enterprise+ You can access Cost Insights in these different dbt platform areas: * [Project dashboard](#project-dashboard) * [Catalog on Model page](#model-performance-in-catalog) * [Job details page](#job-details) Each view provides different levels of detail to help you understand your warehouse spending and optimization impact. Cost and cost reduction estimates are based on historical runs and reflect actual usage, *not* forecasts of future costs. #### Prerequisites To view cost data, ensure you have: * One of the roles listed in [Assign required permissions](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md#assign-required-permissions). * A supported data warehouse: * Snowflake * BigQuery * Databricks * Amazon Redshift [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") For more information, see [Set up Cost Insights](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md). note For accounts already using dbt State or state-aware orchestration before Cost Insights is enabled, at least one full model build must occur within the last 10 days to establish a baseline for cost reduction calculations. If you don't see cost reduction data, try running a full build to establish the baseline. #### Project dashboard The Cost Insights section in your project dashboard gives you a high-level view of warehouse costs and the impact of optimization through dbt State or state-aware orchestration. [![Cost Insights in the project dashboard](/img/docs/dbt-platform/cost-insights/cost-insights-project.png?v=2 "Cost Insights in the project dashboard")](#)Cost Insights in the project dashboard ##### Access To go to your project dashboard, select your project in the main menu and click **Dashboard**. ##### Key metrics The project dashboard displays the following metrics that summarize the overall cost and optimization impact for your project: * **Total cost reduction** * **Total % reduction** * **Total query run time reduction** * **Reused assets** ##### Filters You can customize the cost data you want to view by: * **Deployment type**: Production or Staging * **Last**: 30 days, 60 days, 90 days, 6 months, or 1 year * **View**: Daily, Weekly, or Monthly * **Assets**: All, Models, Tests ##### Visualization tabs The project dashboard includes the following tabs that help you analyze cost and optimization trends over time. * **Cost**: Shows the estimated build cost reduction when using dbt State or state-aware orchestration. * **Usage**: Shows the estimated warehouse usage consumed and the reduction in usage from dbt State or state-aware orchestration over the selected timeframe. The **Usage** tab represents generic usage for your warehouse. The specific unit depends on your data warehouse: * Snowflake: Credits * BigQuery: Slot hours or bytes scanned (currently combined into one generic usage number) * Databricks: Databricks Units (DBUs) * Amazon Redshift Serverless: Redshift Processing Unit hours (RPU-hours) * Amazon Redshift Provisioned: Node-hours * **Query run time**: Shows the estimated reduction in build time when using dbt State or state-aware orchestration. * **Builds**: Shows total builds split into number of assets rebuilt and assets reused by dbt State or state-aware orchestration. ##### Table view Access the table view by clicking **Show table**, which provides detailed optimization data such as models reused, usage reduction, and cost reduction. Use the **All** and **Jobs** buttons to switch between views: * **All**: Shows aggregated cost data across all jobs in the project. * **Jobs**: Shows cost data broken down by individual jobs. When viewing the table, you can export the data as a CSV file using the **Download** button. When **Jobs** is selected, the CSV export includes job-level data. #### Model performance in Catalog The **Model performance** section in Catalog displays historical trends to help you identify optimization opportunities and understand model resource consumption. [![Cost Insights in Catalog](/img/docs/dbt-platform/cost-insights/cost-insights-model.png?v=2 "Cost Insights in Catalog")](#)Cost Insights in Catalog ##### Access To access model performance data: 1. From the main menu, go to **Catalog**. 2. Click your project from the file tree. 3. Navigate to the model whose cost data you want to view. You can search for it or click **Models** under **Project assets** in the sidebar to view all available models in the project. 4. Go to the the **Performance** tab on the model's details page. ##### Key metrics The **Model performance** section displays the following metrics that summarize the overall cost and optimization impact for your project: * **Total cost reduction** * **Total % reduction** * **Total query run time deduction** * **Reused assets** (when dbt State or state-aware orchestration is enabled) ##### Filters Use the time period filter to customize the data you want to view: from the last 3 months up to the last 1 week. For **Cost insights**, **Usage**, and **Query run time** tabs, you can set the view granularity by **Daily**, **Weekly**, or **Monthly**. ##### Visualization tabs * **Cost insights**: Shows the estimated warehouse costs incurred by this model and cost reduction from dbt State or state-aware orchestration. * **Usage**: Shows the estimated warehouse usage consumed by this model over time. The **Usage** tab represents generic usage for your warehouse. The specific unit depends on your data warehouse: * Snowflake: Credits * BigQuery: Slot hours or bytes scanned (currently combined into one generic usage number) * Databricks: Databricks Units (DBUs) * Amazon Redshift Serverless: Redshift Processing Unit hours (RPU-hours) * Amazon Redshift Provisioned: Node-hours * **Query run time**: Shows the estimated query execution time and the reduction in run duration from dbt State or state-aware orchestration. * **Build time**: Shows average execution time for the model and how it trends over the selected period. * **Build count**: Tracks how many times the model was built or reused, including any failures or errors. * **Test results**: Displays test execution outcomes and pass/fail rates for tests on this model. * **Consumption queries**: Shows queries running against this model, helping you understand downstream usage patterns. ##### Table view For **Cost insights**, **Usage**, and **Query run time** tabs, you can access the table view by clicking **Show table**, which provides detailed optimization data such as models reused, usage reduction, and cost reduction. Use the **All** and **Jobs** buttons to switch between views: * **All**: Shows aggregated cost data across all jobs in the project. * **Jobs**: Shows cost data broken down by individual jobs. When viewing the table, you can export the data as a CSV file using the **Download** button. When **Jobs** is selected, the CSV export includes job-level data. ##### Chart interactions For **Build time** and **Build count** tabs: * Click on any data point in the charts to see a detailed table listing all job runs for that day. * Each row in the table provides a direct link to the run details if you want to investigate further. #### Job details The **Insights** section on the Job details page provides cost and performance data for individual jobs. [![Cost Insights in job details](/img/docs/dbt-platform/cost-insights/cost-insights-job.png?v=2 "Cost Insights in job details")](#)Cost Insights in job details ##### Access To access job details, select your project in the main menu and go to **Orchestration** > **Jobs**. Select the job whose cost data you want to view. ##### Filters For the **Runs** tab, you can use the **Last** filter to view data from the past week, 14 days, or 30 days. For **Cost**, **Usage**, **Query run time**, and **Builds** tabs, you can customize the cost data you want to view by: * **Last**: 30 days, 60 days, 90 days, 6 months, or 1 year * **View**: Daily, Weekly, Monthly * **Assets**: All, Models, Tests ##### Visualization tabs * **Runs**: Displays the success rate and run duration in minutes for recent runs. You can select a time period with options for **Last week**, **Last 14 days**, and **Last 30 days**. * **Cost**: Shows the estimated build cost reduction when using dbt State or state-aware orchestration. * **Usage**: Shows the estimated warehouse usage consumed and the reduction in usage from dbt State or state-aware orchestration over the selected timeframe. The **Usage** tab represents generic usage for your warehouse. The specific unit depends on your data warehouse: * Snowflake: Credits * BigQuery: Slot hours or bytes scanned (currently combined into one generic usage number) * Databricks: Databricks Units (DBUs) * Amazon Redshift Serverless: Redshift Processing Unit hours (RPU-hours) * Amazon Redshift Provisioned: Node-hours * **Query run time**: Shows the estimated query execution time and the reduction in run duration from dbt State or state-aware orchestration. * **Builds**: Shows the number of assets built versus reused by dbt State or state-aware orchestration. ##### Table view For **Cost**, **Usage**, **Query run time**, and **Builds** tabs, you can access the table view by clicking **Show table**, which provides detailed optimization data such as models reused, usage reduction, and cost reduction. Use the **All** and **Jobs** buttons to switch between views: * **All**: Shows aggregated cost data across all jobs in the project. * **Jobs**: Shows cost data broken down by individual jobs. When viewing the table, you can export the data as a CSV file using the **Download** button. When **Jobs** is selected, the CSV export includes job-level data. --- ### Explore multiple projects dbt platform | Enterprise, Enterprise+ View all the projects and public models in your account (where public models are defined) and gain a better understanding of your cross-project resources and how they're used. On-demand learning If you enjoy video courses, check out our [dbt Catalog on-demand course](https://learn.getdbt.com/courses/dbt-catalog) and learn how to best explore your dbt project(s)! The resource-level lineage graph for a project displays cross-project relationships in the DAG. Nodes that represent another dbt project show a project icon to the left of the project name. From a project's **Overview** page, click **View lineage** in the upper right corner to open that project's resource-level lineage graph. In that graph: * View an upstream (parent) project to see the downstream (child) projects that depend on it. * Select a model to reveal its dependent projects in the lineage. * Click on an upstream (parent) project to view the other projects that reference it in the **Relationships** tab, showing the number of downstream (child) projects that depend on them. * This includes all projects listing the upstream one as a dependency in its `dependencies.yml` file, even without a direct `{{ ref() }}`. * Select a project node from a public model to open its detailed lineage graph if you have the [permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) to do so. Indirect dependencies When viewing a project's lineage, Catalog shows only *directly* [referenced](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md) public models. It doesn't show [indirect dependencies](https://docs.getdbt.com/faqs/Project_ref/indirectly-reference-upstream-model.md). If a referenced model in your project depends on another upstream public model, the second-level model won't appear in Catalog, however it will appear in the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) lineage view. [![View your cross-project lineage in a parent project and the other projects that reference it by clicking the 'Relationships' tab.](/img/docs/collaborate/dbt-explorer/cross-project-lineage-parent.png?v=2 "View your cross-project lineage in a parent project and the other projects that reference it by clicking the 'Relationships' tab.")](#)View your cross-project lineage in a parent project and the other projects that reference it by clicking the 'Relationships' tab. When viewing a downstream (child) project that imports and refs public models from upstream (parent) projects: * Public models will show up in the lineage graph and you can click on them to view the model details. * Clicking on a model opens a side panel containing general information about the model, such as the specific dbt project that produces that model, description, package, and more. * Double-clicking on a model from another project opens the resource-level lineage graph of the parent project, if you have the permissions to do so. [![View a downstream (child) project that imports and refs public models from the upstream (parent) project.](/img/docs/collaborate/dbt-explorer/cross-project-child.png?v=2 "View a downstream (child) project that imports and refs public models from the upstream (parent) project.")](#)View a downstream (child) project that imports and refs public models from the upstream (parent) project. #### Explore the project-level lineage graph For cross-project collaboration, you can interact with the DAG in all the same ways as described in [Explore your project's lineage](https://docs.getdbt.com/docs/explore/explore-projects.md#project-lineage). You can also interact with it at the project level and view the details. If you have permissions for a project in the account, you can view all public models used across the entire account. However, you can only view full public model details and private models if you have permissions for the specific project where those models are defined. ##### View account-level lineage To view all projects in your account as a lineage graph or list: 1. In **Catalog**, select your project from the sidebar. 2. Click **Account lineage** in the upper right corner. On the account-level lineage graph, each project appears as its own node with the project name and a count of public models it exposes. Arrows between nodes show how projects depend on one another. The top toolbar includes a search bar and **List view** and lineage view toggles. Zoom controls appear in the lower right corner. [![Account-level lineage graph showing cross-project dependencies between projects.](/img/docs/collaborate/dbt-explorer/account-level-lineage.gif?v=2 "Account-level lineage graph showing cross-project dependencies between projects.")](#)Account-level lineage graph showing cross-project dependencies between projects. From the account-level lineage graph, you can also: * Click **List view** to switch to a table of projects and public models. * Double-click a project node to open that project's resource-level lineage graph. ##### Explore a project's resource-level lineage From a project's **Overview** page, click **View lineage** to open the resource-level lineage graph for that project. In this view, you can see: * Sources, seeds, and models in the current project, with connectors between them. * Downstream projects that reference the project's public models, shown as separate project nodes connected by dashed lines. * A search bar, **Lenses**, a **Resource type** filter, a resource-type legend, and zoom controls. [![Resource-level lineage graph with a selected model and the Relationships tab.](/img/docs/collaborate/dbt-explorer/multi-project-overview.gif?v=2 "Resource-level lineage graph with a selected model and the Relationships tab.")](#)Resource-level lineage graph with a selected model and the Relationships tab. When you select a model, a details panel opens on the right: * **General** tab: the model description and metadata fields such as **Project** and **Relation**. * **Columns** tab: each column's name, data type, and description, plus column test results when tests are defined. A **Search for columns** field appears at the top of the tab. * **Relationships** tab: downstream projects that reference the model, listed under **Referenced by**. --- ### Explore your data dbt platform dbt provides a variety of tools for you to explore your data, models, and other resources. Many of the features you'd traditionally use your data warehouse services to explore are at your fingertips in your dbt account. [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/explore/cost-insights.md) ###### [Cost Insights](https://docs.getdbt.com/docs/explore/cost-insights.md) [Track warehouse compute costs and see realized savings from state-aware orchestration across your dbt projects and models.](https://docs.getdbt.com/docs/explore/cost-insights.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/explore/explore-projects.md) ###### [dbt Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) [Interact with dbt Catalog to understand, improve, and leverage your dbt projects.](https://docs.getdbt.com/docs/explore/explore-projects.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/explore/dbt-insights.md) ###### [dbt Insights](https://docs.getdbt.com/docs/explore/dbt-insights.md) [Query data and perform exploratory data analysis using dbt Insights.](https://docs.getdbt.com/docs/explore/dbt-insights.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) ###### [Documentation](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) [Document your dbt projects so stakeholders, engineers, and analysts can understand your resources and lineage from start to finish.](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) <br /> Some features are only available on [selected plans](https://www.getdbt.com/pricing/). #### Related docs * [dbt plans and pricing](https://www.getdbt.com/pricing/) * [Quickstart guides](https://docs.getdbt.com/docs/get-started-dbt.md) * [Reference material](https://docs.getdbt.com/reference/references-overview.md) --- ### External metadata ingestion Preview ### External metadata ingestion [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt platform | Enterprise, Enterprise+ With external metadata ingestion, you can connect directly to your data warehouse, giving you visibility into tables, views, and other resources that aren't defined in dbt with Catalog. External metadata ingestion support Currently, external metadata ingestion is supported for Snowflake only. External metadata credentials enable ingestion of metadata that exists *outside* your dbt runs like tables, views, or cost information; typically at a higher level than what dbt environments access. This is useful for enriching Catalog with warehouse-native insights (for example, Snowflake views or access patterns) and creating a unified discovery experience. These credentials are configured separately from dbt environment credentials and are scoped at the account level, not the project level. #### Prerequisites * Have a dbt account on the [Enterprise or Enterprise+](https://www.getdbt.com/pricing) plan. * You must be an [account admin with permission](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#account-admin) to edit connections. * The credentials must have [sufficient read-level access to fetch metadata](https://docs.getdbt.com/docs/explore/external-metadata-ingestion.md#configuration-instructions). * Have [**global navigation**](https://docs.getdbt.com/docs/explore/explore-projects.md#catalog-overview) enabled. * Use Snowflake as your data platform. * Support for additional adapters is coming soon! #### Configuration instructions ##### Enable external metadata ingestion 1. Click your account name at the bottom of the left-side menu and click **[Account settings](https://docs.getdbt.com/docs/platform/account-settings.md)**. 2. Under Account information, go to **Settings** and click **Edit** at the top right corner of the page. 3. Select the **Ingest external metadata in dbt Catalog (formerly dbt Explorer)** option (if not already enabled). ##### Configure the warehouse connection 1. Go to **Account settings**. 2. Click **Connections** from the left-hand side panel. 3. Select an existing connection or create a [**New connection**](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-snowflake.md) where you want to ingest metadata from. 4. Scroll to the **Platform metadata credentials** section. Depending on your current configuration, you may see one of the following: * If platform metadata ingestion is not yet configured, the credentials form is immediately in edit mode. * If you previously canceled and see **Add credentials**, click the button to reopen the form. * If you have multiple connections that reference the same account identifier, you will only be prompted to configure platform metadata ingestion for one of them. Other connections using the same account identifier will display a message indicating that platform metadata ingestion is already configured. 5. Enter the necessary credentials. These should have warehouse-level visibility across relevant databases and schemas. 6. Select the **External metadata ingestion** option. * This allows metadata from this connection to populate the Catalog. * *Optional*: Enable additional features such as **Cost insights** in the **Features** section under **Platform metadata credentials**. 7. Under **Catalog filters**, apply filters to restrict which metadata is ingested: * You can filter by **database**, **schema**, **table**, or **view**. * **Note:** To include all databases or schemas, enter `.*` in the **Allow** field. * It is strongly recommended to filter by certain schemas. See [Important considerations](https://docs.getdbt.com/docs/explore/external-metadata-ingestion.md#important-considerations) for more information. * These fields accept CSV-formatted regular expressions: * Example: `DIM` matches `DIM_ORDERS` and `DIMENSION_TABLE` (basic "contains" match). * Wildcards are supported. For example: `DIM*` matches `DIM_ORDERS` and `DIM_PRODUCTS`. #### Required credentials This section sets up the foundational access for dbt in Snowflake. It creates a role (`dbt_metadata_role`) with minimal permissions and a user (`dbt_metadata_user`) dedicated to dbt’s metadata access. This ensures a clear, controlled separation of access, so dbt can read metadata without broader permissions. This setup ensures dbt can read metadata for profiling, documentation, and lineage, without the ability to modify data or manage resources. 1. Create role: ```sql CREATE OR REPLACE ROLE dbt_metadata_role; ``` 2. Grant access to a warehouse to run queries to view metadata: ```sql GRANT USAGE ON WAREHOUSE "<your-warehouse>" TO ROLE dbt_metadata_role; ``` If your warehouse needs to be restarted for metadata ingestion (doesn't have auto-resume enabled), you may need to grant `OPERATE` permissions to the role as well. If you do not already have a user, create a dbt-specific user for metadata access. Replace `<your-password>` with a strong password and `<your-warehouse>` with the warehouse name used above: ```sql CREATE USER dbt_metadata_user DISPLAY_NAME = 'dbt Metadata Integration' PASSWORD = '<your-password>' DEFAULT_ROLE = dbt_metadata_role TYPE = 'LEGACY_SERVICE' DEFAULT_WAREHOUSE = '<your-warehouse>'; ``` 3. Grant the role to the user: ```sql GRANT ROLE dbt_metadata_role TO USER dbt_metadata_user; ``` Note: Use read-only service accounts for least privilege and better auditing. #### Assign metadata access privileges This section outlines the minimum necessary privileges to read metadata from each required Snowflake database. It provides access to schemas, tables, views, and lineage information, ensuring dbt can profile and document your data while preventing any modifications. Replace `your-database` with the name of a Snowflake database to grant metadata access. Repeat this block for each relevant database: ```sql SET db_var = '"<your-database>"'; -- Grant access to view the database and its schemas GRANT USAGE ON DATABASE IDENTIFIER($db_var) TO ROLE dbt_metadata_role; GRANT USAGE ON ALL SCHEMAS IN DATABASE IDENTIFIER($db_var) TO ROLE dbt_metadata_role; GRANT USAGE ON FUTURE SCHEMAS IN DATABASE IDENTIFIER($db_var) TO ROLE dbt_metadata_role; -- Grant REFERENCES to enable lineage and dependency analysis GRANT REFERENCES ON ALL TABLES IN DATABASE IDENTIFIER($db_var) TO ROLE dbt_metadata_role; GRANT REFERENCES ON FUTURE TABLES IN DATABASE IDENTIFIER($db_var) TO ROLE dbt_metadata_role; GRANT REFERENCES ON ALL EXTERNAL TABLES IN DATABASE IDENTIFIER($db_var) TO ROLE dbt_metadata_role; GRANT REFERENCES ON FUTURE EXTERNAL TABLES IN DATABASE IDENTIFIER($db_var) TO ROLE dbt_metadata_role; GRANT REFERENCES ON ALL VIEWS IN DATABASE IDENTIFIER($db_var) TO ROLE dbt_metadata_role; GRANT REFERENCES ON FUTURE VIEWS IN DATABASE IDENTIFIER($db_var) TO ROLE dbt_metadata_role; -- Recommended grant SELECT for privileges to enable metadata introspection and profiling GRANT SELECT ON ALL TABLES IN DATABASE IDENTIFIER($db_var) TO ROLE dbt_metadata_role; GRANT SELECT ON FUTURE TABLES IN DATABASE IDENTIFIER($db_var) TO ROLE dbt_metadata_role; GRANT SELECT ON ALL EXTERNAL TABLES IN DATABASE IDENTIFIER($db_var) TO ROLE dbt_metadata_role; GRANT SELECT ON FUTURE EXTERNAL TABLES IN DATABASE IDENTIFIER($db_var) TO ROLE dbt_metadata_role; GRANT SELECT ON ALL VIEWS IN DATABASE IDENTIFIER($db_var) TO ROLE dbt_metadata_role; GRANT SELECT ON FUTURE VIEWS IN DATABASE IDENTIFIER($db_var) TO ROLE dbt_metadata_role; GRANT SELECT ON ALL DYNAMIC TABLES IN DATABASE IDENTIFIER($db_var) TO ROLE dbt_metadata_role; GRANT SELECT ON FUTURE DYNAMIC TABLES IN DATABASE IDENTIFIER($db_var) TO ROLE dbt_metadata_role; -- Grant MONITOR on dynamic tables (e.g., for freshness or status checks) GRANT MONITOR ON ALL DYNAMIC TABLES IN DATABASE IDENTIFIER($db_var) TO ROLE dbt_metadata_role; GRANT MONITOR ON FUTURE DYNAMIC TABLES IN DATABASE IDENTIFIER($db_var) TO ROLE dbt_metadata_role; ``` #### Grant access to Snowflake metadata This step grants the dbt role (`dbt_metadata_role`) access to Snowflake’s system-level database, enabling it to read usage statistics, query histories, and lineage information required for comprehensive metadata insights. Grant privileges to read usage stats and lineage from Snowflake’s system-level database: ```sql GRANT IMPORTED PRIVILEGES ON DATABASE SNOWFLAKE TO ROLE dbt_metadata_role; ``` #### Important considerations The following are best practices for external metadata ingestion, designed to ensure consistent, reliable, and scalable integration of metadata from third-party systems. * Catalog unifies the shared resources between dbt and Snowflake. For example, if there’s a Snowflake table that represents a dbt model, these are represented as a single resource in Catalog. In order for proper unification to occur, the same connection must be used by both the [production environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md#set-as-production-environment) and the external metadata ingestion credential. * Avoid duplicates: Use one metadata connection per platform if possible (for example, one for Snowflake, one for BigQuery). * Having multiple connections pointing to the same warehouse can cause duplicate metadata. * Align with dbt environment: To unify asset lineage and metadata, ensure the same warehouse connection is used by both the dbt environment and the external metadata ingestion. * Use filters to limit ingestion to relevant assets: * For example: restrict to production schemas only, or ignore transient/temp schemas. External metadata ingestion runs daily at 5 PM UTC, and also runs immediately each time you update and save credentials. --- ### Fill null values for metrics Understanding and implementing strategies to fill null values in metrics is key for accurate analytics. This guide explains `fill_nulls_with` and `join_to_timespine` to ensure data completeness, helping end users make more informed decisions and enhancing your dbt workflows. ##### About null values You can use `fill_nulls_with` to replace null values in metrics with a value like zero (or your chosen integer). This ensures every data row shows a numeric value. This guide explains how to ensure there are no null values in your metrics: * Use `fill_nulls_with` for `simple`, `cumulative`, and `conversion` metrics * Use `join_to_timespine` and `fill_nulls_with` together for derived and ratio metrics to avoid null values appearing. ##### Fill null values for simple metrics For example, if you'd like to handle days with site visits but no leads, you can use `fill_nulls_with` to set the value for leads to zero on days when there are no conversions. Let's say you have three metrics: * `website_visits` and `leads` * and a derived metric called `leads_to_website_visit` that calculates the ratio of leads to site visits. (Applies to dbt v1.12 and later) On the days when there are no conversions, you can set the value for leads to zero by adding the `fill_nulls_with` parameter to the leads metric: models/website\_visits.yml ```yaml # Define simple metrics within a semantic model models: - name: website_visits_model semantic_model: enabled: true # ... other configs ... metrics: - name: website_visits type: simple agg: count - name: leads type: simple agg: count fill_nulls_with: 0 # This fills null values with zero - name: leads_to_website_visit type: derived expr: leads/website_visits input_metrics: - name: leads - name: website_visits ``` The `website_visits` and `leads` metrics have the following data: | metric\_time | website\_visits | | ------------ | --------------- | | 2024-01-01 | 50 | | 2024-01-02 | 37 | | 2024-01-03 | 79 | | metric\_time | leads | | ------------ | ----- | | 2024-01-01 | 5 | | 2024-01-03 | 8 | * Note that there is no data for `2024-01-02` in the `leads` metric. Although there are no days without visits, there are days without leads. After applying `fill_nulls_with: 0` to the `leads` metric, querying these metrics together shows zero for leads on days with no conversions: | metric\_time | website\_visits | leads | | ------------ | --------------- | ----- | | 2024-01-01 | 50 | 5 | | 2024-01-02 | 37 | 0 | | 2024-01-03 | 79 | 8 | ##### Use join\_to\_timespine for derived and ratio metrics (Applies to dbt v1.12 and later) To ensure you have a complete set of data for every and daily coverage for metrics calculated from other metrics, you can use `join_to_timespine` to fill null values for `derived` and `ratio` metrics. These metrics are built from other metrics (other calculations), not direct aggregations, requiring MetricFlow to have an extra subquery layer to render the metric. The subquery nesting is as follows: * For `derived` and `ratio` metrics, there are three levels of subquery nesting — derived or ratio metric → input metrics → simple metrics with aggregations. * For `simple` and `cumulative` metrics, there are only two levels of subquery nesting — simple or cumulative metric → aggregation. Because `coalesce` isn't applied to the third, subquery layer for `derived` or `ratio` metrics, this means you could still have nulls in the final result set. Note you can use `join_to_timespine` with simple metrics as well if you want to include a row for every date, even if there is no data. ##### Fill null values for derived and ratio metrics To fill null values for derived and ratio metrics, you can link them with a time spine to ensure daily data coverage. As mentioned in [the previous section](#use-join_to_timespine-for-derived-and-ratio-metrics), this is because `derived` and `ratio` metrics take *metrics* as inputs. For example, the following structure leaves nulls in the final results (`leads_to_website_visit` column) because `COALESCE` isn't applied at the third outer rendering layer for the final metric calculation in `derived` metrics: | metric\_time | website\_visits | leads | leads\_to\_website\_visit | | ------------ | --------------- | ----- | ------------------------- | | 2024-01-01 | 50 | 5 | .1 | | 2024-01-02 | 37 | 0 | null | | 2024-01-03 | 79 | 8 | .1 | To display a zero value for `leads_to_website_visit` for `2024-01-02`, you would join the `leads` metric to a time spine model to ensure a value for each day. You can do this by adding `join_to_timespine` to the (Applies to dbt v1.12 and later) simple metric in the `leads` metric configuration: (Applies to dbt v1.12 and later) models/leads.yml ```yaml models: - name: leads_model semantic_model: enabled: true # ... other configs ... metrics: - name: leads type: simple agg: count fill_nulls_with: 0 join_to_timespine: true ``` Once you do this, if you query the `leads` metric after the timespine join, there will be a record for each day and any null values will get filled with zero. | metric\_time | leads | leads\_to\_website\_visit | | ------------ | ----- | ------------------------- | | 2024-01-01 | 5 | .1 | | 2024-01-02 | 0 | 0 | | 2024-01-03 | 8 | .1 | Now, if you combine the metrics in a `derived` metric, there will be a zero value for `leads_to_website_visit` on `2024-01-02` and the final result set will not have any null values. #### FAQs  How to handle null values in derived metrics defined on top of multiple tables For additional examples and discussion on how to handle null values in derived metrics that use data from multiple tables, check out [MetricFlow issue #1031](https://github.com/dbt-labs/metricflow/issues/1031). --- ### Frequently asked questions #### [🗃️ Accounts](https://docs.getdbt.com/category/accounts.md) [13 items](https://docs.getdbt.com/category/accounts.md) --- ### Global navigation Preview ### Global navigation [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt platform | Starter, Enterprise, Enterprise+ Search, explore, and analyze data assets across all your dbt projects and connected metadata sources. Discover cross-project lineage, data discovery, and unified analytics governance. **Plan availability** Global navigation search varies depending on your [dbt platform](https://www.getdbt.com/pricing) plan: * Enterprise plans — Catalog lets you search across all [dbt resources](https://docs.getdbt.com/docs/build/projects.md) (models, seeds, snapshots, sources, exposures, and more) in your account. * Starter plans (single project) — Use global navigation to search and navigate resources within your project #### About Global navigation Global navigation in Catalog lets you search, explore, and analyze data assets across all your dbt projects and connected metadata sources—giving you a unified, account-wide view of your analytics ecosystem. With global navigation, you can: * Search data assets — expand your search by including dbt resources (models, seeds, snapshots, sources, exposures, and more) across your entire account. This broadens the results returned and gives you greater insight into all the assets across your dbt projects. * Explore lineage — explore an interactive map of data relationships across all your dbt projects. It lets you: * View upstream/downstream dependencies for models, sources, and more. * Drill into project and column-level lineage, including multi-project (Mesh) links. * Filter with "lineage lenses" by resource type, materialization, layer, or run status. * Troubleshoot data issues by tracing root causes and downstream impacts. * Optimize pipelines by spotting slow, failing, or unused parts of your DAG. * See recommendations — global navigation offers a project-wide snapshot of dbt health, highlighting actionable tips to enhance your analytics engineering. These insights are automatically generated using dbt metadata and best practices from the project evaluator ruleset. * View model query history — see how often each dbt model is queried in your warehouse, helping you: * Track real usage via successful `SELECT`s (excluding builds/tests) * Identify most/least used models for optimization or deprecation * Guide investment and maintenance with data-driven insights * Track downstream exposures — monitor how your dbt models and sources are used by BI tools, apps, ML models, and reports across all connected projects --- ### Google Sheets dbt platform | Starter, Enterprise, Enterprise+ The Semantic Layer offers a seamless integration with Google Sheets through a custom menu. This add-on allows you to build Semantic Layer queries and return data on your metrics directly within Google Sheets #### Prerequisites * You have [configured the Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md) and are using dbt v1.6 or higher. If you haven't created metrics yet, start with the [Quickstart for the Semantic Layer](https://docs.getdbt.com/guides/sl-qs.md). * You need a Google account with access to Google Sheets and the ability to install Google add-ons. * You have a [dbt Environment ID](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md#set-up-dbt-semantic-layer). * You have a [service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) or a [personal access token](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) to authenticate with from a dbt account. * You must have a dbt Starter or Enterprise-tier [account](https://www.getdbt.com/pricing). Suitable for both Multi-tenant and Single-tenant deployment. If you're using [IP restrictions](https://docs.getdbt.com/docs/platform/secure/ip-restrictions.md), ensure you've added [Google’s IP addresses](https://www.gstatic.com/ipranges/goog.txt) to your IP allowlist. Otherwise, the Google Sheets connection will fail. 📹 Learn about the dbt Semantic Layer with on-demand video courses! Explore our [dbt Semantic Layer on-demand course](https://learn.getdbt.com/courses/semantic-layer) to learn how to define and query metrics in your dbt project. Additionally, dive into mini-courses for querying the dbt Semantic Layer in your favorite tools: [Tableau](https://courses.getdbt.com/courses/tableau-querying-the-semantic-layer), [Excel](https://learn.getdbt.com/courses/querying-the-semantic-layer-with-excel), [Hex](https://courses.getdbt.com/courses/hex-querying-the-semantic-layer), and [Mode](https://courses.getdbt.com/courses/mode-querying-the-semantic-layer). #### Installing the add-on 1. Navigate to the [Semantic Layer for Sheets App](https://gsuite.google.com/marketplace/app/foo/392263010968) to install the add-on. You can also find it in Google Sheets by going to [**Extensions -> Add-on -> Get add-ons**](https://support.google.com/docs/answer/2942256?hl=en\&co=GENIE.Platform%3DDesktop\&oco=0#zippy=%2Cinstall-add-ons%2Cinstall-an-add-on) and searching for it there. 2. After installing, open the **Extensions** menu and select **Semantic Layer for Sheets**. This will open a custom menu on the right-hand side of your screen. 3. [Find your](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md#set-up-dbt-semantic-layer) **Host** and **Environment ID** in dbt. * Navigate to **Account Settings** and select **Projects** on the left sidebar. * Select your project and then navigate to the **Semantic Layer** settings. You'll need this to authenticate in Google Sheets in the following step. * You can generate your service token by clicking **Generate service token** within the Semantic Layer configuration page or navigating to **API tokens** in dbt. Alternatively, you can also create a personal access token by going to **API tokens** > **Personal tokens**. [![Access your Environment ID, Host, and URLs in your dbt Semantic Layer settings. Generate a service token in the Semantic Layer settings or API tokens settings](/img/docs/dbt-platform/semantic-layer/sl-and-gsheets.png?v=2 "Access your Environment ID, Host, and URLs in your dbt Semantic Layer settings. Generate a service token in the Semantic Layer settings or API tokens settings")](#)Access your Environment ID, Host, and URLs in your dbt Semantic Layer settings. Generate a service token in the Semantic Layer settings or API tokens settings 4. In Google Sheets, authenticate with your Host, dbt Environment ID, and service or personal token. 5. Start querying your metrics using the **Query Builder**. For more info on the menu functions, refer to [Query Builder functions](#query-builder-functions). To cancel a query while running, press the "Cancel" button. When querying your data with Google Sheets: * It returns the data to the cell you clicked on. * The custom menu operation has a timeout limit of six (6) minutes. * If you're using this extension, make sure you're signed into Chrome with the same Google profile you used to set up the Add-On. Log in with one Google profile at a time as using multiple Google profiles at once might cause issues. * Note that only standard granularities are currently available, custom time granularities aren't currently supported for this integration. #### Query Builder functions The Google Sheets **Query Builder** custom menu has the following capabilities: | Menu items | Description | | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Metrics | Search and select metrics. | | Group By | Search and select dimensions or entities to group by. Dimensions are grouped by the entity of the semantic model they come from. You may choose dimensions on their own without metrics. | | Time Range | Quickly select time ranges to look at the data, which applies to the main time series for the metrics (metric time), or do more advanced filter using the "Custom" selection. | | Where | Filter your data. This includes categorical and time filters. | | Order By | Return your data order. | | Limit | Set a limit for the rows of your output. | Note: Click the **info** button next to any metric or dimension to see its defined description from your dbt project. ###### Modifying time granularity When you select time dimensions in the **Group By** menu, you'll see a list of available time granularities. The lowest granularity is selected by default. Metric time is the default time dimension for grouping your metrics. info Note: [Custom time granularities](https://docs.getdbt.com/docs/build/metricflow-time-spine.md#add-custom-granularities) (like fiscal year) aren't currently supported or accessible in this integration. Only [standard granularities](https://docs.getdbt.com/docs/build/dimensions.md?dimension=time_gran#time) (like day, week, month, and so on) are available. If you'd like to access custom granularities, consider using the [Semantic Layer APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md). ###### Filtering data To use the filter functionality, choose the [dimension](https://docs.getdbt.com/docs/build/dimensions.md) you want to filter by and select the operation you want to filter on. * For categorical dimensions, you can type a value into search or select from a populated list. * For entities, you must type the value you are looking for as we do not load all of them given the large number of values. * Continue adding additional filters as needed with AND and OR. * For time dimensions, you can use the time range selector to filter on presets or custom options. The time range selector applies only to the primary time dimension (`metric_time`). For all other time dimensions that aren't `metric_time`, you can use the "Where" option to apply filters. ###### Other settings If you would like to just query the data values without the headers, you can optionally select the **Exclude column names** box. To return your results and keep any previously selected data below it intact, un-select the **Clear trailing rows** box. By default, we'll clear all trailing rows if there's stale data. [![Run a query in the Query Builder. Use the arrow next to the Query button to select additional settings.](/img/docs/dbt-platform/semantic-layer/query-builder.png?v=2 "Run a query in the Query Builder. Use the arrow next to the Query button to select additional settings.")](#)Run a query in the Query Builder. Use the arrow next to the Query button to select additional settings. #### Using saved selections Saved selections allow you to save the inputs you've created in the Google Sheets **Query Builder** and easily access them again so you don't have to continuously build common queries from scratch. To create a saved selection: 1. Run a query in the **Query Builder**. 2. Save the selection by selecting the arrow next to the **Query** button and then select **Query & Save Selection**. 3. The application saves these selections, allowing you to view and edit them from the hamburger menu under **Saved Selections**. You can also make these selections private or public. Public selections mean your inputs are available in the menu to everyone on the sheet. Private selections mean your inputs are only visible to you. Note that anyone added to the sheet can still see the data from these private selections, but they won't be able to interact with the selection in the menu or benefit from the automatic refresh. ##### Refreshing selections Set your saved selections to automatically refresh every time you load the addon. You can do this by selecting **Refresh on Load** when creating the saved selection. When you access the addon and have saved selections that should refresh, you'll see "Loading..." in the cells that are refreshing. Public saved selections will refresh for anyone who edits the sheet. What's the difference between saved selections and saved queries? * Saved selections are saved components that you can create only when using the application. * Saved queries, explained in the next section, are code-defined sections of data you create in your dbt project that you can easily access and use for building selections. You can also use the results from a saved query to create a saved selection. #### Using saved queries Access [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md), powered by MetricFlow, in Google Sheets to quickly get results from pre-defined sets of data. To access the saved queries in Google Sheets: 1. Open the hamburger menu in Google Sheets. 2. Navigate to **Saved Queries** to access the ones available to you. 3. You can also select **Build Selection**, which allows you to explore the existing query. This won't change the original query defined in the code. * If you use a `WHERE` filter in a saved query, Google Sheets displays the advanced syntax for this filter. **Limited use policy disclosure** The Semantic Layer for Sheet's use and transfer to any other app of information received from Google APIs will adhere to [Google API Services User Data Policy](https://developers.google.com/terms/api-services-user-data-policy), including the Limited Use requirements. #### FAQs I'm receiving an \`Failed ALPN\` error when trying to connect to the dbt Semantic Layer. If you're receiving a `Failed ALPN` error when trying to connect the dbt Semantic Layer with the various [data integration tools](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) (such as Tableau, DBeaver, Datagrip, ADBC, or JDBC), it typically happens when connecting from a computer behind a corporate VPN or Proxy (like Zscaler or Check Point). The root cause is typically the proxy interfering with the TLS handshake as the Semantic Layer uses gRPC/HTTP2 for connectivity. To resolve this: * If your proxy supports gRPC/HTTP2 but isn't configured to allow ALPN, adjust its settings accordingly to allow ALPN. Or create an exception for the dbt domain. * If your proxy does not support gRPC/HTTP2, add an SSL interception exception for the dbt domain in your proxy settings This should help in successfully establishing the connection without the Failed ALPN error. --- ### Headless mode Beta ### Headless mode [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Local development dbt Wizard can run without the interactive TUI — useful for scripts, CI pipelines, pre-commit hooks, and any workflow where you want a one-shot result without human-in-the-loop approval. See it in action and share your feedback Want to see dbt Wizard in action? Check out the [demo video](https://www.youtube.com/watch?v=-lIzh1xQWMA). We'd love to hear how dbt Wizard is working for you. Share your feedback by either running the `/feedback` slash command in your interactive terminal session or by going to the [#dbt-wizard](https://getdbt.slack.com/archives/C0B6KLW6T26) channel in the [dbt Community Slack](https://docs.getdbt.com/community/join?version=2.0). Thanks so much for your help in improving dbt Wizard and dbt data development! #### `exec` — one-shot prompts Run a single prompt and exit: ```bash wizard exec "list all models with no tests" ``` Pipe input via stdin: ```bash echo "which sources have stale freshness?" | wizard exec - ``` Use `exec` in CI to gate on quality checks: ```bash # Check test coverage before merging wizard exec "are there any models in models/marts/ with no tests?" ``` ##### JSON output For downstream processing, emit a structured JSON event stream: ```bash wizard exec --json "summarize test coverage by schema" > coverage.json ``` With a JSON Schema to constrain the response shape: ```bash wizard exec \ --json \ --output-schema ./schemas/coverage-response.json \ "summarize test coverage by schema" ``` Write the final message to a file: ```bash wizard exec \ --output-last-message ./review-output.md \ "review the changes in this branch for correctness" ``` #### `review` — automated code review Review uncommitted changes: ```bash wizard review --uncommitted ``` Review a branch diff in CI: ```bash wizard review --base main ``` Review a specific commit: ```bash wizard review --commit abc1234 ``` ##### Example: GitHub Actions code review ```yaml - name: dbt Wizard review run: | wizard review \ --base ${{ github.base_ref }} > review.md env: OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} ``` #### Permissions in headless mode In headless `exec` mode, Wizard runs without interactive approval prompts. Pre-grant the sandbox permissions you need: ```bash # Read-only analysis (default — safe for CI) wizard exec "list models with no documentation" # Allow file writes inside the workspace wizard exec -s workspace-write "add not_null tests to all primary keys in staging" # Allow shell commands like dbt compile wizard exec -s workspace-write "compile and validate fct_orders" ``` For read-only analysis tasks (coverage checks, impact queries, documentation gaps), the default permissions are sufficient. For tasks that write files or run dbt commands, pass the appropriate flags explicitly. #### Related docs * [dbt Wizard command reference](https://docs.getdbt.com/docs/dbt-ai/wizard-cli-reference.md) * [Use cases and examples](https://docs.getdbt.com/docs/dbt-ai/wizard-use-cases.md) --- ### Hooks and operations #### Related documentation * [pre-hook & post-hook](https://docs.getdbt.com/reference/resource-configs/pre-hook-post-hook.md) * [on-run-start & on-run-end](https://docs.getdbt.com/reference/project-configs/on-run-start-on-run-end.md) * [`run-operation` command](https://docs.getdbt.com/reference/commands/run-operation.md) ##### Assumed knowledge * [Project configurations](https://docs.getdbt.com/reference/dbt_project.yml.md) * [Model configurations](https://docs.getdbt.com/reference/model-configs.md) * [Macros](https://docs.getdbt.com/docs/build/jinja-macros.md#macros) #### Getting started with hooks and operations Effective database administration sometimes requires additional SQL statements to be run, for example: * Creating UDFs * Managing row- or column-level permissions * Vacuuming tables on Redshift * Creating partitions in Redshift Spectrum external tables * Resuming/pausing/resizing warehouses in Snowflake * Refreshing a pipe in Snowflake * Create a share on Snowflake * Cloning a database on Snowflake dbt provides hooks and operations so you can version control and execute these statements as part of your dbt project. #### About hooks Hooks are snippets of SQL that are executed at different times: * `pre-hook`: executed *before* a model, seed or snapshot is built. * `post-hook`: executed *after* a model, seed or snapshot is built. * `on-run-start`: executed at the *start* of `dbt build`, `dbt compile`, `dbt docs generate`, `dbt run`, `dbt seed`, `dbt snapshot`, or `dbt test`. * `on-run-end`: executed at the *end* of `dbt build`, `dbt compile`, `dbt docs generate`, `dbt run`, `dbt seed`, `dbt snapshot`, or `dbt test`. Hooks are a more-advanced capability that enable you to run custom SQL, and leverage database-specific actions, beyond what dbt makes available out-of-the-box with standard materializations and configurations. If (and only if) you can't leverage the [`grants` resource-config](https://docs.getdbt.com/reference/resource-configs/grants.md), you can use `post-hook` to perform more advanced workflows: * Need to apply `grants` in a more complex way, which the dbt `grants` config doesn't (yet) support. * Need to perform post-processing that dbt does not support out-of-the-box. For example, `analyze table`, `alter table set property`, `alter table ... add row access policy`, etc. ##### Examples using hooks You can use hooks to trigger actions at certain times when running an operation or building a model, seed, or snapshot. For more information about when hooks can be triggered, see reference sections for [`on-run-start` and `on-run-end` hooks](https://docs.getdbt.com/reference/project-configs/on-run-start-on-run-end.md) and [`pre-hook`s and `post-hook`s](https://docs.getdbt.com/reference/resource-configs/pre-hook-post-hook.md). You can use hooks to provide database-specific functionality not available out-of-the-box with dbt. For example, you can use a `config` block to run an `ALTER TABLE` statement right after building an individual model using a `post-hook`: models/\<model\_name>.sql ```sql {{ config( post_hook=[ "alter table {{ this }} ..." ] ) }} ``` ##### Calling a macro in a hook You can also use a [macro](https://docs.getdbt.com/docs/build/jinja-macros.md#macros) to bundle up hook logic. Check out some of the examples in the reference sections for [on-run-start and on-run-end hooks](https://docs.getdbt.com/reference/project-configs/on-run-start-on-run-end.md) and [pre- and post-hooks](https://docs.getdbt.com/reference/resource-configs/pre-hook-post-hook.md). models/\<model\_name>.sql ```sql {{ config( pre_hook=[ "{{ some_macro() }}" ] ) }} ``` models/properties.yml ```yaml models: - name: <model_name> config: pre_hook: - "{{ some_macro() }}" ``` dbt\_project.yml ```yaml models: <project_name>: +pre-hook: - "{{ some_macro() }}" ``` #### About operations Operations are [macros](https://docs.getdbt.com/docs/build/jinja-macros.md#macros) that you can run using the [`run-operation`](https://docs.getdbt.com/reference/commands/run-operation.md) command. As such, operations aren't actually a separate resource in your dbt project — they are just a convenient way to invoke a macro without needing to run a model. Explicitly execute the SQL in an operation Unlike hooks, you need to explicitly execute a query within a macro, by using either a [statement block](https://docs.getdbt.com/reference/dbt-jinja-functions/statement-blocks.md) or a helper macro like the [run\_query](https://docs.getdbt.com/reference/dbt-jinja-functions/run_query.md) macro. Otherwise, dbt will return the query as a string without executing it. This macro performs a similar action as the above hooks: macros/grant\_select.sql ```sql {% macro grant_select(role) %} {% set sql %} grant usage on schema {{ target.schema }} to role {{ role }}; grant select on all tables in schema {{ target.schema }} to role {{ role }}; grant select on all views in schema {{ target.schema }} to role {{ role }}; {% endset %} {% do run_query(sql) %} {% do log("Privileges granted", info=True) %} {% endmacro %} ``` To invoke this macro as an operation, execute `dbt run-operation grant_select --args '{role: reporter}'`. ```text $ dbt run-operation grant_select --args '{role: reporter}' Running with dbt=1.6.0 Privileges granted ``` Full usage docs for the `run-operation` command can be found [here](https://docs.getdbt.com/reference/commands/run-operation.md). #### Additional examples These examples from the community highlight some of the use-cases for hooks and operations! * [In-depth discussion of granting privileges using hooks and operations, for dbt v1 versions prior to 1.2](https://discourse.getdbt.com/t/the-exact-grant-statements-we-use-in-a-dbt-project/430) * [Staging external tables](https://github.com/dbt-labs/dbt-external-tables) * [Performing a zero copy clone on Snowflake to reset a dev environment](https://discourse.getdbt.com/t/creating-a-dev-environment-quickly-on-snowflake/1151/2) * [Running `vacuum` and `analyze` on a Redshift warehouse](https://github.com/dbt-labs/redshift/tree/0.2.3/#redshift_maintenance_operation-source) * [Creating a Snowflake share](https://discourse.getdbt.com/t/how-drizly-is-improving-collaboration-with-external-partners-using-dbt-snowflake-shares/1110) * [Unloading files to S3 on Redshift](https://github.com/dbt-labs/redshift/tree/0.2.3/#unload_table-source) * [Creating audit events for model timing](https://github.com/dbt-labs/dbt-event-logging) * [Creating UDFs](https://discourse.getdbt.com/t/using-dbt-to-manage-user-defined-functions/18) --- ### How dbt Wizard works dbt Wizard helps teams develop, troubleshoot, harden, and ship trusted dbt projects faster and with less risk. Built for governed data development in dbt, dbt Wizard understands your project, routes to the right dbt tools, and validates work with awareness of warehouse operations. Use it to investigate failed runs, debug models, assess impact, make changes, and ship trusted data work in one place. Most of how dbt Wizard works is the same in the [dbt platform](https://docs.getdbt.com/docs/platform/wizard-platform.md) and in the [terminal CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-cli.md). The following sections explain shared behavior first, then call out what differs in each environment. #### Native metadata engine dbt Wizard ships with a metadata engine — a pre-built, structured index of your entire project that's ready before your first prompt. Think of it like a map of your whole city: dbt Wizard knows how everything connects before it starts, rather than walking every street to figure out the layout. That index gives dbt Wizard four capabilities that aren't possible from file-reading alone: | Capability | Description | | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Impact analysis | When you ask "what breaks if I change this column?", dbt Wizard returns the exact set of downstream models, metrics, tests, and exposures affected — instantly, from the index. Column-level lineage is tracked, so dbt Wizard knows not just which models reference a table, but which ones reference that specific column. | | Health checks | dbt Wizard knows which models have tests, which don't, which have stale data, which have failing contracts, and which had recent run failures — as a structured dataset. You can ask "what's unhealthy in this part of my DAG?" and get a precise answer without running anything. | | Data profiling | dbt Wizard can profile your data — row counts, column distributions, null rates — and use that context when deciding how to build or refactor a model. It can reason about your data without materializing models or running expensive queries. | | Validation planning | dbt Wizard uses project metadata to identify affected resources and choose relevant checks. In the CLI, you control the depth of structured validation before commands run. | dbt Wizard builds and updates this index from dbt artifacts. In the dbt platform, project state comes from your connected development environment. In the terminal, run `dbt parse`, `dbt compile`, or `dbt build` before a session so dbt Wizard has your latest local project state. dbt version shown in the status panel The dbt version dbt Wizard displays comes from your project's manifest (`target/manifest.json`), not the `dbt` executable on your `PATH`. If you generated the manifest with a different binary, dbt Wizard reports that version until you recompile. Run `dbt compile` (or `dbt parse`/`dbt build`) with your intended dbt to refresh the manifest and the displayed version. #### Validation mechanics Validation can combine static checks, dbt commands, development builds, downstream impact analysis, and development-to-production comparisons. The checks that run depend on the surface, available tools, project state, permissions, and the validation depth you approve. In dbt Wizard CLI, choose light, medium, heavy, or skipped validation. Medium validation is the default: * Light validation focuses on syntax, linting where supported, tests, and code review without materializing the changed models. * Medium validation adds development materialization and downstream checks. * Heavy validation adds explicit expectations and development-to-production comparisons when the required relations are available. dbt Wizard reports failures and checks it couldn't complete. A passing check doesn't remove the need to review business logic, and a skipped check should remain visible in your review. For a complete procedure and the approval points for warehouse commands, refer to [Validate dbt changes with dbt Wizard](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-3-validate-changes.md). #### Tools and capabilities dbt Wizard takes action through a defined set of tools — from reading files to running dbt commands — so you can see exactly what it's doing and why. | Tool | Purpose | Where available | | --------------- | ------------------------------------------------------------ | ----------------------------------------------------------------------- | | File read/write | Read files and propose edits as diffs | Platform and CLI | | Project queries | Query lineage, tests, metadata, metrics, and run results | Platform and CLI | | dbt commands | Run commands like `dbt compile`, `dbt build`, and `dbt test` | Platform and CLI | | Bash | Execute shell commands in your project directory | CLI only | | Web search | Look up dbt docs and troubleshooting information | Platform and CLI | | MCP | Access connected MCP servers | CLI (add servers); platform includes built-in docs and platform context | dbt Wizard never runs destructive commands (such as `dbt build --full-refresh`, `dbt run --full-refresh`, or `git reset --hard`) without approval. #### Skills and memories dbt Wizard supports reusable skills and memories that help it apply your team's conventions across sessions. | Feature | Description | | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Skills | Standardize repeatable workflows and best practices. For example, a skill can tell dbt Wizard how your team structures staging models, writes tests, or documents columns. | | Memories | Store project-specific context that should carry between sessions. For example, dbt Wizard can remember that your project uses `customer_id` as the standard customer key or that finance models require extra review. | dbt Wizard automatically loads skills from your project and local directories in both the platform and the CLI. In the platform, start a new chat after adding or changing skills so they are picked up. Refer to the [Skills](https://docs.getdbt.com/docs/dbt-ai/wizard-skills.md) page for more details. #### In the dbt platform Use dbt Wizard in the [dbt platform](https://docs.getdbt.com/docs/platform/wizard-platform.md) from the home app or Studio IDE. [![How data flows when dbt Wizard runs in the dbt platform, and what is and isn't shared with the AI provider.](/img/docs/dbt-platform/wizard-architecture-platform.png?v=2 "How data flows when dbt Wizard runs in the dbt platform, and what is and isn't shared with the AI provider.")](#)How data flows when dbt Wizard runs in the dbt platform, and what is and isn't shared with the AI provider. Your prompt, project metadata, and any query results you approve are sent to the AI provider over TLS. Warehouse credentials are never shared, queries run only after you approve them, and dbt Labs-managed providers can't retain your data or train on it. ##### Approval and review By default, dbt Wizard keeps you in control before it changes your project or runs commands. In the platform: * dbt Wizard shows file edits as diffs for you to accept or reject before anything is persisted * dbt commands ask for confirmation before running * In the home app and Studio IDE, toggle **Ask for approval** or **Edit files automatically** to choose how much freedom the agent has per session There is no bash sandbox in the platform — shell access is not exposed the way it is in the CLI. ##### Sessions and conversations In the platform, a session is a saved conversation in the dbt Wizard panel or home app: * Ask follow-up questions without restating full context * Continue an earlier investigation, refactor, or validation task * Review prior prompts, responses, and proposed changes in the conversation list Start a new session with **Start new dbt Wizard chat** in the panel. Chat history is retained for 90 days. Refreshing the same browser tab keeps your active session; opening a new tab starts empty. For Studio-specific behavior and availability, refer to [dbt Wizard in Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md). #### In the terminal (CLI) Use the [dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-cli.md) for local development. ##### Connections and authentication (MCP) dbt Wizard can connect to MCP servers from the CLI, including the [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md), for access to platform APIs, Semantic Layer metadata, and cross-project context. For the complete setup (like supported server types, configuration keys, authentication, and examples), refer to [Use MCP servers with the dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-mcp.md). To connect to an MCP server, run the following commands, replacing `MCP_NAME` with a name of your choice and `YOUR_MCP_URL` with the URL of the MCP server: ```bash wizard mcp add MCP_NAME --url https://YOUR_MCP_URL wizard mcp login MCP_NAME ``` You can see more options by running `wizard mcp --help`. For example, to connect to the dbt MCP server, replace `DBT_MCP_ENDPOINT` with your endpoint and run: ```bash wizard mcp add dbt --url DBT_MCP_ENDPOINT ``` You should see output similar to `Added global MCP server 'dbt'.` Then authenticate: ```bash wizard mcp login dbt ``` The dbt MCP server reads its connection settings — such as `DBT_HOST`, `DBT_TOKEN`, `DBT_PROJECT_DIR`, `DBT_PATH`, `DBT_PROD_ENV_ID`, and `DBT_ACCOUNT_ID` — from environment variables, typically a `.env` file in your dbt project root. If dbt Wizard can't reach the server, confirm these values are set, then restart `wizard` so it picks up the changes. For the full list of variables and an example `.env` file, refer to [Set up self-hosted MCP](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md) and the [Environment variables reference](https://docs.getdbt.com/docs/dbt-ai/mcp-environment-variables.md). ##### Deferral and state When you work on part of a project, dbt Wizard uses [deferral](https://docs.getdbt.com/reference/node-selection/defer.md) so it can reuse models that are already built elsewhere (for example, in production) instead of rebuilding everything. This saves time and warehouse cost. How deferral is handled depends on the mode set up in `wizard_config.toml` for your project: | `deferral.mode` value | Behavior | | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `"wizard"` | dbt Wizard handles deferral for you. You tell dbt Wizard which target from your `profiles.yml` to defer to (it tries to detect one automatically when you first set up the project). dbt Wizard then compiles that target and reuses its models for any upstream models you haven't built yourself. | | `"fusion_cloud"` | The dbt platform handles deferral against your connected environment, so dbt Wizard doesn't manage local state. | | `"cloud_cli"` | The dbt platform CLI handles credentials and deferral through the dbt platform, so dbt Wizard doesn't manage local state or inject deferral flags. | | `"dbt_state"` | dbt State or run cache handles deferral, so dbt Wizard skips its own production compile. | | `"manual"` | You maintain the deferral manifest path manually. | | `"disabled"` | Deferral is disabled for the project. | For more about dbt State, refer to [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md). The per-project `favor_state` setting defaults to `true`. With favor-state on, deferred relations take precedence. Set `favor_state = false` when you want dbt to use relations you have already built in development and fall back to the deferred environment for relations that aren't available there. dbt Wizard stores the deferral mode for each project in `wizard_config.toml` under `deferral.mode`. For a complete setup and verification procedure, refer to [Developing with production deferral](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-6-production-deferral.md). ##### Approval and sandboxing By default, the CLI keeps you in control before it changes your project or runs commands. In the terminal: * dbt Wizard shows file changes as diffs so you can review them * Commands that need permission under the active approval policy request confirmation before running * The active sandbox profile limits where shell commands can read and write. dbt Wizard shows the active profile when the session starts. Choose the sandbox profile and approval behavior that match the task: ```bash # Restrict shell commands to read-only access. wizard --sandbox read-only # Never prompt for approval during this session. # Useful for trusted tasks where you want Wizard to iterate without stopping. wizard --ask-for-approval never # Allow shell commands to write inside your workspace directory. # Useful for commands that generate files. wizard --sandbox workspace-write ``` Use relaxed settings when you want dbt Wizard to move faster on a scoped task you trust, such as a large refactor, generating documentation, or applying repetitive test updates. The tradeoff is that dbt Wizard has more freedom to act before you review each step, so use these settings in a clean working tree or feature branch. Approval settings apply to the current session. They are not scoped per action type in the command line flags shown above; for example, `--ask-for-approval never` relaxes prompts broadly for that session. To set defaults that persist across sessions or apply only to a specific project, use the [configuration file](https://docs.getdbt.com/docs/dbt-ai/wizard-config.md). You can view more options by running: ```bash wizard --help ``` ##### Sessions In the CLI, a session is a saved conversation and task history from a previous run on your machine. Sessions help you return to earlier work, continue a multi-step task, or review what the agent did in a past interaction. Within a session, you can: * Ask follow-up questions without restating the full context * Continue work on an earlier investigation, refactor, or validation task * Review prior prompts, responses, tool calls, and proposed changes Resume a previous session with: ```bash wizard resume # choose from a list of saved sessions wizard resume --last # resume the most recent session ``` Each CLI session is saved locally. This is separate from platform conversations, which are stored in your dbt platform account. #### Related docs * [dbt Wizard overview](https://docs.getdbt.com/docs/platform/wizard-overview.md) * [dbt Wizard in the dbt platform](https://docs.getdbt.com/docs/platform/wizard-platform.md) * [Use dbt Wizard locally](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) * [dbt Wizard command reference](https://docs.getdbt.com/docs/dbt-ai/wizard-cli-reference.md) * [How to use dbt Wizard in your dbt project](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-1-intro.md) for recommended workflows --- ### How to find your dbt MCP IDs Several dbt MCP environment variables and headers require numeric IDs from your dbt platform account. This guide shows exactly where to find each one. Use numeric IDs, not full URLs ID variables expect integers, not URLs. A common mistake is copying the URL from your browser address bar. ```bash # ✅ Correct DBT_HOST=cloud.getdbt.com # https://cloud.getdbt.com also works DBT_PROD_ENV_ID=54321 DBT_USER_ID=123 # ❌ Wrong — IDs must be numeric, not full URLs DBT_PROD_ENV_ID=https://cloud.getdbt.com/deploy/12345/projects/67890/environments/54321 DBT_USER_ID=https://cloud.getdbt.com/settings/profile ``` #### DBT\_HOST (account hostname) Your hostname is the domain you use to access dbt platform. Both `cloud.getdbt.com` and `https://cloud.getdbt.com` are accepted. 1. Log in to your dbt platform account. 2. Go to **Account settings**. 3. Copy the **Access URL** value. | Account type | Example Access URL | DBT\_HOST value | | -------------------------------- | ------------------------- | ------------------------- | | US multi-tenant | `cloud.getdbt.com` | `cloud.getdbt.com` | | Accounts with a subdomain prefix | `abc123.us1.dbt.com` | `abc123.us1.dbt.com` | | Single-tenant | `your-company.getdbt.com` | `your-company.getdbt.com` | For more information on regions and hosting, refer to [Access, Regions, & IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). #### DBT\_ACCOUNT\_ID (account ID) 1. Log in to your dbt platform account. 2. Go to **Account settings**. 3. The account ID is displayed on the settings page, or you can find it in the URL: `https://cloud.getdbt.com/settings/accounts/ACCOUNT_ID/`. So for example, if the URL is `https://YOUR_ACCESS_URL/settings/accounts/12345/`, the account ID would be `12345`. Alternatively, see [Finding your user and account IDs](https://docs.getdbt.com/faqs/Accounts/find-user-id.md) for additional methods. #### DBT\_PROD\_ENV\_ID (production environment ID) 1. Log in to your dbt platform account. 2. Go to **Orchestration** → **Environments**. 3. Click on your production environment. 4. The environment ID is in the URL: `https://cloud.getdbt.com/deploy/ACCOUNT_ID/projects/PROJECT_ID/environments/ENVIRONMENT_ID`. So for example, if the URL is `https://YOUR_ACCESS_URL/deploy/12345/projects/67890/environments/54321`, the environment ID would be `54321`. Copy only the number at the end — for example, `54321`. #### DBT\_DEV\_ENV\_ID (development environment ID) Follow the same steps as for `DBT_PROD_ENV_ID`, but click on your development environment instead of production. This variable is required for `execute_sql`. If you don't have a dedicated development environment, you can use your production environment ID here, though a separate development environment is recommended. #### DBT\_USER\_ID (user ID) 1. Log in to your dbt platform account. 2. Go to **Account settings** → **Profile** (or click your profile/avatar). 3. Your user ID is in the URL: `https://cloud.getdbt.com/settings/profile/USER_ID`. So for example, if the URL is `https://YOUR_ACCESS_URL/settings/profile/123`, the user ID would be `123`. Copy only the number. Alternatively, see [Finding your user and account IDs](https://docs.getdbt.com/faqs/Accounts/find-user-id.md). #### DBT\_TOKEN (access token) The following sections explain how to find your Personal Access Token (PAT) and Service token. ##### Personal Access Token (PAT) Required for `execute_sql`. Tied to your personal account. 1. Go to **Account settings** → **API tokens** → **Personal tokens**. 2. Click **+ New token**, give it a name, and copy the token value. 3. Store it somewhere safe — you can't view it again after closing the dialog. ##### Service token Used for shared or team setups. Better for CI/automation. 1. Go to **Account settings** → **API tokens** → **Service tokens**. 2. Click **+ New token**, assign the required permissions, and copy the token value. 3. For full MCP access, the service token needs at least `Semantic Layer Only`, `Metadata Only`, and `Developer` permissions. For more information, see [User tokens (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) and [Service tokens](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md). --- ### Hybrid setup dbt platform | Enterprise+ Set up Hybrid projects to upload dbt v1 artifacts into dbt for better collaboration and visibility. Available in public preview Hybrid projects is available in public preview to [dbt Enterprise accounts](https://www.getdbt.com/pricing). #### Set up Hybrid projects In a hybrid project, you use dbt v1 locally and can upload artifacts of that dbt v1 project to dbt for central visibility, cross-project referencing, and easier collaboration. This setup requires connecting your dbt v1 project to a dbt project and configuring a few environment variables and access settings. Follow these steps to set up a dbt Hybrid project and upload dbt v1 artifacts into dbt: * [Make dbt models public](#make-dbt-models-public) (optional) * [Create hybrid project](#create-hybrid-project) * [Generate service token and artifact upload values](#generate-service-token-and-artifact-upload-values) * [Configure dbt v1 project and upload artifacts](#configure-dbt-core-project-and-upload-artifacts) * [Review artifacts in dbt](#review-artifacts-in-dbt-cloud) Make sure to enable the hybrid projects toggle in dbt’s **Account settings** page. ##### Make dbt models public (optional) This step is optional and and only needed if you want to share your dbt v1 models with other dbt projects using the [cross-project referencing](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#how-to-write-cross-project-ref) feature. Before connecting your dbt v1 project to a dbt project, make sure models that you want to share have `access: public` in their model configuration. This setting makes those models visible to other dbt projects for better collaboration, such as [cross-project referencing](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#how-to-write-cross-project-ref). 1. The easiest way to set this would be in your `dbt_project.yml` file, however you can also set this in the following places: * `dbt_project.yml` (project-level) * `properties.yml` (for individual models) * A model's `.sql` file using a `config` block Here's an example using a `dbt_project.yml` file where the marts directory is set as public so they can be consumed by downstream tools: dbt\_project.yml ```yaml models: define_public_models: # This is my project name, remember it must be specified marts: +access: public ``` 2. After defining `access: public`, rerun a dbt execution in the dbt v1 command line interface (CLI) (like `dbt run`) to apply the change. 3. For more details on how to set this up, see [access modifier](https://docs.getdbt.com/docs/mesh/govern/model-access.md#access-modifiers) and [`access` config](https://docs.getdbt.com/reference/resource-configs/access.md). ##### Create hybrid project Create a hybrid project in dbt to allow you to upload your dbt v1 artifacts to dbt. A [dbt account admin](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#permission-sets) should perform the following steps and share the artifacts information with a dbt v1 user: 1. To create a new project in dbt, navigate to **Account home**. 2. Click on **+New project**. 3. Fill out the **Project name**. Name the project something that allows you to recognize it's a dbt v1 project. * You don't need to set up a [data warehouse](https://docs.getdbt.com/docs/supported-data-platforms.md) or [Git connection](https://docs.getdbt.com/docs/platform/git/configure-git.md), however to upgrade the hybrid project to a full dbt project, you'd need to set up data warehouse and Git connection. 4. Select the **Advanced settings** toggle and then select the **Hybrid development** checkbox. Click **Continue**. * The hybrid project will have a visible **Hybrid** indicator in the project list to help you identify it. [![Hybrid project new project](/img/docs/deploy/hp-new-project.jpg?v=2 "Hybrid project new project")](#)Hybrid project new project 5. After creating a project, create a corresponding [production environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md#create-a-deployment-environment) and click **Save**. You will need to create a placeholder [profile](https://docs.getdbt.com/docs/platform/about-profiles.md) and assign it to the environment to save. 6. (Optional) To update an existing dbt project to a hybrid project, navigate to **Account settings** and then select the **Project**. Click **Edit** and then check the **Hybrid development** checkbox. [![Hybrid project for an existing project](/img/docs/deploy/hp-existing-project.jpg?v=2 "Hybrid project for an existing project")](#)Hybrid project for an existing project ##### Generate service token and artifact upload values A dbt admin should perform these steps to generate a [service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md#enterprise-plans-using-service-account-tokens) (with both **Job Runner** *and* **Job Viewer** permissions) and copy the values needed to configure a dbt v1 project so it's ready to upload generated artifacts to dbt. The dbt admin should share the values with a dbt v1 user. 1. Go to the Hybrid project environment you created in the previous step by navigating to **Deploy** > **Environments** and selecting the environment. 2. Select the **Artifact upload** button and copy the following values, which the dbt v1 user will need to reference in their dbt v1's `dbt_project.yml` configuration: * **[Tenant URL](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md)** * **Account ID** * **Environment ID** * **Create a service token** * dbt creates a service token with both **Job Runner** *and* **Job Viewer** permissions. * Note if you don't see the **Create service token** button, it's likely you don't have the necessary permissions to create a service token. Contact your dbt admin to either get the necessary permissions or create the service token for you. [![Generate hybrid project service token](/img/docs/deploy/hp-artifact-upload.png?v=2 "Generate hybrid project service token")](#)Generate hybrid project service token 3. Make sure to copy and save the values as they're needed to configure your dbt v1 project in the next step. Once the service token is created, you can't access it again. ##### Configure dbt project and upload artifacts Once you have the values from the previous step, you can prepare your dbt v1 project for artifact upload by following these steps: 1. Check your dbt version by running `dbt --version` and you should see the following: ```bash Core: - installed: 1.10.0-b1 - latest: 1.9.3 - Ahead of latest version! ``` 2. If you don't have the latest version (1.10 or later), [upgrade](https://docs.getdbt.com/docs/local/install-dbt.md?version=1#change-dbt-core-versions) your dbt v1 project by running `python -m pip install --upgrade dbt-core`. 3. Set the following environment variables in your dbt v1 project by running the following commands in the CLI. Replace the `your_account_id`, `your_environment_id`, and `your_token` with the actual values in the [previous step](#generate-service-token-and-artifact-upload-values). (Applies to dbt v1.11 and later) ```bash export DBT_CLOUD_ACCOUNT_ID=your_account_id export DBT_CLOUD_ENVIRONMENT_ID=your_environment_id export DBT_CLOUD_TOKEN=your_token export DBT_ENGINE_UPLOAD_TO_ARTIFACTS_INGEST_API=True ``` * Set the environment variables in whatever way you use them in your project. * To unset an environment variable, run `unset environment_variable_name`, replacing `environment_variable_name` with the actual name of the environment variable. 4. In your local dbt v1 project, add the following items you copied in the [previous section](https://docs.getdbt.com/docs/deploy/hybrid-setup.md#enable-artifact-upload) to the dbt v1's `dbt_project.yml` file: * `tenant_hostname` ```yaml name: "jaffle_shop" version: "3.0.0" require-dbt-version: ">=1.5.0" ....rest of dbt_project.yml configuration... dbt-cloud: tenant_hostname: cloud.getdbt.com # Replace with your Tenant URL ``` 5. Once you set the environment variables using the `export` command in the same dbt CLI session, you can execute a `dbt run` in the CLI. ```bash dbt run ``` To override the environment variables set, execute a `dbt run` with the environment variable prefix. For example, to use a different account ID and environment ID: ```bash DBT_CLOUD_ACCOUNT_ID=1 DBT_CLOUD_ENVIRONMENT_ID=123 dbt run ``` 6. After the run completes, you should see a `Artifacts uploaded successfully to artifact ingestion API: command run completed successfully` message and a run in dbt under your production environment. ##### Review artifacts in the dbt platform Now that you've uploaded dbt v1 artifacts into the dbt platform and executed a `dbt run`, you can view the artifacts job run: 1. Navigate to **Deploy** 2. Click on **Jobs** and then the **Runs** tab. 3. You should see a job run with the status **Success** with a `</> Artifact ingestion` indicator. 4. Click on the job run to review the logs to confirm a successfully artifacts upload message. If there are any errors, resolve them by checking out the debug logs. [![Hybrid project job run with artifact ingestion](/img/docs/deploy/hp-artifact-job.jpg?v=2 "Hybrid project job run with artifact ingestion")](#)Hybrid project job run with artifact ingestion #### Benefits of using Hybrid projects Now that you've integrated dbt v1 artifacts with your dbt project, you can now: * Collaborate with dbt users by enabling them to visualize and perform [cross-project references](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#how-to-write-cross-project-ref) to dbt models that live in dbt v1 projects. * (Coming soon) New users interested in the [Canvas](https://docs.getdbt.com/docs/platform/canvas.md) can build off of dbt models already created by a central data team in dbt v1 rather than having to start from scratch. * dbt v1 users can navigate to [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) and view their models and assets. To view Catalog, you must have a [read-only seat](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md). --- ### Install dbt OSS Local development (Applies to dbt v2.0 and later) The open-source v2 foundation is licensed under Apache 2.0. Most users don't need this page — [install dbt normally](https://docs.getdbt.com/docs/local/install-dbt.md) with the standard instructions. This page is for organizations that require the Apache 2.0 codebase specifically. #### Install Install the dbt OSS prerelease with `pip`: ```shell python -m pip install --pre dbt-core ``` Confirm the installed version begins with `2.`: ```shell dbt --version ``` During beta, you must target either the pre-release version or an explicit pin. After install, immediately update to the most recent version: Explicit pin: `python -m pip install dbt-core==2.0.0rc2` For adapter install details, refer to the [`dbt` repository](https://github.com/dbt-labs/dbt). #### What's included * The open-source, Rust-based dbt runtime. * The dbt project language and DAG semantics. * The standard dbt command set (`run`, `build`, `test`, `compile`, `parse`, and more). #### What's not included The [standard dbt install](https://docs.getdbt.com/docs/local/install-dbt.md) gives you dbt v2, which adds the following on top of the open source layer: * SQL comprehension and static analysis * LSP features (autocomplete, hover info, inline errors) * `dbt lint` and error diagnostics * dbt VS Code extension integration For the full picture of what you get with dbt, refer to [v2 availability](https://docs.getdbt.com/docs/dbt/dbt-availability.md). #### Contributing To contribute, refer to the [`dbt` repository](https://github.com/dbt-labs/dbt) and its [CONTRIBUTING guide](https://github.com/dbt-labs/dbt/blob/HEAD/CONTRIBUTING.md), or ask in the [dbt Community](https://docs.getdbt.com/community/resources/getting-help.md). #### License dbt OSS is licensed under Apache 2.0. Refer to the [LICENSE file](https://github.com/dbt-labs/dbt/blob/HEAD/LICENSE) in the repository. Refer to [dbt licensing](https://docs.getdbt.com/docs/dbt-licensing.md?version=2.0) for more info. #### Related * [Install dbt](https://docs.getdbt.com/docs/local/install-dbt.md) (standard install) * [Upgrade to v2](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md) * [`dbt` repository on GitHub](https://github.com/dbt-labs/dbt) --- ### Install dbt Wizard CLI Beta ### Install dbt Wizard CLI [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Local development Install the dbt Wizard CLI from your terminal for agentic and governed data development in dbt. This guide explains how to install, verify, update, and uninstall the dbt Wizard CLI on your local machine. Wizard usage and billing From September 1st, 2026, dbt Wizard usage is metered per token against your account's usage credits. All credit amounts are per account, not per user. Enterprise and Enterprise+ accounts get monthly credits. Developer and Starter plans start with a 30-day trial and $100 in credits, as do CLI users via a free dbt account. Refer to [Trial and billing](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/trial-and-billing.md) for what each plan gets, spend limits, and paid access. You can run the dbt Wizard CLI locally from any project on any dbt engine. Be warned, the wizard has been known to cast spells . #### Prerequisites * macOS, Windows, or Linux * A dbt project with a built `target/` directory (`dbt parse`, `dbt compile`, or `dbt build`) * Access to a supported AI provider. You can use a managed provider in dbt, or configure [BYOK](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) with your own provider credentials. ##### Supported AI models dbt Wizard supports [managed models](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/overview.md#dbt-managed-providers) (billed by dbt Labs, no key to manage) and [bring-your-own-key (BYOK)](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) models (billed directly by your provider). Here are the following AI providers supported depending on where you work. Refer to [Model Provider Rate table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) for the full list of available models. ##### dbt platform | Provider | Access | | ---------------------------------------------------------------------------- | ------------------- | | [OpenAI](https://openai.com/policies/row-terms-of-use/) (default) | dbt managed or BYOK | | [Anthropic](https://www.anthropic.com/legal/consumer-terms)† | dbt managed or BYOK | | Open weight models (like DeepSeek, Kimi, and so on). | dbt managed | | [Azure AI Foundry](https://www.microsoft.com/licensing/terms) / Azure OpenAI | BYOK | ##### Locally (CLI) | Provider | Access | | ---------------------------------------------------------------------------- | ------------------- | | [OpenAI](https://openai.com/policies/row-terms-of-use/) | dbt managed or BYOK | | [Anthropic](https://www.anthropic.com/legal/consumer-terms)† | dbt managed or BYOK | | Open weight models (like DeepSeek, Kimi, and so on). | dbt managed | | [Azure AI Foundry](https://www.microsoft.com/licensing/terms) / Azure OpenAI | BYOK | | [AWS Bedrock](https://aws.amazon.com/service-terms/) | BYOK | | [Google Gemini](https://ai.google.dev/gemini-api/terms) | BYOK | | [Snowflake Cortex](https://www.snowflake.com/en/legal/terms-of-service/) | BYOK | | [Databricks Unity AI Gateway](https://www.databricks.com/legal/mcsa) | BYOK | You can also connect a personal OpenAI ChatGPT subscription instead of a key. †Anthropic enterprise and subscription licenses (such as Claude Enterprise) aren't supported per Anthropic's [terms of service](https://www.anthropic.com/legal/consumer-terms). New to the terminal? If you've never used the terminal before, check out the [terminal guide](https://docs.getdbt.com/guides/terminal-guide.md) for for some helpful tips to help you get started! #### Install and set up dbt Wizard CLI 12345 View all stepsNext 1 Install the dbt Wizard CLI Run the install script for your operating system: macOS/Linux: ```bash curl -fsSL https://public.cdn.getdbt.com/dbt-wizard/install/install-wizard.sh | sh ``` Windows (PowerShell): ```powershell irm https://public.cdn.getdbt.com/dbt-wizard/install/install-wizard.ps1 | iex ``` This installs dbt Wizard to `/usr/local/bin/wizard`, along with the dbt [metadata engine](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md#native-metadata-engine) that powers dbt Wizard's project-aware answers. For install and update details, refer to [Install dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-cli.md); to remove them, refer to [Uninstall](https://docs.getdbt.com/docs/dbt-ai/wizard-cli.md#uninstall). #### Update Run the following command to update dbt Wizard to the latest version: ```bash wizard update ``` #### Uninstall 1. Run the built-in uninstall command. It lists every binary, config, and data directory it's about to remove, then asks you to confirm (`Proceed? [Y/N]`) before deleting anything: ```shell wizard system uninstall ``` Uninstalling Wizard Removing `~/.dbt/wizard` deletes your local config, logs, and cache, and can't be undone. Your dbt profiles (`~/.dbt/`) and dbt projects aren't part of dbt Wizard and won't be touched. 2. Confirm the binary is deleted by checking your system path: ```bash which wizard ``` If no output path is returned, dbt Wizard is successfully uninstalled. #### Telemetry dbt Wizard collects anonymous product telemetry to improve the AI agent experience, understand usage patterns, optimize performance, and attribute compute costs without capturing your code, queries, prompts, responses, or file contents. For details about what is collected, what is not collected, and how to opt out of client telemetry, refer to [dbt Wizard CLI data use and telemetry](https://docs.getdbt.com/docs/dbt-ai/wizard-telemetry.md). Best practices for using dbt Wizard Once you're set up, refer to [How to use dbt Wizard in your dbt project](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-1-intro.md) for recommended workflows on real project tasks. #### Related docs * [Use dbt Wizard locally](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md): Install dbt Wizard and start a local terminal session * [Configure BYOK](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md): Manage your API key and choose an AI model * [Command reference](https://docs.getdbt.com/docs/dbt-ai/wizard-cli-reference.md): Full reference for all `wizard` subcommands and global flags * [Use cases and examples](https://docs.getdbt.com/docs/dbt-ai/wizard-use-cases.md): Realistic analytics engineering scenarios * [Migrate from another AI agent](https://docs.getdbt.com/docs/dbt-ai/wizard-migrate.md): Migrate from another AI agent to dbt Wizard * [CLI data use and telemetry](https://docs.getdbt.com/docs/dbt-ai/wizard-telemetry.md): What dbt Wizard CLI collects and how to opt out * [How to use dbt Wizard in your dbt project](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-1-intro.md) for recommended workflows See it in action and share your feedback Want to see dbt Wizard in action? Check out the [demo video](https://www.youtube.com/watch?v=-lIzh1xQWMA). We'd love to hear how dbt Wizard is working for you. Share your feedback by either running the `/feedback` slash command in your interactive terminal session or by going to the [#dbt-wizard](https://getdbt.slack.com/archives/C0B6KLW6T26) channel in the [dbt Community Slack](https://docs.getdbt.com/community/join?version=2.0). Thanks so much for your help in improving dbt Wizard and dbt data development! --- ### Install the dbt VS Code extension Preview ### Install the dbt VS Code extension [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Local development The dbt extension — available for [VS Code and Cursor](https://marketplace.visualstudio.com/items?itemName=dbtLabsInc.dbt\&ssr=false#overview), and [Windsurf](https://open-vsx.org/extension/dbtLabsInc/dbt) — makes dbt development smoother and more efficient. dbt v1 and v2 both support the extension — refer to [Version compatibility](https://docs.getdbt.com/docs/about-dbt-extension.md#version-compatibility) for which features need which setup. note This is the only official dbt Labs VS Code extension. Other extensions *can* work alongside the dbt VS Code extension, but they aren't tested or supported by dbt Labs. #### Prerequisites To use the extension, you need the following: | Prerequisite | Details | | ---------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Project files** | You need a `profiles.yml` file. You may also need a [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md) file for some dbt platform features or credential-based workflows. You don't need a dbt platform project to use the extension. | | **Editor** | [VS Code](https://code.visualstudio.com/), [Cursor](https://www.cursor.com/en), or [Windsurf](https://windsurf.com/editor). | | **Operating system** | macOS, Windows, or Linux. | | **Local configuration** (optional) | [Configure the extension](https://docs.getdbt.com/docs/configure-dbt-extension.md) to mirror your dbt environment locally and set any environment variables required by your project. | | **Project migration support** (optional) | The extension has [dbt-autofix](https://github.com/dbt-labs/dbt-autofix) built in, so you can fix deprecations from the **Problems** pane or in a single pull request. You can also use the **Migrate dbt v1 to dbt v2** agent skill or [run dbt-autofix](https://docs.getdbt.com/guides/prepare-v2-upgrade.md?step=5#what-the-autofix-tool-handles) yourself. | #### Install the extension 1. In your editor, open the **Extensions** tab and search for `dbt`. 2. Locate the extension from the publisher `dbtLabsInc` or `dbt Labs Inc`, then click **Install**. [![Search for the extension](/img/docs/extension/extension-marketplace.png?v=2 "Search for the extension")](#)Search for the extension 3. Open a dbt project in your editor. Make sure the project is added to your current workspace. 4. Confirm that the extension is active by checking for the **dbt Extension** label in the status bar. Hover over the label to view diagnostic information. [![If you see the 'dbt Extension' label, the extension is activated](/img/docs/extension/dbt-extension-statusbar.png?v=2 "If you see the 'dbt Extension' label, the extension is activated")](#)If you see the 'dbt Extension' label, the extension is activated 5. After the extension activates, it automatically downloads the correct dbt Language Server (LSP) for your operating system. [![The dbt Language Server will be installed automatically](/img/docs/extension/extension-lsp-download.png?v=2 "The dbt Language Server will be installed automatically")](#)The dbt Language Server will be installed automatically 6. If dbt isn't installed, the extension prompts you to download and install it. Follow the notification steps, or [install it manually from the command line](#install-dbt-v2-from-the-command-line-if-you-havent-already). [![Follow the prompt to install v2](/img/docs/extension/install-dbt-fusion-engine.png?v=2 "Follow the prompt to install v2")](#)Follow the prompt to install v2 7. Run the VS Code extension [upgrade tool](https://docs.getdbt.com/docs/upgrade-to-dbt-extension.md) to check whether your project is ready for v2 and fix any errors or deprecations. 8. Optional: [Configure your local environment](https://docs.getdbt.com/docs/configure-dbt-extension.md) to mirror your dbt platform environment and [set environment variables](https://docs.getdbt.com/docs/configure-dbt-extension.md#configure-environment-variables) required by your project. The language server ships with dbt v2 The dbt language server is part of the v2 binary rather than a separate download — the extension runs it through v2. If you install v2 manually (for example, in an air-gapped environment) instead of letting the extension manage it, use the [version compatibility matrix](https://docs.getdbt.com/docs/dbt-versions/dbt-version-compatibility.md) to pick a binary that matches your extension version. You're ready to use the dbt extension. Next, you can: * Follow the [getting started](#getting-started) workflow to finish setup. * [Upgrade your project to dbt v2](https://docs.getdbt.com/docs/upgrade-to-dbt-extension.md) if you're migrating from v1. * [Sign in or register](https://docs.getdbt.com/docs/sign-in-dbt-extension.md) for a dbt platform account to keep using advanced features after the 14-day trial. * Review the [limitations and unsupported features](https://docs.getdbt.com/docs/dbt/supported-features.md#limitations).  Install dbt v2 from the command line, if you haven't already. Choose your preferred installation method:  Pip installation for Windows, macOS, and Linux ```shell python -m pip install --pre dbt ``` To upgrade to a newer version: ```shell python -m pip install --upgrade --pre dbt ```  Standalone installation for macOS and Linux ```shell curl -fsSL https://public.cdn.getdbt.com/fs/install/install.sh | sh -s -- --update ``` To use `dbt` immediately after installation, close and reopen your terminal or reload your shell so that the new `$PATH` is recognized: ```shell exec $SHELL ``` To upgrade to a newer version: ```shell dbt system update ``` note `dbtf system update` installs dbt v2 globally — it updates your `PATH` in `~/.zshrc` and creates a `dbtf` alias. To manage multiple versions or isolate your install, use separate shell profiles or virtual environments.  Standalone installation for Windows ```powershell irm https://public.cdn.getdbt.com/fs/install/install.ps1 | iex ``` To use `dbt` immediately after installation, close and reopen or reload your shell so that the new `Path` is recognized: ```powershell Start-Process powershell ``` To upgrade to a newer version: ```shell dbt system update ```  Homebrew installation for macOS ```shell brew tap dbt-labs/dbt brew install dbt-labs/dbt/dbt ``` To upgrade to a newer version: ```shell brew tap dbt-labs/dbt brew upgrade dbt ```  Winget installation for Windows ```shell winget install --id dbtLabs.dbt --exact ``` To upgrade to a specific version: ```shell winget install --id dbtLabs.dbt --exact --version <version> ``` Run the following command to verify your installation: ```bash dbt --version ``` You can use `dbt` or its dbt v2 alias `dbtf` (handy if you already have another dbt CLI installed). Default install path: * macOS/Linux: `$HOME/.local/bin/dbt` * Windows: `C:\Users\<username>\.local\bin\dbt.exe` The installer adds this path automatically, but you may need to reload your shell for the `dbtf` command to work. After installation, follow the [getting started](#getting-started) workflow. You can get started by: * Running `dbt init --fusion-upgrade` to start terminal onboarding. * Running **dbt: Register dbt extension** from the command palette. * Selecting **Get started** from the extension menu. #### Getting started After v2 and the dbt VS Code extension are installed, the dbt logo appears in the sidebar. Click it to open the **Get started** panel. The **Get started** panel is a persistent setup companion available in VS Code and Cursor. It monitors your environment and shows the most important next action. As new v2 releases ship or your project changes, the panel resurfaces relevant steps automatically. For more details refer to [Upgrade to dbt v2](https://docs.getdbt.com/docs/upgrade-to-dbt-extension.md). [![The Get started panel in VS Code showing the setup steps, with the active step highlighted.](/img/docs/extension/vsce-get-started.png?v=2 "The Get started panel in VS Code showing the setup steps, with the active step highlighted.")](#)The Get started panel in VS Code showing the setup steps, with the active step highlighted. The panel guides you through: 1. **Install or update dbt v2**: Detects whether the v2 binary is missing or outdated and installs or updates it with a single click. 2. **Open project**: Checks for a `dbt_project.yml` file in your workspace to confirm a valid dbt project is open before proceeding. 3. **Check dbt v2 compatibility**: Guides you through upgrading your project to v2. You can choose between an agentic migration or manual CLI onboarding — refer to [Upgrade to dbt v2](https://docs.getdbt.com/docs/upgrade-to-dbt-extension.md). 4. **Register**: Confirms you've registered your email to use the extension beyond the 14-day trial period — refer to [Sign in or register](https://docs.getdbt.com/docs/sign-in-dbt-extension.md). When all setup steps are complete, the panel shows a green **Extension setup complete** button. [![The Get started panel showing Extension setup complete with all four steps checked.](/img/docs/extension/vsce-get-started-complete.png?v=2 "The Get started panel showing Extension setup complete with all four steps checked.")](#)The Get started panel showing Extension setup complete with all four steps checked. #### Next steps Once you've installed the dbt VS Code extension, go to the next pages to get started: 1. Review the [Upgrade to dbt v2](https://docs.getdbt.com/docs/upgrade-to-dbt-extension.md) page to upgrade your dbt project to the next-gen engine today! 2. [Sign in or register](https://docs.getdbt.com/docs/sign-in-dbt-extension.md) for a free dbt platform account to keep using advanced features after the 14-day trial. 3. Review the [limitations and unsupported features](https://docs.getdbt.com/docs/dbt/supported-features.md#limitations). --- ### Integrate Claude with dbt MCP Claude is an AI assistant from Anthropic with two primary interfaces: * [Claude Desktop](#claude-desktop): A GUI with MCP support for file access and commands, plus basic coding features. * [Claude Code](#claude-code): A terminal/IDE tool for development. Both interfaces can connect to either: * Self-hosted dbt MCP server (runs on your machine, supports CLI commands like `dbt run`) * Remote dbt MCP server (HTTP, consumption-focused). OAuth needs no local install. Token-based auth on Claude Desktop uses the `mcp-remote` proxy and requires Node.js. #### Prerequisites * You use Claude for AI or agentic work * For OAuth (self-hosted or remote), use your [access URL with a static subdomain](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). * Remote MCP OAuth is available for Starter, Enterprise, and Enterprise+ accounts. Static subdomains required Only accounts with static subdomains (for example, `abc123` in `abc123.us1.dbt.com`) can use OAuth with MCP servers. Follow [these](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) instructions to find your account subdomain. If your account does not have a subdomain, contact support for more information. #### Claude Desktop [Claude Desktop](https://claude.ai/download) reads MCP servers from `claude_desktop_config.json`. Open it from **Settings → Developer → Edit Config**. ##### Set up with self-hosted dbt MCP server For a fast first install, you can download the prebuilt `.mcpb` file; for more control, edit the JSON directly. tip You don't need to clone the dbt-mcp repository — for self-hosted setups, install [uv](https://docs.astral.sh/uv/getting-started/installation/) and run `uvx dbt-mcp` (or use the configs later in this page). Only clone the repository if you want to [contribute to dbt MCP](https://github.com/dbt-labs/dbt-mcp/issues). ###### Quick install with the .mcpb file 1. Go to the [latest dbt MCP release](https://github.com/dbt-labs/dbt-mcp/releases/latest) and download the `dbt-mcp.mcpb` file. 2. Double-click the downloaded file to open it in Claude Desktop. 3. Configure the **dbt platform Host**. You can find this in your dbt platform account by navigating to **Account settings** and copying the **Access URL**. 4. Enable the server in Claude Desktop. 5. Ask Claude a data-related question and see dbt MCP in action! ###### Advanced config with Claude Desktop Use advanced configuration when you want to define the dbt MCP server yourself in Claude's configuration file — the same JSON where Claude stores every MCP server, under `mcpServers`, with fields like `command`, `args`, and `env`. See the [MCP install pattern](https://modelcontextprotocol.io/quickstart/user#installing-the-filesystem-server) for the underlying convention. To open the configuration file and add or replace the dbt MCP server entry: 1. From Claude, go to **Settings…** 2. In the Settings window, select the **Developer** tab. 3. Click **Edit Config** and open the file in a text editor. 4. Add your server configuration under `mcpServers`. Choose the option that fits your use case:  Self-hosted MCP with OAuth[Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") Configuration for users who want seamless OAuth authentication with the dbt platform. ##### dbt platform only This option is for users who only want dbt platform features (Discovery API, Semantic Layer, job management) without self-hosted dbt platform CLI commands. When you use only the dbt platform, the CLI tools are automatically disabled. You can find the `DBT_HOST` field value in your dbt platform account information under **Access URLs**. ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "YOUR-ACCESS-URL" } } } } ``` **Note:** Replace `YOUR-ACCESS-URL` with your Access URL hostname (for example, `abc123.us1.dbt.com`). Both `abc123.us1.dbt.com` and `https://abc123.us1.dbt.com` are accepted. This enables OAuth authentication without requiring self-hosted dbt installation. ##### dbt platform + CLI This option is for users who want both dbt platform CLI commands and dbt platform features (Discovery API, Semantic Layer, job management). The `DBT_PROJECT_DIR` and `DBT_PATH` fields are required for CLI access. You can find the `DBT_HOST` field value in your dbt platform account information under **Access URLs**. ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "YOUR-ACCESS-URL", "DBT_PROJECT_DIR": "/path/to/project", "DBT_PATH": "/path/to/dbt/executable" } } } } ``` **Note:** Replace `YOUR-ACCESS-URL` with your Access URL hostname (for example, `abc123.us1.dbt.com`). Both `abc123.us1.dbt.com` and `https://abc123.us1.dbt.com` are accepted. This enables OAuth authentication.  Self-hosted MCP (CLI only) Self-hosted configuration for users who only want to use dbt commands with dbt v1 or dbt v2 ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_PROJECT_DIR": "/path/to/your/dbt/project", "DBT_PATH": "/path/to/your/dbt/executable" } } } } ``` Finding your paths: * **DBT\_PROJECT\_DIR**: Full path to the folder containing your `dbt_project.yml` file * **DBT\_PATH**: Find by running `which dbt` in Terminal (macOS/Linux) or `where dbt` (Windows) in Powershell  Self-hosted MCP with .env Advanced configuration for users who need custom [environment variables](https://docs.getdbt.com/docs/dbt-ai/mcp-environment-variables.md). Put your `.env` file in your *dbt project root* (same folder as `dbt_project.yml`) and use an absolute path with `--env-file`. Using the `env` field (single-file configuration): IDs are integers, not URLs `DBT_PROD_ENV_ID`, `DBT_DEV_ENV_ID`, and `DBT_USER_ID` must be numeric IDs (for example, `54321`), not full URLs copied from your browser. `DBT_HOST` accepts both `cloud.getdbt.com` and `https://cloud.getdbt.com`. Using an `.env` file (use an absolute path to `.env` in your dbt project root): ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "cloud.getdbt.com", "DBT_TOKEN": "your-token-here", "DBT_PROD_ENV_ID": "12345", "DBT_PROJECT_DIR": "/path/to/project", "DBT_PATH": "/path/to/dbt" } } } } ``` Using an .env file (alternative): ```json { "mcpServers": { "dbt": { "command": "uvx", "args": [ "--env-file", "/absolute/path/to/your-dbt-project/.env", "dbt-mcp" ] } } } ``` 5. Save the file and restart Claude Desktop. You'll see an MCP server indicator in the bottom-right corner of the conversation input box. For more configuration options (env vars, service tokens, tool-access controls), refer to [Set up self-hosted MCP](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md). ##### Set up with remote dbt MCP server The remote dbt MCP server runs in dbt platform. For OAuth, Claude Desktop connects over HTTP with a custom connector. For token-based auth, use the `mcp-remote` proxy described in the token tab (Claude Desktop does not accept `"type": "http"` in `claude_desktop_config.json`). info Remote MCP OAuth is available in public beta for Starter, Enterprise, and Enterprise+ accounts. Get your MCP URL first. You'll need it for both auth methods: You can copy your full **MCP URL** from **Account settings** → **Access URLs** → **MCP Endpoint URL** in dbt platform, and paste it directly into your AI tool.  Build your own MCP URL We recommend using the MCP URL from **Account settings** → **Access URLs** → **MCP Endpoint URL** in dbt platform. However, if you want to build your own MCP URL, use your **Access URL** from **Account settings** in dbt platform. The remote MCP endpoint is `https://YOUR_DBT_HOST_URL/api/ai/v1/mcp`. Replace `YOUR_DBT_HOST_URL` with your hostname only (no `https://`). For default hosts, multi-cell accounts, and regions, see [Access, Regions, & IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). Then follow the tab that matches your auth method: ##### OAuth (remote) *Remote MCP OAuth is available in public beta for Starter, Enterprise, and Enterprise+ accounts.* **Before you connect** * Your MCP client must support OAuth for HTTP-based MCP servers. If it doesn't, use [token-based authentication](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#token-based-authentication) instead. * On first connect, your client opens a browser for sign-in. dbt then shows a consent screen with the scopes (the specific permissions the client is allowed to use) it's requesting — see [Scopes and consent](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#scopes-and-consent) for what each scope means. * Most modern MCP clients self-register on first connect via [dynamic registration (RFC 7591)](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#dynamic-registration). Clients that don't support it need an admin to register them in **Account settings → Integrations → App integrations**. See [Manual registration](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#manual-registration). For the full flow, sessions, and limitations, refer to [OAuth (remote MCP)](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#oauth-remote-mcp). For OAuth, add dbt as a custom connector through Claude Desktop's settings. You don't need to edit `claude_desktop_config.json`. The following steps show how to connect dbt as a custom connector in Claude Desktop. The exact UI varies by tool, but the flow is the same: add a custom connector with your MCP URL, complete the OAuth consent flow, then connect. 1. In your AI tool, go to its connector settings and choose to add a custom connector (in Claude Desktop, go to **Chat → Customize → Connectors**, then click **Add custom connector**). 2. Enter a name (for example, `dbt`) and paste your dbt platform MCP URL (for example, `https://abc123.us1.dbt.com/api/ai/v1/mcp`), then click **Add**. [![Custom connector dialog showing the dbt MCP URL](/img/docs/dbt-cloud/oauth-add-custom-connector.png?v=2 "Custom connector dialog showing the dbt MCP URL")](#)Custom connector dialog showing the dbt MCP URL 3. Click **Connect**. The tool redirects you to dbt to complete the OAuth consent flow, where you can approve or deny individual [scopes](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#scopes-and-consent). [![OAuth consent screen showing requested scopes and project access](/img/docs/dbt-cloud/oauth-consent-screen.png?v=2 "OAuth consent screen showing requested scopes and project access")](#)OAuth consent screen showing requested scopes and project access 4. After you approve, the connector is added to the **Custom connectors** table and shows as connected. [![Adding a custom dbt connector in an AI tool's connector settings](/img/docs/dbt-cloud/oauth-connectors-page.png?v=2 "Adding a custom dbt connector in an AI tool's connector settings")](#)Adding a custom dbt connector in an AI tool's connector settings 5. That's it 🎉! Ask your tool a data question like *"What is the total revenue for the last 30 days?"* to confirm the connection. ##### Token-based Use token-based auth when your client doesn't yet support OAuth for HTTP MCP servers, or when you need a shared or CI-style setup. `"type": "http"` is not valid in Claude Desktop Claude Desktop doesn't accept `"type": "http"` in `claude_desktop_config.json`. If you paste that block (the format that works in [Claude Code](#code-remote)'s `.mcp.json`), Claude Desktop skips the server and shows a message like "some MCP servers could not be loaded." Use the `mcp-remote` proxy config in step 2 instead. Token-based remote MCP on Claude Desktop requires [Node.js](https://nodejs.org/) so `npx` can run `mcp-remote`. 1. From Claude Desktop, go to **Settings → Developer → Edit Config** to open `claude_desktop_config.json`. 2. Add a `dbt-remote` entry under `mcpServers`: ```json { "mcpServers": { "dbt-remote": { "command": "npx", "args": [ "mcp-remote", "https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/", "--header", "Authorization: Token YOUR_DBT_ACCESS_TOKEN", "--header", "x-dbt-prod-environment-id: DBT_PROD_ENV_ID" ] } } } ``` Replace `YOUR_DBT_HOST_URL` with your hostname (for example, `abc123.us1.dbt.com`), and replace the token and environment ID placeholders with your values. To pass optional headers such as `x-dbt-user-id` or `x-dbt-dev-environment-id`, add more `"--header"` and `"Header-Name: value"` pairs to the `args` array. For token-based remote MCP, set these headers in your client's MCP config: * **`Authorization`** *(required)* — `Token YOUR_DBT_ACCESS_TOKEN` or `Bearer YOUR_DBT_ACCESS_TOKEN`. Use a [personal access token (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) or a [service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) with at least Semantic Layer, Metadata, and Developer permissions. * **`x-dbt-prod-environment-id`** *(required)* — your dbt platform production environment ID. Refer to [How to find your dbt MCP IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md#dbt-prod-env-id) for step-by-step instructions. * **`x-dbt-dev-environment-id`** — required for `execute_sql` and dbt v2 tools. Refer to [How to find your dbt MCP IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md#dbt-dev-env-id) for step-by-step instructions. * **`x-dbt-user-id`** — required for `execute_sql` with token-based auth. Refer to [Where can I find my user ID?](https://docs.getdbt.com/faqs/Accounts/find-user-id.md) for details. Use numeric IDs, not full URLs Headers like `x-dbt-prod-environment-id`, `x-dbt-dev-environment-id`, and `x-dbt-user-id` expect numeric IDs (for example, `54321`), not full URLs copied from your browser. The MCP host URL must include `https://`, whether you set it in a `url` field or pass it as an `mcp-remote` argument. If you use token-based auth, `execute_sql` requires a personal access token (PAT). Service tokens won't work. To run SQL without creating a PAT, connect with [OAuth (remote MCP)](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#oauth-remote-mcp) instead. Refer to [Set up remote MCP](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#token-based-authentication) for the full list of headers. Full path to `npx` If Claude Desktop shows `Server disconnected` or can't find `npx`, set `"command"` to the full path from `which npx` (macOS or Linux) or `where npx` (Windows), for example `"command": "/opt/homebrew/bin/npx"`. 3. Save the file and restart Claude Desktop. In **Settings → Developer**, confirm `dbt-remote` is running. Ask Claude a data question to confirm the server is connected. #### Claude Code [Claude Code](https://www.anthropic.com/claude-code) reads MCP servers from `.mcp.json` at the root of your project (the repository root for your workspace). For self-hosted setups, you can reuse the same `command` / `args` / `env` shape you use in other clients. For remote token-based auth, Claude Code supports `"type": "http"` in `.mcp.json`, which Claude Desktop does not. ##### Set up with self-hosted dbt MCP server tip You don't need to clone the dbt-mcp repository — for self-hosted setups, install [uv](https://docs.astral.sh/uv/getting-started/installation/) and run `uvx dbt-mcp` (or use the configs later in this page). Only clone the repository if you want to [contribute to dbt MCP](https://github.com/dbt-labs/dbt-mcp/issues). 1. Follow [Set up self-hosted MCP](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md) to choose your auth pattern: * OAuth with the dbt platform * [CLI only](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#cli-only) * [Environment variables](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#environment-variable-configuration) (including an `.env` file with `--env-file`) 2. Create `.mcp.json` at the project root and add a `dbt` entry under the top-level `mcpServers` key. Pick the option that fits your use case:  Self-hosted MCP with OAuth[Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") Configuration for users who want seamless OAuth authentication with the dbt platform. ##### dbt platform only This option is for users who only want dbt platform features (Discovery API, Semantic Layer, job management) without self-hosted dbt platform CLI commands. When you use only the dbt platform, the CLI tools are automatically disabled. You can find the `DBT_HOST` field value in your dbt platform account information under **Access URLs**. ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "YOUR-ACCESS-URL" } } } } ``` **Note:** Replace `YOUR-ACCESS-URL` with your Access URL hostname (for example, `abc123.us1.dbt.com`). Both `abc123.us1.dbt.com` and `https://abc123.us1.dbt.com` are accepted. This enables OAuth authentication without requiring self-hosted dbt installation. ##### dbt platform + CLI This option is for users who want both dbt platform CLI commands and dbt platform features (Discovery API, Semantic Layer, job management). The `DBT_PROJECT_DIR` and `DBT_PATH` fields are required for CLI access. You can find the `DBT_HOST` field value in your dbt platform account information under **Access URLs**. ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "YOUR-ACCESS-URL", "DBT_PROJECT_DIR": "/path/to/project", "DBT_PATH": "/path/to/dbt/executable" } } } } ``` **Note:** Replace `YOUR-ACCESS-URL` with your Access URL hostname (for example, `abc123.us1.dbt.com`). Both `abc123.us1.dbt.com` and `https://abc123.us1.dbt.com` are accepted. This enables OAuth authentication.  Self-hosted MCP (CLI only) Self-hosted configuration for users who only want to use dbt commands with dbt v1 or dbt v2 ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_PROJECT_DIR": "/path/to/your/dbt/project", "DBT_PATH": "/path/to/your/dbt/executable" } } } } ``` Finding your paths: * **DBT\_PROJECT\_DIR**: Full path to the folder containing your `dbt_project.yml` file * **DBT\_PATH**: Find by running `which dbt` in Terminal (macOS/Linux) or `where dbt` (Windows) in Powershell  Self-hosted MCP with .env Advanced configuration for users who need custom [environment variables](https://docs.getdbt.com/docs/dbt-ai/mcp-environment-variables.md). Put your `.env` file in your *dbt project root* (same folder as `dbt_project.yml`) and use an absolute path with `--env-file`. Using the `env` field (single-file configuration): IDs are integers, not URLs `DBT_PROD_ENV_ID`, `DBT_DEV_ENV_ID`, and `DBT_USER_ID` must be numeric IDs (for example, `54321`), not full URLs copied from your browser. `DBT_HOST` accepts both `cloud.getdbt.com` and `https://cloud.getdbt.com`. Using an `.env` file (use an absolute path to `.env` in your dbt project root): ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "cloud.getdbt.com", "DBT_TOKEN": "your-token-here", "DBT_PROD_ENV_ID": "12345", "DBT_PROJECT_DIR": "/path/to/project", "DBT_PATH": "/path/to/dbt" } } } } ``` Using an .env file (alternative): ```json { "mcpServers": { "dbt": { "command": "uvx", "args": [ "--env-file", "/absolute/path/to/your-dbt-project/.env", "dbt-mcp" ] } } } ``` About `claude mcp add` The Claude Code CLI can register MCP servers with `claude mcp add`, but it typically writes to the user-level config (`~/.claude.json`) rather than the project's `.mcp.json`. For dbt MCP, we recommend committing `.mcp.json` to your repository so the setup is project-scoped and easier to share. ##### Set up with remote dbt MCP server Claude Code can connect to the remote dbt MCP server over HTTP using `"type": "http"` in `.mcp.json`. That HTTP config shape is for Claude Code (and similar clients), not for Claude Desktop's `claude_desktop_config.json`. info Remote MCP OAuth is available in public beta for Starter, Enterprise, and Enterprise+ accounts. 1. Open `.mcp.json` at the root of your project (create it if it doesn't exist). 2. Get your MCP URL: You can copy your full **MCP URL** from **Account settings** → **Access URLs** → **MCP Endpoint URL** in dbt platform, and paste it directly into your AI tool.  Build your own MCP URL We recommend using the MCP URL from **Account settings** → **Access URLs** → **MCP Endpoint URL** in dbt platform. However, if you want to build your own MCP URL, use your **Access URL** from **Account settings** in dbt platform. The remote MCP endpoint is `https://YOUR_DBT_HOST_URL/api/ai/v1/mcp`. Replace `YOUR_DBT_HOST_URL` with your hostname only (no `https://`). For default hosts, multi-cell accounts, and regions, see [Access, Regions, & IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). 3. Add the `dbt` entry to `.mcp.json` using the tab that matches your auth method: ##### OAuth (remote) *Remote MCP OAuth is available in public beta for Starter, Enterprise, and Enterprise+ accounts.* **Before you connect** * Your MCP client must support OAuth for HTTP-based MCP servers. If it doesn't, use [token-based authentication](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#token-based-authentication) instead. * On first connect, your client opens a browser for sign-in. dbt then shows a consent screen with the scopes (the specific permissions the client is allowed to use) it's requesting — see [Scopes and consent](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#scopes-and-consent) for what each scope means. * Most modern MCP clients self-register on first connect via [dynamic registration (RFC 7591)](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#dynamic-registration). Clients that don't support it need an admin to register them in **Account settings → Integrations → App integrations**. See [Manual registration](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#manual-registration). For the full flow, sessions, and limitations, refer to [OAuth (remote MCP)](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#oauth-remote-mcp). Add the following to `.mcp.json` at your project root. On first connect, Claude Code opens a browser for sign-in and consent. ```json { "mcpServers": { "dbt": { "type": "http", "url": "https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/" } } } ``` Replace `YOUR_DBT_HOST_URL` with your hostname (for example, `abc123.us1.dbt.com`). You can find the URL in dbt platform under **Account settings** → **Access URLs** → **MCP Endpoint URL**. You can also register the same server from the CLI: ```bash claude mcp add --transport http dbt https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/ ``` ##### Token-based ```json { "mcpServers": { "dbt": { "type": "http", "url": "https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/", "headers": { "Authorization": "Token YOUR_DBT_ACCESS_TOKEN", "x-dbt-prod-environment-id": "DBT_PROD_ENV_ID", "x-dbt-user-id": "DBT_USER_ID", "x-dbt-dev-environment-id": "DBT_DEV_ENV_ID" } } } } ``` For token-based remote MCP, set these headers in your client's MCP config: * **`Authorization`** *(required)* — `Token YOUR_DBT_ACCESS_TOKEN` or `Bearer YOUR_DBT_ACCESS_TOKEN`. Use a [personal access token (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) or a [service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) with at least Semantic Layer, Metadata, and Developer permissions. * **`x-dbt-prod-environment-id`** *(required)* — your dbt platform production environment ID. Refer to [How to find your dbt MCP IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md#dbt-prod-env-id) for step-by-step instructions. * **`x-dbt-dev-environment-id`** — required for `execute_sql` and dbt v2 tools. Refer to [How to find your dbt MCP IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md#dbt-dev-env-id) for step-by-step instructions. * **`x-dbt-user-id`** — required for `execute_sql` with token-based auth. Refer to [Where can I find my user ID?](https://docs.getdbt.com/faqs/Accounts/find-user-id.md) for details. Use numeric IDs, not full URLs Headers like `x-dbt-prod-environment-id`, `x-dbt-dev-environment-id`, and `x-dbt-user-id` expect numeric IDs (for example, `54321`), not full URLs copied from your browser. The MCP host URL must include `https://`, whether you set it in a `url` field or pass it as an `mcp-remote` argument. If you use token-based auth, `execute_sql` requires a personal access token (PAT). Service tokens won't work. To run SQL without creating a PAT, connect with [OAuth (remote MCP)](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#oauth-remote-mcp) instead. Refer to [Set up remote MCP](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#token-based-authentication) for the full list of headers. 4. Save the file. Claude Code picks up `.mcp.json` on startup — ask it a data question to confirm the connection. #### Troubleshooting  Claude Desktop errors Claude Desktop may return errors such as `Error: spawn uvx ENOENT`, `Error: spawn npx ENOENT`, `Could not connect to MCP server dbt-mcp`, or `Server disconnected`. For self-hosted installations, replace the `command` with the full path to `uvx`: run `which uvx` on Unix systems or `where uvx` on Windows and paste the full path into your JSON (for example, `"command": "/the/full/path/to/uvx"`). For token-based remote setups with `mcp-remote`, apply the same full-path fix for `npx`. If Claude reports that entries in `claude_desktop_config.json` are not valid and were skipped, check that you are not using `"type": "http"` in that file. Use the [token-based `mcp-remote` config](#desktop-remote) instead. For remote setups, double-check that the MCP URL ends in `/api/ai/v1/mcp/` (including the trailing slash) and that your `Authorization` header value is `Token YOUR_DBT_ACCESS_TOKEN` or `Bearer YOUR_DBT_ACCESS_TOKEN`. Logs are at `~/Library/Logs/Claude` (macOS) or `%APPDATA%\Claude\logs` (Windows).  Claude Code If the dbt MCP server doesn't connect, confirm `.mcp.json` is at the *project root* and that the `dbt` block matches one of the examples on this page. For self-hosted installations, apply the same full-path fix for `uvx` (and for `--env-file` paths). For remote setups, verify the URL and headers, and try the equivalent `claude mcp add --transport http` command to compare. --- ### Integrate Cursor with dbt MCP [Cursor](https://docs.cursor.com/context/model-context-protocol) is an AI-powered code editor, powered by Microsoft Visual Studio Code (VS Code). After setting up your MCP server, you connect it to Cursor. Log in to Cursor and follow the steps that align with your use case. #### Set up with self-hosted dbt MCP server Choose your setup based on your workflow: * OAuth for dbt platform connections * CLI only if using dbt v1 or dbt v2 locally. * Configure environment variables if you're using them in your dbt platform account. ##### OAuth or CLI Click one of the following application links with Cursor open to automatically configure your MCP server: ##### CLI only (dbt v1 and dbt v2) Self-hosted configuration for users who only want to use dbt commands with dbt v1 or dbt v2 (no dbt platform features). [Add dbt v1 or dbt v2 to Cursor](cursor://anysphere.cursor-deeplink/mcp/install?name=dbt\&config=eyJlbnYiOnsiREJUX1BST0pFQ1RfRElSIjoiL3BhdGgvdG8veW91ci9kYnQvcHJvamVjdCIsIkRCVF9QQVRIIjoiL3BhdGgvdG8veW91ci9kYnQvZXhlY3V0YWJsZSJ9LCJjb21tYW5kIjoidXZ4IiwiYXJncyI6WyJkYnQtbWNwIl19) After clicking: 1. Update (Applies to dbt v1.11 and later) `DBT_ENGINE_PROJECT_DIR` with the full path to your dbt project (the folder containing `dbt_project.yml`). 2. Update `DBT_PATH` with the full path to your dbt executable: * macOS/Linux: Run `which dbt` in Terminal. * Windows: Run `where dbt` in Command Prompt or PowerShell. 3. Save the configuration. ##### OAuth with dbt platform *OAuth with the self-hosted dbt MCP server is available for Starter, Enterprise, and Enterprise+ accounts.* Configuration settings for users who want OAuth authentication with the dbt platform. AI features are enabled by default. Admins can [turn them off or back on anytime](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md). * [dbt platform only](cursor://anysphere.cursor-deeplink/mcp/install?name=dbt\&config=eyJlbnYiOnsiREJUX0hPU1QiOiJodHRwczovLzx5b3VyLWRidC1ob3N0LXdpdGgtY3VzdG9tLXN1YmRvbWFpbj4iLCJESVNBQkxFX0RCVF9DTEkiOiJ0cnVlIn0sImNvbW1hbmQiOiJ1dngiLCJhcmdzIjpbImRidC1tY3AiXX0%3D) * [dbt platform + CLI](cursor://anysphere.cursor-deeplink/mcp/install?name=dbt\&config=eyJlbnYiOnsiREJUX0hPU1QiOiJodHRwczovLzx5b3VyLWRidC1ob3N0LXdpdGgtY3VzdG9tLXN1YmRvbWFpbj4iLCJEQlRfUFJPSkVDVF9ESVIiOiIvcGF0aC90by9wcm9qZWN0IiwiREJUX1BBVEgiOiJwYXRoL3RvL2RidC9leGVjdXRhYmxlIn0sImNvbW1hbmQiOiJ1dngiLCJhcmdzIjpbImRidC1tY3AiXX0%3D) After clicking: 1. Replace `<your-dbt-host-with-custom-subdomain>` with your actual host (for example, `abc123.us1.dbt.com`). 2. (For dbt platform + CLI) Update (Applies to dbt v1.11 and later) `DBT_ENGINE_PROJECT_DIR` and `DBT_PATH` as described above. 3. Save the configuration. ##### Custom environment variables Use this method if you need custom environment variables or prefer to use service tokens. Refer to the [Environment variables reference](https://docs.getdbt.com/docs/dbt-ai/mcp-environment-variables.md) for the complete list of available environment variables for the self-hosted MCP server. 1. Click the following link with Cursor open: [Add to Cursor](cursor://anysphere.cursor-deeplink/mcp/install?name=dbt\&config=eyJjb21tYW5kIjoidXZ4IiwiYXJncyI6WyJkYnQtbWNwIl0sImVudiI6e319) 2. In the template, add your environment variables to the `env` section based on your needs. 3. Save the configuration. ###### Using an `.env` file If you prefer to manage environment variables in a separate file, put the `.env` file in your *dbt project root* (same folder as `dbt_project.yml`). Click this link: [Add to Cursor (with .env file)](cursor://anysphere.cursor-deeplink/mcp/install?name=dbt-mcp\&config=eyJjb21tYW5kIjoidXZ4IC0tZW52LWZpbGUgPGVudi1maWxlLXBhdGg%252BIGRidC1tY3AifQ%3D%3D) Then update `env-file-path` with the absolute path to your `.env` file (for example, `/absolute/path/to/your-dbt-project/.env`). #### Set up with remote dbt MCP server Remote MCP supports **OAuth** or **token-based** headers. * *Remote MCP OAuth is available in public beta for Starter, Enterprise, and Enterprise+ accounts.* * For either method, the MCP URL is `https://<Access URL>/api/ai/v1/mcp`. You can find the URL in dbt platform under **Account settings** → **Access URLs** → **MCP Endpoint URL**. **Before you connect** * Your MCP client must support OAuth for HTTP-based MCP servers. If it doesn't, use [token-based authentication](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#token-based-authentication) instead. * On first connect, your client opens a browser for sign-in. dbt then shows a consent screen with the scopes (the specific permissions the client is allowed to use) it's requesting — see [Scopes and consent](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#scopes-and-consent) for what each scope means. * Most modern MCP clients self-register on first connect via [dynamic registration (RFC 7591)](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#dynamic-registration). Clients that don't support it need an admin to register them in **Account settings → Integrations → App integrations**. See [Manual registration](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#manual-registration). For the full flow, sessions, and limitations, refer to [OAuth (remote MCP)](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#oauth-remote-mcp). The deeplink below configures **token-based** authentication (URL and headers). For OAuth setup, follow the [remote MCP quickstart](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-remote.md#5-configure-your-mcp-client). 1. Click the following application link with Cursor open: [Add to Cursor](cursor://anysphere.cursor-deeplink/mcp/install?name=dbt\&config=eyJ1cmwiOiJodHRwczovLzxob3N0Pi9hcGkvYWkvdjEvbWNwLyIsImhlYWRlcnMiOnsiQXV0aG9yaXphdGlvbiI6InRva2VuIDx0b2tlbj4iLCJ4LWRidC1wcm9kLWVudmlyb25tZW50LWlkIjoiPHByb2QtaWQ%252BIn19) 2. Provide your URL/headers by updating the **host**, **production environment ID**, and **service token** in the template. Use `https://YOUR_DBT_HOST_URL/api/ai/v1/mcp` as the server URL (no trailing slash). IDs are integers, not URLs `DBT_PROD_ENV_ID`, `DBT_USER_ID`, and `DBT_DEV_ENV_ID` must be numeric IDs (for example, `54321`), not full URLs. `DBT_HOST` field accepts the `https://` prefix and without the `https://` prefix. The following are valid examples: ```bash DBT_HOST=https://ab123.us1.dbt.com DBT_HOST=ab123.us1.dbt.com ``` 3. Save, and now you have access to the dbt MCP server! --- ### Integrate Snowflake Cortex agents with dbt MCP dbt platform [Snowflake Cortex agents](https://docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-agents) can call external [MCP servers](https://docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-agents-mcp-connectors) as tools. This guide walks you through connecting a Cortex agent to the remote dbt MCP server so it can query your Semantic Layer metrics and dimensions in plain English from Snowflake Intelligence. The connection uses OAuth: Snowflake registers itself with dbt platform through Dynamic Client Registration (DCR) with PKCE, so you don't store a client secret in Snowflake. Each user completes their own OAuth consent the first time they use the agent, which means the agent respects each user's existing dbt permissions and project access. #### Prerequisites Before connecting a Cortex agent to the remote dbt MCP server, make sure you have the following in place. ###### In dbt platform Make sure the following are set up before connecting from Snowflake: * **Account setup** * Have [AI features](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md) enabled. * [Remote MCP OAuth enabled](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md). The remote MCP server is generally available, but the OAuth connection method is in public beta for Starter and Enterprise-tiered accounts. * A [static subdomain](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) configured, for example `abc123` in `abc123.us1.dbt.com`. If your account doesn't have a subdomain, contact support. * **Access and permissions** * Read-only or higher access to dbt platform. The agent inherits each connected user's permissions. * **Semantic Layer setup** * A configured [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) with [metrics and dimensions](https://docs.getdbt.com/docs/build/build-metrics-intro.md). * **MCP endpoint** You can copy your full **MCP URL** from **Account settings** → **Access URLs** → **MCP Endpoint URL** in dbt platform, and paste it directly into your AI tool.  Build your own MCP URL We recommend using the MCP URL from **Account settings** → **Access URLs** → **MCP Endpoint URL** in dbt platform. However, if you want to build your own MCP URL, use your **Access URL** from **Account settings** in dbt platform. The remote MCP endpoint is `https://YOUR_DBT_HOST_URL/api/ai/v1/mcp`. Replace `YOUR_DBT_HOST_URL` with your hostname only (no `https://`). For default hosts, multi-cell accounts, and regions, see [Access, Regions, & IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). Use the host portion of this URL in the Snowflake SQL, for example `abc123.us1.dbt.com`. ###### In Snowflake * Snowflake Intelligence and Cortex agents enabled in your account and region. * External MCP connectors available for your account. Confirm availability with your Snowflake account team. * The `ACCOUNTADMIN` role, or a role with account-level `CREATE INTEGRATION`. * A database, schema, and role for creating the MCP server and agent. Optional: native SQL execution Cortex agents can compile and execute Semantic Layer queries without extra setup. To also allow ad hoc SQL against Snowflake, configure project read credentials in **Settings → Credentials** in dbt platform. Without those credentials, the agent can still compile Semantic Layer SQL and execute it natively on Snowflake. #### Parameters The SQL on this page uses placeholders. Replace each one with your own value before running: | Placeholder | Description | | ------------------------------- | ------------------------------------------------------------------------------ | | `YOUR_DBT_HOST_URL` | Your dbt host, with no `https://` (for example, `abc123.us1.dbt.com`). | | `INTEGRATION_NAME` | A name for the Snowflake API integration (for example, `dbt_mcp_integration`). | | `TARGET_DATABASE.TARGET_SCHEMA` | The database and schema where the MCP server and agent live. | | `TARGET_ROLE` | The Snowflake role that uses the agent and MCP server. | | `MCP_SERVER_NAME` | A name for the external MCP server object. | | `AGENT_NAME` | A name for the Cortex agent. | | `CORTEX_MODEL` | The orchestration model for the agent (for example, `claude-4-sonnet`). | #### Set up the connection The following steps register the remote dbt MCP server with Snowflake and connect it to a Cortex agent. Steps 1–3 run as Snowflake SQL. Step 4 starts in Snowflake Intelligence and redirects each user to dbt platform to complete OAuth authorization. ##### Step 1: Create the API integration In Snowflake, run this as `ACCOUNTADMIN`. The integration tells Snowflake how to reach the remote dbt MCP endpoint and how to complete OAuth using Dynamic Client Registration with PKCE (no client secret). ```sql -- Create an API integration for the remote dbt MCP server. -- Uses OAuth Dynamic Client Registration + PKCE (no client secret). CREATE API INTEGRATION IF NOT EXISTS INTEGRATION_NAME API_PROVIDER = EXTERNAL_MCP API_ALLOWED_PREFIXES = ('https://YOUR_DBT_HOST_URL/api/ai/v1/mcp') API_USER_AUTHENTICATION = ( TYPE = OAUTH_DYNAMIC_CLIENT OAUTH_CLIENT_AUTH_METHOD = NONE OAUTH_TOKEN_ENDPOINT = 'https://YOUR_DBT_HOST_URL/oauth/token' OAUTH_AUTHORIZATION_ENDPOINT = 'https://YOUR_DBT_HOST_URL/oauth/authorize' OAUTH_RESOURCE_URL = 'https://YOUR_DBT_HOST_URL/api/ai/v1/mcp' OAUTH_ALLOWED_SCOPES = ('user_access', 'offline_access') ) ENABLED = TRUE; ``` The `OAUTH_TOKEN_ENDPOINT` and `OAUTH_AUTHORIZATION_ENDPOINT` use the same host as your MCP URL. The `user_access` and `offline_access` scopes let the agent act on your behalf and refresh its session without you re-authenticating each time. ##### Step 2: Create the external MCP server In Snowflake, grant your role the ability to create an external MCP server, then create one that references the integration from Step 1. ```sql GRANT CREATE EXTERNAL MCP SERVER ON SCHEMA TARGET_DATABASE.TARGET_SCHEMA TO ROLE TARGET_ROLE; -- Create an external MCP server pointing to the remote dbt MCP endpoint. CREATE EXTERNAL MCP SERVER IF NOT EXISTS TARGET_DATABASE.TARGET_SCHEMA.MCP_SERVER_NAME WITH DISPLAY_NAME = 'dbt Semantic Layer MCP' URL = 'https://YOUR_DBT_HOST_URL/api/ai/v1/mcp' API_INTEGRATION = INTEGRATION_NAME; ``` ##### Step 3: Create the Cortex agent In Snowflake, grant your role the privileges to create an agent and to use the MCP server and its integration, then create the agent. ```sql GRANT CREATE AGENT ON SCHEMA TARGET_DATABASE.TARGET_SCHEMA TO ROLE TARGET_ROLE; -- Grant the role access to the MCP server and its underlying integration. GRANT USAGE ON EXTERNAL MCP SERVER TARGET_DATABASE.TARGET_SCHEMA.MCP_SERVER_NAME TO ROLE TARGET_ROLE; GRANT USAGE ON INTEGRATION INTEGRATION_NAME TO ROLE TARGET_ROLE; CREATE AGENT IF NOT EXISTS TARGET_DATABASE.TARGET_SCHEMA.AGENT_NAME COMMENT = 'Analytics agent powered by the dbt Semantic Layer via MCP' PROFILE = '{"display_name": "dbt Semantic Layer Agent"}' FROM SPECIFICATION $$ models: orchestration: "CORTEX_MODEL" instructions: response: 'Answer questions about business data using the dbt Semantic Layer. Present results clearly in plain English with context about what the metrics mean. When data is returned, summarize the key insight before showing details.' orchestration: 'Always use the dbt MCP tools to query metrics and dimensions rather than writing raw SQL. First explore available metrics and dimensions if you are unsure what is available, then construct and execute the appropriate semantic layer query.' sample_questions: - question: 'What are the top 10 products by revenue this quarter?' - question: 'How has inventory turnover trended over the last 12 months?' - question: 'Which regions have the highest order volume?' mcp_servers: - server_spec: name: "TARGET_DATABASE.TARGET_SCHEMA.MCP_SERVER_NAME" $$; ``` Update the `instructions` and `sample_questions` to match the metrics and dimensions in your own Semantic Layer. The `orchestration` instruction steers the agent toward the dbt Semantic Layer tools (like `list_metrics`, `get_dimensions`, and `query_metrics`) instead of writing raw SQL. ##### Step 4: Complete the OAuth flow The agent can't query dbt until each user authorizes it. Complete the OAuth flow once per user: 1. In the Snowflake Intelligence user interface, open your MCP connectors. Depending on your Snowflake version, this is under **Settings → User → MCP Connectors**, or in the **Connectors** panel of the agent's sources. 2. Find the **dbt Semantic Layer MCP** connector you created and select **Connect**. 3. Snowflake redirects you to dbt platform to sign in and approve the requested [scopes](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#scopes-and-consent) on the consent screen. You can scope the connection to a specific project (recommended) so the agent only sees that project's data. 4. After you approve, the connector shows as **Connected** and you're returned to Snowflake. Snowflake self-registers with dbt platform through Dynamic Client Registration on first connect, so no admin action is needed to register it. In dbt platform, admins can review and audit the connected client, and manage sessions and scopes, in **Account settings → Integrations → App integrations**. For the full registration, consent, and session model, see [Connect apps with OAuth](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md). #### Verify the connection Open your agent in Snowflake Intelligence and ask one of its sample questions, such as *"What are the top 10 products by revenue this quarter?"* If the connection is working, the agent calls the dbt Semantic Layer tools and returns an answer grounded in your metrics. info Only [`text_to_sql`](https://docs.getdbt.com/docs/dbt-ai/mcp-available-tools.md) consumes your dbt Copilot action allotment. Other MCP tools do not. When your account runs out of dbt Copilot actions, the remote MCP server blocks every tool that runs through it, including tools invoked from a self-hosted MCP server and [proxied](https://github.com/dbt-labs/dbt-mcp/blob/main/src/dbt_mcp/tools/toolsets.py#L24) to remote MCP, such as SQL and remote dbt v2 tools. If you reach your dbt Copilot actions limit, remote MCP tools remain unavailable until the limit resets. If you need help, contact your account manager. #### Troubleshooting  The connector won't authorize or the OAuth flow fails * Confirm your account has a [static subdomain](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). OAuth with MCP requires one. * Verify the host in `API_ALLOWED_PREFIXES`, `OAUTH_TOKEN_ENDPOINT`, `OAUTH_AUTHORIZATION_ENDPOINT`, and `OAUTH_RESOURCE_URL` all match your MCP URL host exactly, and that the integration `ENABLED = TRUE`. * AI features are enabled by default and admins can [turn them off or back on anytime](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md). * Make sure remote MCP OAuth is available for your account tier.  The agent returns no metrics or empty results * Confirm the project you authorized has a configured [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) with metrics and dimensions. * Check that the user who connected has at least read-only access to that project — the agent only sees what the user can see. * If you scoped the OAuth connection to a single project, make sure it's the project that contains your metrics.  Permission errors when creating the integration or MCP server `CREATE API INTEGRATION` and `CREATE EXTERNAL MCP SERVER` require `ACCOUNTADMIN` (or a role with account-level `CREATE INTEGRATION`). Run Steps 1–2 as `ACCOUNTADMIN`, then grant `USAGE` on the MCP server and integration to the role that runs the agent (Step 3). #### Related docs * [Set up the remote MCP server](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md) * [Available MCP tools](https://docs.getdbt.com/docs/dbt-ai/mcp-available-tools.md) * [Connect apps with OAuth](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md) * [Snowflake: MCP Connectors](https://docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-agents-mcp-connectors) --- ### Integrate VS Code with MCP [Microsoft Visual Studio Code (VS Code)](https://code.visualstudio.com/mcp) is a powerful and popular integrated development environment (IDE). VS Code can connect to either the **self-hosted** dbt MCP server (runs on your machine, supports CLI commands like `dbt run`) or the **remote** dbt MCP server (HTTP, no install, consumption-focused). Before starting, make sure you have: * VS Code installed with the latest updates. * For self-hosted MCP: completed the [self-hosted MCP setup](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md) and configured your dbt project paths. * For remote MCP: your **MCP URL** from **Account settings** → **Access URLs** → **MCP Endpoint URL** in dbt platform. #### Set up with self-hosted dbt MCP server To get started, in VS Code: 1. Open the **Settings** menu and select the correct tab atop the page for your use case: * **Workspace**: Configures the server in the context of your workspace * **User**: Configures the server in the context of your user <br /> **Note for WSL users**: If you're using VS Code with Windows Subsystem for Linux (WSL), you'll need to configure WSL-specific settings. Run the **Preferences: Open Remote Settings** command from the **Command Palette** (F1) or select the **Remote** tab in the **Settings** editor. Local user settings are reused in WSL but can be overridden with WSL-specific settings. Configuring MCP servers in the local user settings will not work properly in a WSL environment. 2. Select **Features** --> **Chat** 3. Ensure that **MCP** is **Enabled** [![mcp-vscode-settings](/img/mcp/vscode_mcp_enabled_image.png?v=2 "mcp-vscode-settings")](#)mcp-vscode-settings 4. Open the command palette `Control/Command + Shift + P`, and select either: * **MCP: Open Workspace Folder MCP Configuration** — if you want to install the MCP server for this workspace * **MCP: Open User Configuration** — if you want to install the MCP server for the user 5. Add your server configuration (`dbt`) to the provided `mcp.json` file as one of the servers: tip You do not need to clone the dbt-mcp repository. Install [uv](https://docs.astral.sh/uv/getting-started/installation/) and run `uvx dbt-mcp` (or use the config below); cloning is only for contributing.  Self-hosted MCP with dbt platform OAuth Self-hosted MCP with OAuth is for users who want to use the dbt platform features. AI features are enabled by default. Admins can [turn them off or back on anytime](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md). Choose your configuration based on your use case: ##### dbt platform only This option is for users who only want dbt platform features (Discovery API, Semantic Layer, job management) without self-hosted dbt platform CLI commands. When you use only the dbt platform, the CLI tools are automatically disabled. You can find the `DBT_HOST` field value in your dbt platform account information under **Access URLs**. ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "YOUR-ACCESS-URL" } } } } ``` **Note:** Replace `YOUR-ACCESS-URL` with your Access URL hostname (for example, `abc123.us1.dbt.com`). Both `abc123.us1.dbt.com` and `https://abc123.us1.dbt.com` are accepted. This enables OAuth authentication without requiring self-hosted dbt installation. ##### dbt platform + CLI This option is for users who want both dbt platform CLI commands and dbt platform features (Discovery API, Semantic Layer, job management). The `DBT_PROJECT_DIR` and `DBT_PATH` fields are required for CLI access. You can find the `DBT_HOST` field value in your dbt platform account information under **Access URLs**. ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "YOUR-ACCESS-URL", "DBT_PROJECT_DIR": "/path/to/project", "DBT_PATH": "/path/to/dbt/executable" } } } } ``` **Note:** Replace `YOUR-ACCESS-URL` with your Access URL hostname (for example, `abc123.us1.dbt.com`). Both `abc123.us1.dbt.com` and `https://abc123.us1.dbt.com` are accepted. This enables OAuth authentication.  Self-hosted MCP (CLI only) For users who only want to use dbt commands with dbt v1 or dbt v2 (Applies to dbt v1.11 and later) ```json { "servers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_ENGINE_PROJECT_DIR": "/path/to/your/dbt/project", "DBT_PATH": "/path/to/your/dbt/executable" } } } } ``` **Finding your paths:** * (Applies to dbt v1.11 and later) **DBT\_ENGINE\_PROJECT\_DIR**: Full path to the folder containing your `dbt_project.yml` file * macOS/Linux: Run `pwd` from your project folder. * Windows: Run `cd` from your project folder in Command Prompt. * **DBT\_PATH**: Path to dbt executable * macOS/Linux: Run `which dbt`. * Windows: Run `where dbt`.  Self-hosted MCP with .env For advanced users who need custom environment variables or service token authentication. Put your `.env` file in your *dbt project root* (same folder as `dbt_project.yml`) and use an absolute path with `--env-file`. Refer to the [Environment variables reference](https://docs.getdbt.com/docs/dbt-ai/mcp-environment-variables.md) for the complete list of available environment variables for the self-hosted MCP server. Using the `env` field (single-file configuration): IDs are integers, not URLs `DBT_PROD_ENV_ID`, `DBT_DEV_ENV_ID`, and `DBT_USER_ID` must be numeric IDs (for example, `54321`), not full URLs copied from your browser. `DBT_HOST` accepts both `cloud.getdbt.com` and `https://cloud.getdbt.com`. (Applies to dbt v1.11 and later) ```json { "servers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_ENGINE_HOST": "cloud.getdbt.com", "DBT_TOKEN": "your-token-here", "DBT_PROD_ENV_ID": "12345", "DBT_ENGINE_PROJECT_DIR": "/path/to/project", "DBT_PATH": "/path/to/dbt" } } } } ``` Using an `.env` file (alternative - two-file configuration): ```json { "servers": { "dbt": { "command": "uvx", "args": [ "--env-file", "/absolute/path/to/your-dbt-project/.env", "dbt-mcp" ] } } } ``` 6. You can start, stop, and configure your MCP servers by: * Running the `MCP: List Servers` command from the Command Palette (Control/Command + Shift + P) and selecting the server. * Utilizing the keywords inline within the `mcp.json` file. [![VS Code inline management](/img/mcp/vscode_run_server_keywords_inline.png?v=2 "VS Code inline management")](#)VS Code inline management Now, you can access the dbt MCP server in VS Code through interfaces like GitHub Copilot. #### Set up with remote dbt MCP server The remote dbt MCP server runs in dbt platform — no `uvx` or self-hosted install needed. VS Code connects to it over HTTP from the same `mcp.json` you use for self-hosted servers. info Remote MCP OAuth is available in public beta for Starter, Enterprise, and Enterprise+ accounts. 1. Open the command palette (`Control/Command + Shift + P`) and select one of: * **MCP: Open Workspace Folder MCP Configuration** — for this workspace. * **MCP: Open User Configuration** — for your user. 2. Get your MCP URL: You can copy your full **MCP URL** from **Account settings** → **Access URLs** → **MCP Endpoint URL** in dbt platform, and paste it directly into your AI tool.  Build your own MCP URL We recommend using the MCP URL from **Account settings** → **Access URLs** → **MCP Endpoint URL** in dbt platform. However, if you want to build your own MCP URL, use your **Access URL** from **Account settings** in dbt platform. The remote MCP endpoint is `https://YOUR_DBT_HOST_URL/api/ai/v1/mcp`. Replace `YOUR_DBT_HOST_URL` with your hostname only (no `https://`). For default hosts, multi-cell accounts, and regions, see [Access, Regions, & IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). 3. Add a `dbt` entry under the top-level `servers` key. (VS Code uses `servers`, not `mcpServers`.) Pick the tab that matches your auth method: ##### OAuth (remote) *Remote MCP OAuth is available in public beta for Starter, Enterprise, and Enterprise+ accounts.* **Before you connect** * Your MCP client must support OAuth for HTTP-based MCP servers. If it doesn't, use [token-based authentication](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#token-based-authentication) instead. * On first connect, your client opens a browser for sign-in. dbt then shows a consent screen with the scopes (the specific permissions the client is allowed to use) it's requesting — see [Scopes and consent](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#scopes-and-consent) for what each scope means. * Most modern MCP clients self-register on first connect via [dynamic registration (RFC 7591)](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#dynamic-registration). Clients that don't support it need an admin to register them in **Account settings → Integrations → App integrations**. See [Manual registration](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#manual-registration). For the full flow, sessions, and limitations, refer to [OAuth (remote MCP)](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#oauth-remote-mcp). Add the following to `mcp.json`. VS Code opens a browser for sign-in and consent the first time the server connects. ```json { "servers": { "dbt": { "type": "http", "url": "https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/" } } } ``` Replace `YOUR_DBT_HOST_URL` with your hostname (for example, `abc123.us1.dbt.com`). You can find the URL in dbt platform under **Account settings** → **Access URLs** → **MCP Endpoint URL**. ##### Token-based Use token-based auth when your client doesn't yet support OAuth for HTTP MCP servers, or when you need a shared/CI setup. ```json { "servers": { "dbt": { "type": "http", "url": "https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/", "headers": { "Authorization": "Token YOUR_DBT_ACCESS_TOKEN", "x-dbt-prod-environment-id": "DBT_PROD_ENV_ID", "x-dbt-user-id": "DBT_USER_ID", "x-dbt-dev-environment-id": "DBT_DEV_ENV_ID" } } } } ``` For token-based remote MCP, set these headers in your client's MCP config: * **`Authorization`** *(required)* — `Token YOUR_DBT_ACCESS_TOKEN` or `Bearer YOUR_DBT_ACCESS_TOKEN`. Use a [personal access token (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) or a [service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) with at least Semantic Layer, Metadata, and Developer permissions. * **`x-dbt-prod-environment-id`** *(required)* — your dbt platform production environment ID. Refer to [How to find your dbt MCP IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md#dbt-prod-env-id) for step-by-step instructions. * **`x-dbt-dev-environment-id`** — required for `execute_sql` and dbt v2 tools. Refer to [How to find your dbt MCP IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md#dbt-dev-env-id) for step-by-step instructions. * **`x-dbt-user-id`** — required for `execute_sql` with token-based auth. Refer to [Where can I find my user ID?](https://docs.getdbt.com/faqs/Accounts/find-user-id.md) for details. Use numeric IDs, not full URLs Headers like `x-dbt-prod-environment-id`, `x-dbt-dev-environment-id`, and `x-dbt-user-id` expect numeric IDs (for example, `54321`), not full URLs copied from your browser. The MCP host URL must include `https://`, whether you set it in a `url` field or pass it as an `mcp-remote` argument. If you use token-based auth, `execute_sql` requires a personal access token (PAT). Service tokens won't work. To run SQL without creating a PAT, connect with [OAuth (remote MCP)](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#oauth-remote-mcp) instead. Refer to [Set up remote MCP](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#token-based-authentication) for the full list of headers. 4. Save the file. Use **MCP: List Servers** from the command palette to start the server, then ask Copilot Chat a data-related question to confirm the connection. #### Troubleshooting This section contains troubleshooting steps for errors you might encounter when integrating VS Code with MCP.  Cannot find \`uvx\` executable If you see errors like `Could not connect to MCP server dbt` or `spawn uvx ENOENT`, VS Code may be unable to find the `uvx` executable. To resolve, use the full path to `uvx` in your configuration: 1. Find the full path: * macOS/Linux: Run `which uvx` in Terminal. * Windows: Run `where uvx` in Command Prompt or PowerShell. 2. Update your `mcp.json` to use the full path: ```json { "servers": { "dbt": { "command": "/full/path/to/uvx", "args": ["dbt-mcp"], "env": { ... } } } } ``` Example on macOS with Homebrew: `"command": "/opt/homebrew/bin/uvx"`  Configuration not working in WSL If you're using VS Code with Windows Subsystem for Linux (WSL), make sure you've configured the MCP server in the WSL-specific settings, not the local user settings. Use the **Remote** tab in the Settings editor or run **Preferences: Open Remote Settings** from the Command Palette.  Server not starting Check the MCP server status: 1. Run `MCP: List Servers` from the Command Palette (Control/Command + Shift + P). 2. Look for any error messages next to the dbt server. 3. Click on the server to see detailed logs. Common issues: * Missing or incorrect paths for (Applies to dbt v1.11 and later) `DBT_ENGINE_PROJECT_DIR` or `DBT_PATH` * Invalid authentication tokens * Missing required environment variables #### Resources * [Microsoft VS Code MCP documentation](https://code.visualstudio.com/docs/copilot/chat/mcp-servers) --- ### Integrate with other orchestration tools Alongside [dbt](https://docs.getdbt.com/docs/deploy/jobs.md), discover other ways to schedule and run your dbt jobs with the help of tools such as the ones described on this page. Build and install these tools to automate your data workflows, trigger dbt jobs (including those hosted on dbt), and enjoy a hassle-free experience, saving time and increasing efficiency. #### Airflow If your organization uses [Airflow](https://airflow.apache.org/), there are a number of ways you can run your dbt jobs, including: ##### dbt platform Installing the [dbt Provider](https://airflow.apache.org/docs/apache-airflow-providers-dbt-cloud/stable/index.html) to orchestrate dbt jobs. This package contains multiple Hooks, Operators, and Sensors to complete various actions within dbt. [![Airflow DAG using DbtCloudRunJobOperator](/img/docs/running-a-dbt-project/airflow_dbt_connector.png?v=2 "Airflow DAG using DbtCloudRunJobOperator")](#)Airflow DAG using DbtCloudRunJobOperator [![dbt job triggered by Airflow](/img/docs/running-a-dbt-project/dbt_cloud_airflow_trigger.png?v=2 "dbt job triggered by Airflow")](#)dbt job triggered by Airflow ##### dbt v1 Invoking dbt v1 jobs through the [BashOperator](https://registry.astronomer.io/providers/apache-airflow/modules/bashoperator). In this case, be sure to install dbt into a virtual environment to avoid issues with conflicting dependencies between Airflow and dbt. For more details on both of these methods, including example implementations, check out [this guide](https://docs.astronomer.io/learn/airflow-dbt-cloud). #### Automation servers Automation servers (such as CodeDeploy, GitLab CI/CD ([video](https://youtu.be/-XBIIY2pFpc?t=1301)), Bamboo and Jenkins) can be used to schedule bash commands for dbt. They also provide a UI to view logging to the command line, and integrate with your git repository. #### Azure Data Factory Integrate dbt and [Azure Data Factory](https://learn.microsoft.com/en-us/azure/data-factory/) (ADF) for a smooth data process from data ingestion to data transformation. You can seamlessly trigger dbt jobs upon completion of ingestion jobs by using the [dbt API](https://docs.getdbt.com/docs/dbt-apis/overview.md) in ADF. The following steps provides you with a detailed overview of how to trigger a dbt job using the API in Azure Data Factory. To use the dbt API to trigger a job in dbt through ADF: 1. In dbt, go to the job settings of the daily production job and turn off the scheduled run in the **Trigger** section. 2. You'll want to create a pipeline in ADF to trigger a dbt job. 3. Securely fetch the dbt service token from a key vault in ADF, using a web call as the first step in the pipeline. 4. Set the parameters in the pipeline, including the dbt account ID and job ID, as well as the name of the key vault and secret that contains the service token. * You can find the dbt job and account id in the URL, for example, if your URL is `https://YOUR_ACCESS_URL/deploy/88888/projects/678910/jobs/123456`, the account ID is 88888 and the job ID is 123456 5. Trigger the pipeline in ADF to start the dbt job and monitor the status of the dbt job in ADF. 6. In dbt, you can check the status of the job and how it was triggered in dbt. #### Cron Cron is a decent way to schedule bash commands. However, while it may seem like an easy route to schedule a job, writing code to take care of all of the additional features associated with a production deployment often makes this route more complex compared to other options listed here. #### Dagster If your organization uses [Dagster](https://dagster.io/), you can use the [dagster\_dbt](https://docs.dagster.io/_apidocs/libraries/dagster-dbt) library to integrate dbt commands into your pipelines. This library supports the execution of dbt through dbt or dbt v1. Running dbt from Dagster automatically aggregates metadata about your dbt runs. Refer to the [example pipeline](https://dagster.io/blog/dagster-dbt) for details. #### Databricks workflows Use Databricks workflows to call the dbt job API, which has several benefits such as integration with other ETL processes, utilizing dbt job features, separation of concerns, and custom job triggering based on custom conditions or logic. These advantages lead to more modularity, efficient debugging, and flexibility in scheduling dbt jobs. For more info, refer to the guide on [Databricks workflows and dbt jobs](https://docs.getdbt.com/guides/databricks-workflows.md). #### Kestra If your organization uses [Kestra](http://kestra.io/), you can leverage the [dbt plugin](https://kestra.io/plugins/plugin-dbt) to orchestrate dbt and dbt v1 jobs. Kestra's user interface (UI) has built-in [Blueprints](https://kestra.io/docs/user-interface-guide/blueprints), providing ready-to-use workflows. Navigate to the Blueprints page in the left navigation menu and [select the dbt tag](https://demo.kestra.io/ui/blueprints/community?selectedTag=36) to find several examples of scheduling dbt v1 commands and dbt jobs as part of your data pipelines. After each scheduled or ad-hoc workflow execution, the Outputs tab in the Kestra UI allows you to download and preview all dbt build artifacts. The Gantt and Topology view additionally render the metadata to visualize dependencies and runtimes of your dbt models and tests. The dbt task provides convenient links to easily navigate between Kestra and dbt UI. #### Orchestra If your organization uses [Orchestra](https://getorchestra.io), you can trigger dbt jobs using the dbt API. Create an API token from your dbt account and use this to authenticate Orchestra in the [Orchestra Portal](https://app.getorchestra.io). For details, refer to the [Orchestra docs on dbt](https://orchestra-1.gitbook.io/orchestra-portal/integrations/transformation/dbt-cloud). Orchestra automatically collects metadata from your runs so you can view your dbt jobs in the context of the rest of your data stack. The following is an example of the run details in dbt for a job triggered by Orchestra: [![Example of Orchestra triggering a dbt job](/img/docs/running-a-dbt-project/dbt_cloud_orchestra_trigger.png?v=2 "Example of Orchestra triggering a dbt job")](#)Example of Orchestra triggering a dbt job The following is an example of viewing lineage in Orchestra for dbt jobs: [![Example of a lineage view for dbt jobs in Orchestra](/img/docs/running-a-dbt-project/orchestra_lineage_dbt_cloud.png?v=2 "Example of a lineage view for dbt jobs in Orchestra")](#)Example of a lineage view for dbt jobs in Orchestra #### Prefect If your organization uses [Prefect](https://www.prefect.io/), the way you will run your jobs depends on the dbt version you're on, and whether you're orchestrating dbt or dbt v1 jobs. Refer to the following variety of options: [![Prefect DAG using a dbt job run flow](/img/docs/running-a-dbt-project/prefect_dag_dbt_cloud.jpg?v=2 "Prefect DAG using a dbt job run flow")](#)Prefect DAG using a dbt job run flow ##### Prefect 2 ##### dbt platform * Use the [trigger\_dbt\_cloud\_job\_run\_and\_wait\_for\_completion](https://prefecthq.github.io/prefect-dbt/cloud/jobs/#prefect_dbt.cloud.jobs.trigger_dbt_cloud_job_run_and_wait_for_completion) flow. * As jobs are executing, you can poll dbt to see whether or not the job completes without failures, through the [Prefect user interface (UI)](https://docs.prefect.io/ui/overview/). ##### dbt v1 * Use the [trigger\_dbt\_cli\_command](https://prefecthq.github.io/prefect-dbt/cli/commands/#prefect_dbt.cli.commands.trigger_dbt_cli_command) task. * For details on both of these methods, see [prefect-dbt docs](https://prefecthq.github.io/prefect-dbt/). ##### Prefect 1 ##### dbt platform * Trigger dbt jobs with the [DbtCloudRunJob](https://docs.prefect.io/api/latest/tasks/dbt.html#dbtcloudrunjob) task. * Running this task will generate a markdown artifact viewable in the Prefect UI. * The artifact will contain links to the dbt artifacts generated as a result of the job run. ##### dbt v1 * Use the [DbtShellTask](https://docs.prefect.io/api/latest/tasks/dbt.html#dbtshelltask) to schedule, execute, and monitor your dbt runs. * Use the supported [ShellTask](https://docs.prefect.io/api/latest/tasks/shell.html#shelltask) to execute dbt commands through the shell. #### Related docs * [dbt plans and pricing](https://www.getdbt.com/pricing/) * [Quickstart guides](https://docs.getdbt.com/guides.md) * [Webhooks for your jobs](https://docs.getdbt.com/docs/deploy/webhooks.md) * [Orchestration guides](https://docs.getdbt.com/guides.md) * [Commands for your production deployment](https://discourse.getdbt.com/t/what-are-the-dbt-commands-you-run-in-your-production-deployment-of-dbt/366) --- ### Jinja and macros #### Related reference docs * [Jinja Template Designer Documentation](https://jinja.palletsprojects.com/page/templates/) (external link) * [dbt Jinja context](https://docs.getdbt.com/reference/dbt-jinja-functions-context-variables.md) * [Macro properties](https://docs.getdbt.com/reference/macro-properties.md) #### Overview In dbt, you can combine SQL with [Jinja](https://jinja.palletsprojects.com), a templating language. Using Jinja turns your dbt project into a programming environment for SQL, giving you the ability to do things that aren't normally possible in SQL. It's important to note that Jinja itself isn't a programming language; instead, it acts as a tool to enhance and extend the capabilities of SQL within your dbt projects. For example, with Jinja, you can: * Use control structures (for example, `if` statements and `for` loops) in SQL * Use [environment variables](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md) in your dbt project for production deployments * Change the way your project builds based on the current target. * Operate on the results of one query to generate another query, for example: * Return a list of payment methods, to create a subtotal column per payment method (pivot) * Return a list of columns in two relations, and select them in the same order to make it easier to union them together * Abstract snippets of SQL into reusable [**macros**](#macros) — these are analogous to functions in most programming languages. If you've used the [`{{ ref() }}` function](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md), you're already using Jinja! Jinja can be used in any SQL in a dbt project, including [models](https://docs.getdbt.com/docs/build/sql-models.md), [analyses](https://docs.getdbt.com/docs/build/analyses.md), [data tests](https://docs.getdbt.com/docs/build/data-tests.md), and even [hooks](https://docs.getdbt.com/docs/build/hooks-operations.md). Ready to get started with Jinja and macros? Check out the [tutorial on using Jinja](https://docs.getdbt.com/guides/using-jinja.md) for a step-by-step example of using Jinja in a model, and turning it into a macro! #### Getting started ##### Jinja Here's an example of a dbt model that leverages Jinja: /models/order\_payment\_method\_amounts.sql ```sql {% set payment_methods = ["bank_transfer", "credit_card", "gift_card"] %} select order_id, {% for payment_method in payment_methods %} sum(case when payment_method = '{{payment_method}}' then amount end) as {{payment_method}}_amount, {% endfor %} sum(amount) as total_amount from app_data.payments group by 1 ``` This query will get compiled to: /models/order\_payment\_method\_amounts.sql ```sql select order_id, sum(case when payment_method = 'bank_transfer' then amount end) as bank_transfer_amount, sum(case when payment_method = 'credit_card' then amount end) as credit_card_amount, sum(case when payment_method = 'gift_card' then amount end) as gift_card_amount, sum(amount) as total_amount from app_data.payments group by 1 ``` You can recognize Jinja based on the delimiters the language uses, which we refer to as "curlies": * **Expressions `{{ ... }}`**: Expressions are used when you want to output a string. You can use expressions to reference [variables](https://docs.getdbt.com/reference/dbt-jinja-functions/var.md) and call [macros](https://docs.getdbt.com/docs/build/jinja-macros.md#macros). * **Statements `{% ... %}`**: Statements don't output a string. They are used for control flow, for example, to set up `for` loops and `if` statements, to [set](https://jinja.palletsprojects.com/en/3.1.x/templates/#assignments) or [modify](https://jinja.palletsprojects.com/en/3.1.x/templates/#expression-statement) variables, or to define macros. * **Comments `{# ... #}`**: Jinja comments are used to prevent the text within the comment from executing or outputing a string. Don't use `--` for comment. When used in a dbt model, your Jinja needs to compile to a valid query. To check what SQL your Jinja compiles to: * **Using dbt:** Click the compile button to see the compiled SQL in the Compiled SQL pane * **Using dbt v1:** Run `dbt compile` from the command line. Then open the compiled SQL file in the `target/compiled/{project name}/` directory. Use a split screen in your code editor to keep both files open at once. ##### Macros [Macros](https://docs.getdbt.com/docs/build/jinja-macros.md) in Jinja are pieces of code that can be reused multiple times – they are analogous to "functions" in other programming languages, and are extremely useful if you find yourself repeating code across multiple models. Macros are defined in `.sql` files, typically in your `macros` directory ([docs](https://docs.getdbt.com/reference/project-configs/macro-paths.md)). Macro files can contain one or more macros — here's an example: macros/cents\_to\_dollars.sql ```sql {% macro cents_to_dollars(column_name, scale=2) %} ({{ column_name }} / 100)::numeric(16, {{ scale }}) {% endmacro %} ``` A model which uses this macro might look like: models/stg\_payments.sql ```sql select id as payment_id, {{ cents_to_dollars('amount') }} as amount_usd, ... from app_data.payments ``` This would be *compiled* to: target/compiled/models/stg\_payments.sql ```sql select id as payment_id, (amount / 100)::numeric(16, 2) as amount_usd, ... from app_data.payments ``` 💡 Use Jinja's whitespace control to tidy your macros! When you're modifying macros in your project, you might notice extra white space in your code in the `target/compiled` folder. You can remove unwanted spaces and lines with Jinja's [whitespace control](https://docs.getdbt.com/faqs/Jinja/jinja-whitespace.md) by using a minus sign. For example, use `{{- ... -}}` or `{%- ... %}` around your macro definitions (such as `{%- macro generate_schema_name(...) -%} ... {%- endmacro -%}`). ##### Using a macro from a package A number of useful macros have also been grouped together into [packages](https://docs.getdbt.com/docs/build/packages.md) — our most popular package is [dbt-utils](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/). After installing a package into your project, you can use any of the macros in your own project — make sure you qualify the macro by prefixing it with the [package name](https://docs.getdbt.com/reference/dbt-jinja-functions/project_name.md): ```sql select field_1, field_2, field_3, field_4, field_5, count(*) from my_table {{ dbt_utils.dimensions(5) }} ``` You can also qualify a macro in your own project by prefixing it with your [package name](https://docs.getdbt.com/reference/dbt-jinja-functions/project_name.md) (this is mainly useful for package authors). #### FAQs What parts of Jinja are dbt-specific? There are certain expressions that are specific to dbt — these are documented in the [Jinja function reference](https://docs.getdbt.com/reference/dbt-jinja-functions-context-variables.md) section of these docs. Further, docs blocks, snapshots, and materializations are custom Jinja *blocks* that exist only in dbt. Which docs should I use when writing Jinja or creating a macro? If you are stuck with a Jinja issue, it can get confusing where to check for more information. We recommend you check (in order): 1. [Jinja's Template Designer Docs](https://jinja.palletsprojects.com/page/templates/): This is the best reference for most of the Jinja you'll use 2. [Our Jinja function reference](https://docs.getdbt.com/reference/dbt-jinja-functions-context-variables.md): This documents any additional functionality we've added to Jinja in dbt. 3. [Agate's table docs](https://agate.readthedocs.io/page/api/table.html): If you're operating on the result of a query, dbt will pass it back to you as an agate table. This means that the methods you call on the table belong to the Agate library rather than Jinja or dbt. Why do I need to quote column names in Jinja? In the [macro example](https://docs.getdbt.com/docs/build/jinja-macros.md#macros) we passed the column name `amount` quotes: ```sql {{ cents_to_dollars('amount') }} as amount_usd ``` We have to use quotes to pass the *string* `'amount'` to the macro. Without the quotes, the Jinja parser will look for a variable named `amount`. Since this doesn't exist, it will compile to nothing. Quoting in Jinja can take a while to get used to! The rule is that you're within a Jinja expression or statement (i.e. within `{% ... %}` or `{{ ... }}`), you'll need to use quotes for any arguments that are strings. Single and double quotes are equivalent in Jinja – just make sure you match them appropriately. And if you do need to pass a variable as an argument, make sure you [don't nest your curlies](https://docs.getdbt.com/best-practices/dont-nest-your-curlies.md). My compiled SQL has a lot of spaces and new lines, how can I get rid of it? This is known as "whitespace control". Use a minus sign (`-`, e.g. `{{- ... -}}`, `{%- ... %}`, `{#- ... -#}`) at the start or end of a block to strip whitespace before or after the block (more docs [here](https://jinja.palletsprojects.com/page/templates/#whitespace-control)). Check out the [tutorial on using Jinja](https://docs.getdbt.com/guides/using-jinja.md#use-whitespace-control-to-tidy-up-compiled-code) for an example. Take caution: it's easy to fall down a rabbit hole when it comes to whitespace control! How do I debug my Jinja? You should get familiar with checking the compiled SQL in `target/compiled/<your_project>/` and the logs in `logs/dbt.log` to see what dbt is running behind the scenes. You can also use the [log](https://docs.getdbt.com/reference/dbt-jinja-functions/log.md) function to debug Jinja by printing objects to the command line. How do I document macros? To document macros, use a [properties file](https://docs.getdbt.com/reference/macro-properties.md) and nest the configurations under a `macros:` key #### Example macros/properties.yml ```yml macros: - name: cents_to_dollars description: A macro to convert cents to dollars arguments: - name: column_name type: column description: The name of the column you want to convert - name: precision type: integer description: Number of decimal places. Defaults to 2. ``` tip From dbt v1.10, you can opt into validating the arguments you define in macro documentation using the `validate_macro_args` behavior change flag. When enabled, dbt will: * Infer arguments from the macro and includes them in the [manifest.json](https://docs.getdbt.com/reference/artifacts/manifest-json.md) file if no arguments are documented. * Raise a warning if documented argument names don't match the macro definition. * Raise a warning if `type` fields don't follow [supported formats](https://docs.getdbt.com/reference/resource-properties/arguments.md#supported-types). Learn more about [macro argument validation](https://docs.getdbt.com/reference/global-configs/behavior-flags/validate_macro_args.md). #### Document a custom materialization When you create a [custom materialization](https://docs.getdbt.com/guides/create-new-materializations.md), dbt creates an associated macro with the following format: ```text materialization_{materialization_name}_{adapter} ``` To document a custom materialization, use the previously mentioned format to determine the associated macro name(s) to document. macros/properties.yml ```yaml macros: - name: materialization_my_materialization_name_default description: A custom materialization to insert records into an append-only table and track when they were added. - name: materialization_my_materialization_name_xyz description: A custom materialization to insert records into an append-only table and track when they were added. ``` Why does my dbt output have so many macros in it? The output of a dbt run counts over 100 macros in your project! ```shell $ dbt run Running with dbt=1.7.0 Found 1 model, 0 tests, 0 snapshots, 0 analyses, 138 macros, 0 operations, 0 seed files, 0 sources ``` This is because dbt ships with its own project, which also includes macros! You can learn more about this [here](https://discourse.getdbt.com/t/did-you-know-dbt-ships-with-its-own-project/764). #### dbtonic Jinja Just like well-written python is pythonic, well-written dbt code is dbtonic. ##### Favor readability over DRY-ness Once you learn the power of Jinja, it's common to want to abstract every repeated line into a macro! Remember that using Jinja can make your models harder for other users to interpret — we recommend favoring readability when mixing Jinja with SQL, even if it means repeating some lines of SQL in a few places. If all your models are macros, it might be worth re-assessing. ##### Leverage package macros Writing a macro for the first time? Check whether we've open sourced one in [dbt-utils](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/) that you can use, and save yourself some time! ##### Set variables at the top of a model `{% set ... %}` can be used to create a new variable, or update an existing one. We recommend setting variables at the top of a model, rather than hardcoding it inline. This is a practice borrowed from many other coding languages, since it helps with readability, and comes in handy if you need to reference the variable in two places: ```sql -- 🙅 This works, but can be hard to maintain as your code grows {% for payment_method in ["bank_transfer", "credit_card", "gift_card"] %} ... {% endfor %} -- ✅ This is our preferred method of setting variables {% set payment_methods = ["bank_transfer", "credit_card", "gift_card"] %} {% for payment_method in payment_methods %} ... {% endfor %} ``` --- ### Job commands dbt platform A dbt production job allows you to set up a system to run a dbt job and job commands on a schedule, rather than running dbt commands manually from the command line or [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md). A job consists of commands that are "chained" together and executed as run steps. Each run step can succeed or fail, which may determine the job's run status (Success, Cancel, or Error). Each job allows you to: * Configure job commands * View job run details, including timing, artifacts, and detailed run steps * Access logs to view or help debug issues and historical invocations of dbt * Set up notifications, and [more](https://docs.getdbt.com/docs/deploy/deployments.md#dbt-cloud) #### Job command types Job commands are specific tasks executed by the job, and you can configure them seamlessly by either adding [dbt commands](https://docs.getdbt.com/reference/dbt-commands.md) or using the checkbox option in the **Commands** section. During a job run, the commands are "chained" together and executed as run steps. When you add a dbt command in the **Commands** section, you can expect different outcomes compared to the checkbox option. [![Configuring checkbox and commands list](/img/docs/dbt-platform/using-dbt-platform/job-commands.gif?v=2 "Configuring checkbox and commands list")](#)Configuring checkbox and commands list ##### Built-in commands Every job invocation automatically includes the [`dbt deps`](https://docs.getdbt.com/reference/commands/deps.md) command, meaning you don't need to add it to the **Commands** list in your job settings. You will also notice every job will include a run step to reclone your repository and connect to your data platform, which can affect your job status if these run steps aren't successful. **Job outcome** — During a job run, the built-in commands are "chained" together. This means if one of the run steps in the chain fails, then the next commands aren't executed, and the entire job fails with an "Error" job status. [![A failed job that had an error during the dbt deps run step.](/img/docs/dbt-platform/using-dbt-platform/fail-dbtdeps.png?v=2 "A failed job that had an error during the dbt deps run step.")](#)A failed job that had an error during the dbt deps run step. ##### Checkbox commands For every job, you have the option to select the [Generate docs on run](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) or [Run source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) checkboxes, enabling you to run the commands automatically. **Generate docs on run** checkbox — dbt executes the `dbt docs generate` command (dbt v1 only), *after* the listed commands. If that particular run step in your job fails, the job can still succeed if all subsequent run steps are successful. For jobs running on dbt v2, manually configuring `dbt docs generate` using the checkbox will no longer be required in the future. Read [Set up a documentation job](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md#set-up-a-documentation-job) for more information. **Run source freshness** checkbox — dbt executes the `dbt source freshness` command as the first run step in your job. If that particular run step in your job fails, the job can still succeed if all subsequent run steps are successful. Read [Source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) for more information. ##### Command list You can add or remove as many dbt commands as necessary for every job. However, you need to have at least one dbt command. There are few commands listed as "dbt CLI" or "dbt v1" in the [dbt Command reference page](https://docs.getdbt.com/reference/dbt-commands.md) page. This means they are meant for use in dbt v1 or dbt CLI, and not in Studio IDE. Using selectors Use [selectors](https://docs.getdbt.com/reference/node-selection/syntax.md) as a powerful way to select and execute portions of your project in a job run. For example, to run tests for `one_specific_model`, use the selector: `dbt test --select one_specific_model`. The job will still run if a selector doesn't match any models. ###### Compare changes custom commands For users that have Advanced CI's [compare changes](https://docs.getdbt.com/docs/deploy/advanced-ci.md#compare-changes) feature enabled and selected the **dbt compare** checkbox, you can add custom dbt commands to optimize running the comparison (for example, to exclude specific large models, or groups of models with tags). Running comparisons on large models can significantly increase the time it takes for CI jobs to complete. [![Add custom dbt commands to when using dbt compare.](/img/docs/deploy/dbt-compare.jpg?v=2 "Add custom dbt commands to when using dbt compare.")](#)Add custom dbt commands to when using dbt compare. The following examples highlight how you can customize the dbt compare command box: * Exclude the large `fct_orders` model from the comparison to run a CI job on fewer or smaller models and reduce job time/resource consumption. Use the following command: ```sql --select state:modified --exclude fct_orders ``` * Exclude models based on tags for scenarios like when models share a common feature or function. Use the following command: ```sql --select state:modified --exclude tag:tagname_a tag:tagname_b ``` * Include models that were directly modified and also those one step downstream using the `modified+1` selector. Use the following command: ```sql --select state:modified+1 ``` ###### Job outcome During a job run, the commands are "chained" together and executed as run steps. If one of the run steps in the chain fails, then the subsequent steps aren't executed, and the job will fail. In the following example image, the first four run steps are successful. However, if the fifth run step (`dbt run --select state:modified+ --full-refresh --fail-fast`) fails, then the next run steps aren't executed, and the entire job fails. The failed job returns a non-zero [exit code](https://docs.getdbt.com/reference/exit-codes.md) and "Error" job status: [![A failed job run that had an error during a run step](/img/docs/dbt-platform/using-dbt-platform/skipped-jobs.png?v=2 "A failed job run that had an error during a run step")](#)A failed job run that had an error during a run step #### Job command failures Job command failures can mean different things for different commands. Some common reasons why a job command may fail: * **Failure at `dbt run`** — [`dbt run`](https://docs.getdbt.com/reference/commands/run.md) executes compiled SQL model files against the current target database. It will fail if there is an error in any of the built models. By default, if a model fails, its downstream models are also skipped. In dbt v1.12+, you can set [`on_error: continue`](https://docs.getdbt.com/reference/resource-configs/on_error.md) on a model to allow its downstream models to still attempt to run despite the failure. * **Failure at `dbt test`** — [`dbt test`](https://docs.getdbt.com/reference/commands/test.md) runs tests defined on models, sources, snapshots, and seeds. A test can pass, fail, or warn depending on its [severity](https://docs.getdbt.com/reference/resource-configs/severity.md). Unless you set [warnings as errors](https://docs.getdbt.com/reference/global-configs/warnings.md), only an error fails the command. Tests on upstream resources prevent downstream resources from running and a failed test will skip them. * **Failure at `dbt build`** — [`dbt build`](https://docs.getdbt.com/reference/commands/build.md) runs models, tests, snapshots, and seeds. This command executes resources in the DAG-specified order. If any upstream resource fails, all downstream resources are skipped, and the command exits with an error code of `1`. In dbt v1.12+, you can set [`on_error: continue`](https://docs.getdbt.com/reference/resource-configs/on_error.md) on a model to allow its downstream models to still attempt to run when that model fails. * **Selector failures** * If a [`select`](https://docs.getdbt.com/reference/node-selection/set-operators.md) matches multiple nodes and one of the nodes fails, then the job will have an exit code `1` and the subsequent command will fail. If you specified the [`--fail-fast`](https://docs.getdbt.com/reference/global-configs/failing-fast.md) flag, then the first failure will stop the entire connection for any models that are in progress. * If a selector doesn't match any nodes, it's not considered a failure. #### Related docs * [Job creation best practices](https://discourse.getdbt.com/t/job-creation-best-practices-in-dbt-cloud-feat-my-moms-lasagna/2980) * [dbt Command reference](https://docs.getdbt.com/reference/dbt-commands.md) * [Job notifications](https://docs.getdbt.com/docs/deploy/job-notifications.md) * [Source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) * [Build and view your docs](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) --- ### Job notifications dbt platform Set up notifications in dbt platform to receive alerts about the outcome of a job run. You can choose to be notified by one or more of the following job run outcomes: * **Succeeds** option — A job run completed successfully with no warnings or errors. * **Warns** option — A job run encountered warnings from [data tests](https://docs.getdbt.com/docs/build/data-tests.md) or [source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) checks. * This notification is triggered by warning-level log lines from those steps, not the job's overall run status. A job that shows "success" in the user interface can still trigger a warn notification if test or freshness steps logged warnings. * **Fails** option — A job run failed to complete. * **Is canceled** option — A job run is canceled. * You may not see Slack notifications for runs canceled by dbt platform's inactivity-timeout cleanup process (which cancels runs that have been inactive for 10 minutes). To receive notifications for these cancellations, use [email notifications](#email-notifications) or [webhooks](https://docs.getdbt.com/docs/deploy/webhooks.md). ##### Notification options dbt platform currently supports the following notification channels: * [Email](#email-notifications) * [Slack (user-linked)](#slack-notifications) * [Slack (account-level)](#slack-notifications-account) * [Microsoft Teams](#microsoft-teams-notifications) Microsoft Teams If you can’t use the native Microsoft Teams integration, you can still send job notifications to a Teams channel by using the channel’s email address as an external email, as explained in the next section. #### Email notifications You can receive email alerts about jobs by configuring the dbt email notification settings. ##### Prerequisites * You must be a *developer user*, *account admin*, or have the **Notification Manager** [permission set](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) to configure email notifications in dbt. For more details, refer to [Users and licenses](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md). * As a developer user, you can set up email notifications for yourself. Once an account admin links the Slack app, licensed users can configure job notifications if they have one of the following: * The **Account Admin**, **Owner**, or **Member** default [group](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#groups) * The [**Notification Manager**](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) permission set (Enterprise). This permission manages notifications across all projects without full Account Admin access ##### Configure email notifications 1. Select your profile icon and then click **Notification settings**. 2. By default, dbt sends notifications to the email address that's in your **User profile** page. If you're an account admin, you can choose a different email address to receive notifications: 1. Under Job notifications, click the **Notification email** dropdown. 2. Select another address from the list. The list includes **Internal Users** with access to the account and **External Emails** that have been added. 3. To add an external email address, click the **Notification email** dropdown 4. Click **Add external email**. 5. Enter the email address, and click Add user. After adding an external email, it becomes available for selection in the **Notification email** dropdown list. External emails can be addresses that are outside of your dbt account and also for third-party integrations like [channels in Microsoft Teams](https://support.microsoft.com/en-us/office/tip-send-email-to-a-channel-2c17dbae-acdf-4209-a761-b463bdaaa4ca) and [PagerDuty email integration](https://support.pagerduty.com/docs/email-integration-guide). note External emails and their notification settings persist until edited or removed even if you remove the admin who added them from the account. [![Example of the Notification email dropdown](/img/docs/deploy/example-notification-external-email.png?v=2 "Example of the Notification email dropdown")](#)Example of the Notification email dropdown 3. Select the **Environment** for the jobs you want to receive notifications about from the dropdown. 4. Click **Edit** to configure the email notification settings. Choose one or more of the run statuses for each job you want to receive notifications about. 5. When you're done with the settings, click **Save**. As an account admin, you can add more email recipients by choosing another **Notification email** from the dropdown, **Edit** the job notification settings, and **Save** the changes. To set up alerts on jobs from a different environment, select another **Environment** from the dropdown, **Edit** those job notification settings, and **Save** the changes. [![Example of the Email notifications page](/img/docs/deploy/example-email-notification-settings-page.png?v=2 "Example of the Email notifications page")](#)Example of the Email notifications page ##### Unsubscribe from email notifications 1. Select your profile icon and click on **Notification settings**. 2. On the **Email notifications** page, click **Unsubscribe from all email notifications**. ##### Send job notifications to a Microsoft Teams channel (email) You can send dbt job [notification emails](#configure-email-notifications) directly to a Microsoft Teams channel by using the channel’s email address. 1. In Microsoft Teams, get the email address for the channel you want to send notifications to. See [Send an email to a channel](https://support.microsoft.com/en-us/office/tip-send-email-to-a-channel-2c17dbae-acdf-4209-a761-b463bdaaa4ca). 2. In dbt platform, click on your profile in the left sidebar and then click **Notification settings**. 3. Under **Job notifications**, click the **Notification email** dropdown. 4. To add an external email address, click **Add external email** at the bottom of the dropdown. 5. Enter the Teams channel email address, and click **Add user**. 6. Make sure you select the Teams channel email from the **Notification email** dropdown (it might be selected already). 7. Then choose the environment for the jobs you want to receive notifications from. 8. Click **Edit**, select the job statuses you want. Then click **Save** to save. #### Slack notifications (user) You can receive Slack alerts about jobs by setting up the Slack integration and then configuring the dbt Slack notification settings. dbt integrates with Slack via OAuth to ensure secure authentication. This is the current Slack integration available for all users and set at the user level, not to be confused with the [Slack notifications at the account level](#slack-notifications-account) feature. note Virtual Private Cloud (VPC) admins must [contact support](mailto:support@getdbt.com) to complete the Slack integration. If there has been a change in user roles or Slack permissions where you no longer have access to edit a configured Slack channel, please [contact support](mailto:support@getdbt.com) for assistance. ##### Prerequisites * You have a Slack workspace that you want to receive job notifications from. * You must be a Slack Workspace Owner. * You must be an account admin to configure Slack notifications in dbt. For more details, refer to [Users and licenses](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md). The integration only supports *public* channels in the Slack workspace at this moment. Once an account admin links the Slack app, licensed users can configure job notifications if they have one of the following: * The **Account Admin**, **Owner**, or **Member** default [group](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#groups) * The [**Notification Manager**](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) permission set (Enterprise). This permission manages notifications across all projects without full Account Admin access ##### Set up the Slack integration 1. Select **Account settings** and then select **Integrations** from the left sidebar. 2. Locate the **OAuth** section with the Slack application and click **Link**. [![Link for the Slack app](/img/docs/dbt-platform/Link-your-Slack-Profile.png?v=2 "Link for the Slack app")](#)Link for the Slack app ###### Logged in to Slack If you're already logged in to Slack, the handshake only requires allowing the app access. If you're a member of multiple workspaces, you can select the appropriate workspace from the dropdown menu in the upper right corner. [![Allow dbt access to Slack](/img/docs/dbt-platform/Allow-dbt-to-access-slack.png?v=2 "Allow dbt access to Slack")](#)Allow dbt access to Slack ###### Logged out If you're logged out or the Slack app/website is closed, you must authenticate before completing the integration. 1. Complete the field defining the Slack workspace you want to integrate with dbt. [![Define the workspace](/img/docs/dbt-platform/define-workspace.png?v=2 "Define the workspace")](#)Define the workspace 2. Sign in with an existing identity or use the email address and password. 3. Once you have authenticated successfully, accept the permissions. [![Allow dbt access to Slack](/img/docs/dbt-platform/accept-permissions.png?v=2 "Allow dbt access to Slack")](#)Allow dbt access to Slack ##### Configure Slack notifications 1. Select your profile icon and then click on **Notification settings**. 2. Select **Slack notifications** in the left sidebar. 3. Select the **Notification channel** you want to receive the job run notifications from the dropdown. [![Example of the Notification channel dropdown](/img/docs/deploy/example-notification-slack-channels.png?v=2 "Example of the Notification channel dropdown")](#)Example of the Notification channel dropdown 4. Select the **Environment** for the jobs you want to receive notifications about from the dropdown. 5. Click **Edit** to configure the Slack notification settings. Choose one or more of the run statuses for each job you want to receive notifications about. 6. When you're done with the settings, click **Save**. To send alerts to another Slack channel, select another **Notification channel** from the dropdown, **Edit** those job notification settings, and **Save** the changes. To set up alerts on jobs from a different environment, select another **Environment** from the dropdown, **Edit** those job notification settings, and **Save** the changes. [![Example of the Slack notifications page](/img/docs/deploy/example-slack-notification-settings-page.png?v=2 "Example of the Slack notifications page")](#)Example of the Slack notifications page ##### Disable the Slack integration 1. Select **Account settings** and on the **Integrations** page, scroll to the **OAuth** section. 2. Click the **X** icon (on the far right of the Slack integration) and click **Unlink**. Channels that you configured will no longer receive Slack notifications. *This is not an account-wide action.* Channels configured by other account admins will continue to receive Slack notifications if they still have active Slack integrations. To migrate ownership of a Slack channel notification configuration, have another account admin edit their configuration. #### Slack notifications (account) Integrate Slack with dbt platform at the account level to receive job notifications in Slack. dbt integrates with Slack via OAuth to ensure secure authentication. A single dbt platform account can integrate with one Slack workspace. ##### Prerequisites * You have a Slack workspace that you want to receive job notifications from. * A dbt platform account admin must link the Slack app at the account level. * Install the official dbt platform Slack app using the [steps outlined in the next section](#set-up-the-slack-integration-1). * To install the Slack app to a workspace, your Slack org must permit app installations. In some orgs this requires a Slack admin approval. * The integration only supports *public* channels in the Slack workspace at this moment. Once an account admin links the Slack app, licensed users can configure job notifications if they have one of the following: * The **Account Admin**, **Owner**, or **Member** default [group](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#groups) * The [**Notification Manager**](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) permission set (Enterprise). This permission manages notifications across all projects without full Account Admin access IT licenses can receive job notifications but can't [configure them](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md?version=1.12). Only **Account Admins** can connect or disconnect the Slack workspace. ##### Set up the Slack integration The account-level Slack integration uses the official dbt platform Slack app, which is separate from the [user-linked Slack integration](#slack-notifications). To use Slack notifications, you must unlink the old Slack app and then connect the new official app: 1. Go to **Account settings** > **Integrations** > **OAuth**. 2. Click the **X** icon next to Slack and select **Unlink**. 3. In the same OAuth section, click **Link** to connect the official Slack app. Until you do this, the account-level Slack option will not appear. [![Link for the Slack app](/img/docs/dbt-platform/Link-your-Slack-Profile.png?v=2 "Link for the Slack app")](#)Link for the Slack app ##### Logged in to Slack If you're already logged in to Slack, the integration only requires allowing the app access. If you're a member of multiple workspaces, you can select the appropriate workspace from the dropdown menu in the upper right corner. [![Allow dbt access to Slack](/img/docs/dbt-platform/Allow-dbt-to-access-slack.png?v=2 "Allow dbt access to Slack")](#)Allow dbt access to Slack ##### Logged out If you're logged out or the Slack app/website is closed, you must authenticate before completing the integration. 1. Complete the field defining the Slack workspace you want to integrate with dbt. [![Define the workspace](/img/docs/dbt-platform/define-workspace.png?v=2 "Define the workspace")](#)Define the workspace 2. Sign in with an existing identity or use the email address and password. 3. Once you have authenticated successfully, accept the permissions. [![Allow dbt access to Slack](/img/docs/dbt-platform/accept-permissions.png?v=2 "Allow dbt access to Slack")](#)Allow dbt access to Slack ##### Configure Slack notifications Configure the Slack channel you want to receive job notifications from. 1. Select your profile icon and then click on **Notification settings**. 2. Select **Slack notifications** in the left sidebar. 3. From the first dropdown, select the **Notification channel** you want to receive the job run notifications. [![Example of the Notification channel dropdown](/img/docs/deploy/example-notification-slack-channels.png?v=2 "Example of the Notification channel dropdown")](#)Example of the Notification channel dropdown 4. From the second dropdown, select the **Environment** for the jobs you want to receive notifications about. 5. Click **Edit** to configure the Slack notification settings. Choose one or more of the run statuses for each job you want to receive notifications about. 6. When you're done with the settings, click **Save**. * To send alerts to another Slack channel, select another **Notification channel** from the dropdown, **Edit** those job notification settings, and **Save** the changes. * To set up alerts on jobs from a different environment, select another **Environment** from the dropdown, **Edit** those job notification settings, and **Save** the changes. [![Example of the Slack notifications page](/img/docs/deploy/example-slack-notification-settings-page.png?v=2 "Example of the Slack notifications page")](#)Example of the Slack notifications page That's it! Your Slack channel is now set up to receive dbt job notifications at the account level. This integration is now available throughout the account for all licensed users. ##### Migrating legacy Slack notification settings If you previously configured Slack notifications with the legacy integration, you may see a migration banner on the **Slack notifications** page. [![Example of the Slack migration banner](/img/docs/deploy/migrating-legacy-slack-notifications.png?v=2 "Example of the Slack migration banner")](#)Example of the Slack migration banner The banner appears when all of the following are true: * You have notification settings from a previous Slack integration. * Your account is connected to the dbt platform app. * You have not yet configured Slack notifications with the dbt platform app. Before migrating, you must unlink the legacy Slack integration and link the dbt platform app. Unlinking the legacy integration is a manual step, and only one Slack app can be linked at a time. info The dbt platform Slack app sends job notifications to *public* only channels in your workspace. Private channels aren't supported. 1. Click **Migrate settings** to copy your existing settings to the dbt platform app, including: * Your selected **Notification channel** and **Environment** * Your selected jobs * Your notification toggles (for example, **Succeeds**, **Warns**, **Fails**, and **Is canceled**) 2. Click **Dismiss** to hide the banner for your current session — it reappears on reload until migration completes. When migration succeeds, dbt hides the banner and refreshes your Slack notification settings. If migration fails, the banner remains so you can try again. [![Example of invite guidance for the dbt platform app](/img/docs/deploy/dbt-platform-slack-invite.png?v=2 "Example of invite guidance for the dbt platform app")](#)Example of invite guidance for the dbt platform app ##### Disable the Slack integration In this step, you'll disable the Slack integration and remove the account-level Slack credentials. You can always re-enable the integration by following the [Set up the Slack integration](#set-up-the-slack-integration-1) steps. 1. Select **Account settings** and on the **Integrations** page, scroll to the **OAuth** section. 2. Click the **X** icon (on the far right of the Slack integration) and click **Unlink**. * This removes the account-level Slack credentials. All Slack notifications that rely on the account-level integration will stop sending. * If any legacy, user-linked Slack integrations still exist, those notifications may continue until the legacy link is removed. We recommend migrating to the new account-level app and removing legacy links. #### Microsoft Teams notifications You can receive Microsoft Teams alerts for your dbt jobs by connecting your Teams account to the dbt platform, and configuring your notification preferences. dbt integrates with Teams through Microsoft Entra to authenticate securely when you link Teams and configure notification settings. ##### Prerequisites Before you begin: * You must have a dbt platform account * You have a Microsoft Teams account that you want to receive job notifications from. * Make sure you have permission to view the **Account integrations** and **Job notifications** pages in dbt platform. Once an account admin links the Microsoft Teams app, licensed users can configure job notifications if they have one of the following: * The **Account Admin**, **Owner**, or **Member** default [group](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#groups) * The [**Notification Manager**](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) permission set (Enterprise). This permission manages notifications across all projects without full Account Admin access ##### Set up Microsoft Teams To enable Microsoft Teams job notifications, complete the following sections: 1. [Link dbt platform account to Teams](#link-dbt-platform-account-to-teams) — A user-level connection that links an individual dbt platform account (or a dedicated service account) to a Microsoft Teams user profile within your tenant. 2. [Configure Teams notifications](#configure-teams-notifications) — Configures which Teams channels receive job notifications. 3. [Disable the Teams integration](#disable-the-teams-integration) (optional) — Remove or reset the connection between dbt platform and Microsoft Teams. ##### Link dbt platform account to Teams info You can link any Teams user account from your tenant, but we recommend creating a dedicated account just for posting dbt notifications. During the OAuth process, you’ll need to sign in to a Microsoft account to complete the integration. * If you’re logged into a single Microsoft account, the integration will complete automatically. * If you’re logged into multiple accounts (or none), you’ll be prompted to select or log in to one.  Image of the Microsoft account selection popup [![Example of the Microsoft account popup](/img/docs/deploy/pick-account.png?v=2 "Example of the Microsoft account popup")](#)Example of the Microsoft account popup To link your dbt platform account to Microsoft Teams: 1. In dbt platform, go to the **Account settings** page by clicking on your account name and selecting **Account settings**. 2. In the left sidebar, select **Integrations**. 3. Scroll to the **OAuth** section. 4. Next to **Teams** and click on the **Link** button. 5. After doing this, you’ll either be prompted to choose your Microsoft account before completing the setup, or return directly to the dbt platform with your Teams profile linked. 6. Your dbt platform account is now linked to Microsoft Teams! dbt will now add the **dbt-cloud-integration app** to your Microsoft Entra tenant. This app manages authentication requests and permissions securely. [![Example of the dbt-cloud-integration app overview](/img/docs/deploy/dbt-cloud-integrations.png?v=2 "Example of the dbt-cloud-integration app overview")](#)Example of the dbt-cloud-integration app overview * The current Entra app permissions are: * `profile` * `openid` * `offline_access` * `Team.ReadBasic.All` * `TeamsActivity.Send` * `ChannelMessage.Send` * `ChannelMessage.Read.All` * `Channel.ReadBasic.All` ##### Configure Teams notifications Once you’ve connected dbt platform and Teams, you can configure which Teams channels receive job notifications. The **Teams notifications** menu requires that you have an active integration with Teams on the account. info Currently, dbt only sends notifications to Teams channels (standard, shared, or private) that you belong to. 1. In the dbt platform, click your profile icon and select **Notification settings**. 2. Select **Teams notifications** in the left sidebar. 3. From the first dropdown, select the **Notification team** that you want to send notifications to. 4. From the second dropdown, select the **Notification channel** you want to send notifications to. * dbt platform only sends notifications to Teams channels (standard, shared, or private) that *you* belong to. 5. In the dropdown, choose the environment for the jobs you want to receive notifications about. 6. Click **Edit** on the top right to configure the Teams job notification settings and customize which job statuses trigger job notifications. 7. When finished, click **Save**. Your Teams channel is now set up to receive dbt job notifications! [![Example of the configure Teams notification page](/img/docs/deploy/configure-teams-notification.png?v=2 "Example of the configure Teams notification page")](#)Example of the configure Teams notification page ##### Disable the Teams integration Disabling and unlinking the Teams integration in the dbt platform removes it for the entire account. To disable it: 1. In the dbt platform, go to **Account settings**. 2. Click on **Integrations** and scroll down to **OAuth**. 3. On the far right of the **Teams** integration, click the **X** icon. 4. Confirm the unlinking by selecting **Unlink**. The Teams integration has been disabled. You can always re-enable the integration by following the [Set up Microsoft Teams](#set-up-microsoft-teams) steps. --- ### Job scheduler dbt platform info Use the [dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) to investigate and troubleshoot dbt job and run failures by asking the agent about recent failures, root causes, and fixes — powered by the `troubleshooting-dbt-job-errors` skill in dbt Agent Skills. The job scheduler is the backbone of running jobs in dbt, bringing power and simplicity to building data pipelines in both continuous integration and production contexts. The scheduler frees teams from having to build and maintain their own infrastructure, and ensures the timeliness and reliability of data transformations. The scheduler enables both cron-based and event-driven execution of dbt commands in the user’s data platform. Specifically, it handles: * Cron-based execution of dbt jobs that run on a predetermined cadence * Event-driven execution of dbt jobs that run based on the completion of another job ([trigger on job completion](https://docs.getdbt.com/docs/deploy/deploy-jobs.md#trigger-on-job-completion)) * Event-driven execution of dbt CI jobs triggered when a pull request is merged to the branch ([merge jobs](https://docs.getdbt.com/docs/deploy/merge-jobs.md)) * Event-driven execution of dbt jobs triggered by API * Event-driven execution of dbt jobs manually triggered by a user to **Run now** The scheduler handles various tasks including: * Queuing jobs * Creating temporary environments to run the dbt commands required for those jobs * Providing logs for debugging and remediation * Storing dbt artifacts for direct consumption/ingestion by the Discovery API The scheduler also: * Uses [dbt's Git repository caching](https://docs.getdbt.com/docs/platform/account-settings.md#git-repository-caching) to protect against third-party outages and improve job run reliability. [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") * Powers running dbt in staging and production environments, bringing ease and confidence to CI/CD workflows and enabling observability and governance in deploying dbt at scale. * Uses [Hybrid projects](https://docs.getdbt.com/docs/deploy/hybrid-projects.md) to upload dbt v1 artifacts into dbt for central visibility, cross-project referencing, and easier collaboration. [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") * Uses [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) to decide what needs to be rebuilt based on upstream data freshness and code changes. [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") #### Scheduler terms Familiarize yourself with these useful terms to help you understand how the job scheduler works. | Term | Definition | | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Scheduler | The dbt engine that powers job execution. The scheduler queues scheduled or API-triggered job runs, prepares an environment to execute job commands in your cloud data platform, and stores and serves logs and artifacts that are byproducts of run execution. | | Job | A collection of run steps, settings, and a trigger to invoke dbt commands against a project in the user's cloud data platform. | | Job queue | The job queue acts as a waiting area for job runs when they are scheduled or triggered to run; runs remain in queue until execution begins. More specifically, the Scheduler checks the queue for runs that are due to execute, ensures the run is eligible to start, and then prepares an environment with appropriate settings, credentials, and commands to begin execution. Once execution begins, the run leaves the queue. | | Over-scheduled job | A situation when a cron-scheduled job's run duration becomes longer than the frequency of the job’s schedule, resulting in a job queue that will grow faster than the scheduler can process the job’s runs. | | Deactivated job | A situation where a job has reached 100 consecutive failing runs or belongs to an inactive account. | | Prep time | The time dbt takes to create a short-lived environment to execute the job commands in the user's cloud data platform. Prep time varies most significantly at the top of the hour when the dbt Scheduler experiences a lot of run traffic. | | Run | A single, unique execution of a dbt job. | | Run slot | Run slots control the number of jobs that can run concurrently. Each running job occupies a run slot for the duration of the run. To view the number of run slots available in your plan, check out the [dbt pricing page](https://www.getdbt.com/pricing).<br /><br />Starter and Developer plans are limited to one project each. For additional projects or more run slots, consider upgrading to an [Enterprise-tier plan](https://www.getdbt.com/pricing/). | | Threads | When dbt builds a project's DAG, it tries to parallelize the execution by using threads. The [thread](https://docs.getdbt.com/docs/running-a-dbt-project/using-threads.md) count is the maximum number of paths through the DAG that dbt can work on simultaneously. The default thread count in a job is 4. | | Wait time | Amount of time that dbt waits before running a job, either because there are no available slots or because a previous run of the same job is still in progress. | #### Scheduler queue The scheduler queues a deployment job to be processed when it's triggered to run by a [set schedule](https://docs.getdbt.com/docs/deploy/deploy-jobs.md#schedule-days), [a job completed](https://docs.getdbt.com/docs/deploy/deploy-jobs.md#trigger-on-job-completion), an API call, or manual action. Before the job starts executing, the scheduler checks these conditions to determine if the run can start executing: * **Is there a run slot that's available on the account for use?** — If all run slots are occupied, the queued run will wait. The wait time is displayed in dbt. If there are long wait times, [upgrading to an Enterprise-tier plan](https://www.getdbt.com/contact/) can provide more run slots and allow for higher job concurrency. * **Does this same job have a run already in progress?** — The scheduler executes distinct runs of the same dbt job serially to avoid model build collisions. If there's a job already running, the queued job will wait, and the wait time will be displayed in dbt. If there is an available run slot and there isn't an actively running instance of the job, the scheduler will prepare the job to run in your cloud data platform. This prep involves readying a Kubernetes pod with the right version of dbt installed, setting environment variables, loading data platform credentials, and Git provider authorization, amongst other environment-setting tasks. The time it takes to prepare the job is displayed as **Prep time** in the UI. [![An overview of a dbt job run](/img/docs/dbt-platform/deployment/deploy-scheduler.png?v=2 "An overview of a dbt job run")](#)An overview of a dbt job run ##### Treatment of CI jobs When compared to deployment jobs, the scheduler behaves differently when handling [continuous integration (CI) jobs](https://docs.getdbt.com/docs/deploy/continuous-integration.md). It queues a CI job to be processed when it's triggered to run by a Git pull request, and the conditions the scheduler checks to determine if the run can start executing are also different: * **Will the CI run consume a run slot?** — CI runs don't consume run slots and will never block production runs. * **Does this same job have a run already in progress?** — CI runs can execute concurrently (in parallel). CI runs build into unique temporary schemas, and CI checks execute in parallel to help increase team productivity. Teammates never have to wait to get a CI check review. ##### Treatment of merge jobs When triggered by a *merged* Git pull request, the scheduler queues a [merge job](https://docs.getdbt.com/docs/deploy/merge-jobs.md) to be processed. * **Will the merge job run consume a run slot?** — Yes, merge jobs do consume run slots. * **Does this same job have a run already in progress?** — A merge job can only have one run in progress at a time. If there are multiple runs queued up, the scheduler will enqueue the most recent run and cancel all the other runs. If there is a run in progress, it will wait until the run completes before queuing the next run. #### Job memory In dbt, the setting to provision memory available to a job is defined at the account-level and applies to each job running in the account; the memory limit cannot be customized per job. If a running job reaches its memory limit, the run is terminated with a "memory limit error" message. Jobs consume a lot of memory in the following situations: * A high thread count was specified * Custom dbt macros attempt to load data into memory instead of pushing compute down to the cloud data platform * Having a job that generates dbt project documentation for a large and complex dbt project. * To prevent problems with the job running out of memory, we recommend generating documentation in a separate job that is set aside for that task and removing `dbt docs generate` from all other jobs. This is especially important for large and complex projects. Refer to [dbt architecture](https://docs.getdbt.com/docs/platform/about-platform/architecture.md) for an architecture diagram and to learn how the data flows. #### Run cancellation for over-scheduled jobs Scheduler won't cancel API-triggered jobs The scheduler will not cancel over-scheduled jobs triggered by the [API](https://docs.getdbt.com/docs/dbt-apis/overview.md). The dbt scheduler prevents too many job runs from clogging the queue by canceling unnecessary ones. If a job takes longer to run than its scheduled frequency, the queue will grow faster than the scheduler can process the runs, leading to an ever-expanding queue with runs that don’t need to be processed (called *over-scheduled jobs*). The scheduler prevents queue clog by canceling runs that aren't needed, ensuring there is only one run of the job in the queue at any given time. If a newer run is queued, the scheduler cancels any previously queued run for that job and displays an error message. [![The cancelled runs display an error message explaining why the run was cancelled and recommendations](/img/docs/dbt-platform/deployment/run-error-message.png?v=2 "The cancelled runs display an error message explaining why the run was cancelled and recommendations")](#)The cancelled runs display an error message explaining why the run was cancelled and recommendations To prevent over-scheduling, users will need to take action by either refactoring the job so it runs faster or modifying its [schedule](https://docs.getdbt.com/docs/deploy/deploy-jobs.md#schedule-days). #### Deactivation of jobs To reduce unnecessary resource consumption and reduce contention for run slots in your account, dbt will deactivate a [deploy job](https://docs.getdbt.com/docs/deploy/deploy-jobs.md) or a [CI job](https://docs.getdbt.com/docs/deploy/ci-jobs.md) if it reaches 100 consecutive failing runs. A banner containing this message is displayed when a job is deactivated: "Job has been deactivated due to repeated run failures. To reactivate, verify the job is configured properly and run manually or reenable any trigger". When this happens, scheduled and triggered-to-run jobs will no longer be enqueued. Jobs can also be deactivated when a dbt account is inactive. Account owners receive a warning after 90 days without account activity. If there is no activity for another 7 days, dbt deactivates jobs in the account. A banner containing this message is displayed when a job is deactivated because the account is inactive: "Job has been deactivated because the account is inactive. To reactivate, either log out and log back in to dbt, then wait up to 30 minutes for the system to reactivate the job, or manually edit and save the job." To reactivate a job deactivated due to repeated run failures, you can either: * Update the job's settings to fix the issue and save the job (recommended) * Perform a manual run by clicking **Run now** on the job's page To reactivate jobs deactivated because the account is inactive, either log out and log back in to dbt, then wait up to 30 minutes for the system to reactivate the job, or manually edit and save the job. #### FAQs I'm receiving a 'This run exceeded your account's run memory limits' error in my failed job If you're receiving a `This run exceeded your account's run memory limits` error in your failed job, it means that the job exceeded the [memory limits](https://docs.getdbt.com/docs/deploy/job-scheduler.md#job-memory) set for your account. All dbt accounts have a pod memory of 600Mib and memory limits are on a per run basis. They're typically influenced by the amount of result data that dbt has to ingest and process, which is small but can become bloated unexpectedly by project design choices. ##### Common reasons Some common reasons for higher memory usage are: * dbt run/build: Macros that capture large result sets from run query may not all be necessary and may be memory inefficient. * dbt docs generate: Source or model schemas with large numbers of tables (even if those tables aren't all used by dbt) cause the ingest of very large results for catalog queries. ##### Resolution There are various reasons why you could be experiencing this error but they are mostly the outcome of retrieving too much data back into dbt. For example, using the `run_query()` operations or similar macros, or even using database/schemas that have a lot of other non-dbt related tables/views. Try to reduce the amount of data / number of rows retrieved back into dbt by refactoring the SQL in your `run_query()` operation using `group`, `where`, or `limit` clauses. Additionally, you can also use a database/schema with fewer non-dbt related tables/views. Video example As an additional resource, check out [this example video](https://www.youtube.com/watch?v=sTqzNaFXiZ8), which demonstrates how to refactor the sample code by reducing the number of rows returned. If you've tried the earlier suggestions and are still experiencing failed job runs with this error about hitting the memory limits of your account, please [reach out to support](mailto:support@getdbt.com). We're happy to help! ##### Additional resources * [Blog post on how we shaved 90 mins off](https://docs.getdbt.com/blog/how-we-shaved-90-minutes-off-model) #### Related docs * [dbt architecture](https://docs.getdbt.com/docs/platform/about-platform/architecture.md#dbt-cloud-features-architecture) * [Job commands](https://docs.getdbt.com/docs/deploy/job-commands.md) * [Job notifications](https://docs.getdbt.com/docs/deploy/job-notifications.md) * [Webhooks](https://docs.getdbt.com/docs/deploy/webhooks.md) * [dbt continuous integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md) --- ### Jobs in the dbt platform dbt platform info Use the [dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) to investigate and troubleshoot dbt job and run failures by asking the agent about recent failures, root causes, and fixes — powered by the `troubleshooting-dbt-job-errors` skill in dbt Agent Skills. These are the available job types in dbt: * [Deploy jobs](https://docs.getdbt.com/docs/deploy/deploy-jobs.md) — Build production data assets. Runs on a schedule, by API, or after another job completes. * [Continuous integration (CI) jobs](https://docs.getdbt.com/docs/deploy/continuous-integration.md) — Test and validate code changes before merging. Triggered by commit to a PR or by API. * [Merge jobs](https://docs.getdbt.com/docs/deploy/merge-jobs.md) — Deploy merged changes into production. Runs after a successful PR merge or by API. * [State-aware jobs](https://docs.getdbt.com/docs/deploy/state-aware-about.md) — Intelligently decide what needs to be rebuilt based on source freshness, code, or upstream data changes. Rebuild models only if they are older than the specified interval. The following comparison table describes the behaviors of the different job types: | | **Deploy jobs** | **CI jobs** | **Merge jobs** | **State-aware jobs** | | ---------------------- | ------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | Purpose | Builds production data assets. | Builds and tests new code before merging changes into production. | Build merged changes into production or update state for deferral. | Trigger model builds and job runs only when source data is updated. | | Trigger types | Triggered by a schedule, API, or the successful completion of another job. | Triggered by a commit to a PR or by API. | Triggered by a successful merge into the environment's branch or by API. | Triggered when code, sources, or upstream data changes and at custom refresh intervals and for custom source freshness configurations | | Destination | Builds into a production database and schema. | Builds into a staging database and ephemeral schema, lived for the lifetime of the PR. | Builds into a production database and schema. | Builds into a production database and schema. | | Execution mode | Runs execute sequentially, so as to not have collisions on the underlying DAG. | Runs execute in parallel to promote team velocity. | Runs execute sequentially, so as to not have collisions on the underlying DAG. | | | Efficiency run savings | Detects over-scheduled jobs and cancels unnecessary runs to avoid queue clog. | Cancels existing runs when a newer commit is pushed to avoid redundant work. | N/A | Runs jobs and build models *only* when source data is updated or if models are older than what you specified in the project refresh interval | | State comparison | Only sometimes needs to detect state. | Almost always needs to compare state against the production environment to build on modified code and its dependents. | Almost always needs to compare state against the production environment to build on modified code and its dependents. | | | Job run duration | Limit is 24 hours. | Limit is 24 hours. | Limit is 24 hours. | Limit is 24 hours. | --- ### Joins Joins are a powerful part of MetricFlow and simplify the process of making all valid dimensions available for your metrics at query time, regardless of where they are defined in different semantic models. With Joins, you can also create metrics using (Applies to dbt v1.12 and later) simple metrics from different semantic models. Joins use `entities` defined in your semantic model configs as the join keys between tables. Assuming entities are defined in the semantic model, MetricFlow creates a graph using the semantic models as nodes and the join paths as edges to perform joins automatically. MetricFlow chooses the appropriate join type and avoids fan-out or chasm joins with other tables based on the entity types.  What are fan-out or chasm joins? * Fan-out joins are when one row in a table is joined to multiple rows in another table, resulting in more output rows than input rows. * Chasm joins are when two tables have a many-to-many relationship through an intermediate table, and the join results in duplicate or missing data. #### Types of joins Joins are auto-generated MetricFlow automatically generates the necessary joins to the defined semantic objects, eliminating the need for you to create new semantic models or configuration files. This section explains the different types of joins that can be used with entities and how to query them. Metricflow uses these specific join strategies: * Primarily uses left joins when joining `fct` and `dim` models. Left joins make sure all rows from the "base" table are retained, while matching rows are included from the joined table. * For queries that involve multiple `fct` models, MetricFlow uses full outer joins to ensure all data points are captured, even when some `dim` or `fct` models are missing in certain tables. * MetricFlow restricts the use of fan-out and chasm joins. Refer to [SQL examples](#sql-examples) for more information on how MetricFlow handles joins in practice. The following table identifies which joins are allowed based on specific entity types to prevent the creation of risky joins. This table primarily represents left joins unless otherwise specified. For scenarios involving multiple `fct` models, MetricFlow uses full outer joins. | entity type - Table A | entity type - Table B | Join type | | --------------------- | --------------------- | ------------------------ | | Primary | Primary | ✅ Left | | Primary | Unique | ✅ Left | | Primary | Foreign | ❌ Fan-out (Not allowed) | | Unique | Primary | ✅ Left | | Unique | Unique | ✅ Left | | Unique | Foreign | ❌ Fan-out (Not allowed) | | Foreign | Primary | ✅ Left | | Foreign | Unique | ✅ Left | | Foreign | Foreign | ❌ Fan-out (Not allowed) | ##### Semantic validation MetricFlow performs semantic validation by executing `explain` queries in the data platform to ensure that the generated SQL gets executed without errors. This validation includes: * Verifying that all referenced tables and columns exist. * Ensuring the data platform supports SQL functions, such as `date_diff(x, y)`. * Checking for ambiguous joins or paths in multi-hop joins. If validation fails, MetricFlow surfaces errors for users to address before executing the query. #### Example The following example uses two semantic models with a common entity and shows a MetricFlow query that requires a join between the two semantic models: `transactions` and `user_signup`. (Applies to dbt v1.12 and later) models/model.yml ```yaml models: - name: transactions_model semantic_model: enabled: true name: transactions columns: - name: id entity: name: transaction type: primary - name: user_id entity: name: user type: foreign metrics: - name: average_purchase_price type: simple agg: average expr: purchase_price - name: user_model semantic_model: enabled: true name: user_signup columns: - name: user_id entity: name: user type: primary - name: type dimension: type: categorical ``` * MetricFlow uses `user_id` as the join key to link two semantic models, `transactions` and `user_signup`. This allows you to query the `average_purchase_price` metric in the `transactions` semantic model, grouped by the `type` dimension in the `user_signup` semantic model. * Note that the `average_purchase_price` simple metric is defined in `transactions`, where `user_id` is a foreign entity. However, `user_signup` has `user_id` as a primary entity. * Since `user_id` is a foreign key in `transactions` and a primary key in `user_signup`, MetricFlow performs a left join where `transactions` joins `user_signup` to access the `average_purchase_price` simple metric defined in `transactions`. * To query dimensions from different semantic models, add a double underscore (or dunder) to the dimension name after joining the entity in your editing tool. The following query, `user_id__type` is included as a dimension using the `--group-by` flag (`type` is the dimension). ```yaml dbt sl query --metrics average_purchase_price --group-by metric_time,user_id__type # In dbt platform /> ``` ```yaml mf query --metrics average_purchase_price --group-by metric_time,user_id__type # In dbt v1 ``` ###### SQL examples These SQL examples show how MetricFlow handles both left join and full outer join scenarios in practice: ##### SQL example for left join Using the previous example for `transactions` and `user_signup` semantic models, this shows a left join between those two semantic models. ```sql select transactions.user_id, transactions.purchase_price, user_signup.type from transactions left outer join user_signup on transactions.user_id = user_signup.user_id where transactions.purchase_price is not null group by transactions.user_id, user_signup.type; ``` ##### SQL example for outer joins If you have multiple `fct` models, let's say `sales` and `returns`, MetricFlow uses full outer joins to ensure all data points are captured. This example shows a full outer join between the `sales` and `returns` semantic models. ```sql select sales.user_id, sales.total_sales, returns.total_returns from sales full outer join returns on sales.user_id = returns.user_id where sales.user_id is not null or returns.user_id is not null; ``` #### Multi-hop joins MetricFlow allows users to join (Applies to dbt v1.12 and later) simple metrics and dimensions across a graph of entities by moving from one table to another within a graph. This is referred to as "multi-hop join". MetricFlow can join up to three tables, supporting multi-hop joins with a limit of two hops. This does the following: * Enables complex data analysis without ambiguous paths. * Supports navigating through data models, like moving from `orders` to `customers` to `country` tables. While direct three-hop paths are limited to prevent confusion from multiple routes to the same data, MetricFlow does allow joining more than three tables if the joins don’t exceed two hops to reach a dimension. For example, if you have two models, `country` and `region`, where customers are linked to countries, which in turn are linked to regions, you can join all of them in a single SQL query and can dissect `orders` by `customer__country_country_name` but not by `customer__country__region_name`. ![Multi-Hop-Join](/assets/images/multihop-diagram-03171b81496cb0fd452d2c2f0b5e0ed3.png "Example schema for reference") (Applies to dbt v1.12 and later) Notice how the schema can be translated into the following three MetricFlow semantic models to create the metric 'Average purchase price by country' using the `purchase_price` simple metric from the sales table and the `country_name` dimension from the `country_dim` table. models/model.yml ```yaml models: - name: sales_model semantic_model: enabled: true name: sales agg_time_dimension: first_ordered_at columns: - name: id entity: type: primary - name: user_id entity: type: foreign - name: purchase_price # Column for the simple metric expression - name: first_ordered_at granularity: day dimension: type: time name: metric_time metrics: - name: average_purchase_price type: simple agg: avg expr: purchase_price - name: user_signup_model semantic_model: enabled: true name: user_signup columns: - name: user_id entity: type: primary - name: country_id entity: type: unique - name: signup_date granularity: day dimension: type: time - name: country_dim dimension: type: categorical - name: country_model semantic_model: enabled: true name: country columns: - name: country_id entity: type: primary - name: country_name dimension: type: categorical ``` ##### Query multi-hop joins To query dimensions *without* a multi-hop join involved, you can use the fully qualified dimension name with the syntax entity double underscore (dunder) dimension, like `entity__dimension`. For dimensions retrieved by a multi-hop join, you need to additionally provide the entity path as a list, like `user_id`. --- ### Materializations #### Overview Materializations are strategies for persisting dbt models in a warehouse. There are five types of materializations built into dbt. They are: * table * view * incremental * ephemeral * materialized view You can also configure [custom materializations](https://docs.getdbt.com/guides/create-new-materializations.md?step=1) in dbt. Custom materializations are a powerful way to extend dbt's functionality to meet your specific needs. For a detailed guide on materializations, refer to [Materializations best practices](https://docs.getdbt.com/best-practices/materializations/1-guide-overview.md). For information about data streaming, refer to [How to handle real-time data](https://docs.getdbt.com/best-practices/how-we-handle-real-time-data/1-intro.md). Learn by video! For video tutorials on Materializations, go to dbt Learn and check out the [Materializations fundamentals course](https://learn.getdbt.com/courses/materializations-fundamentals). #### Configuring materializations By default, dbt models are materialized as "views". Models can be configured with a different materialization by supplying the [`materialized` configuration](https://docs.getdbt.com/reference/resource-configs/materialized.md) parameter as shown in the following tabs. ##### Project file dbt\_project.yml ```yaml # The following dbt_project.yml configures a project that looks like this: # . # └── models # ├── csvs # │   ├── employees.sql # │   └── goals.sql # └── events # ├── stg_event_log.sql # └── stg_event_sessions.sql name: my_project version: 1.0.0 config-version: 2 models: my_project: events: # materialize all models in models/events as tables +materialized: table csvs: # this is redundant, and does not need to be set +materialized: view ``` ##### Model file Alternatively, materializations can be configured directly inside of the model SQL files. This can be useful if you are also setting \[Performance Optimization] configs for specific models (for example, [Redshift specific configurations](https://docs.getdbt.com/reference/resource-configs/redshift-configs.md) or [BigQuery specific configurations](https://docs.getdbt.com/reference/resource-configs/bigquery-configs.md)). models/events/stg\_event\_log.sql ```sql {{ config(materialized='table', sort='timestamp', dist='user_id') }} select * from ... ``` ##### Property file Materializations can also be configured in the model's `properties.yml` file. The following example shows the `table` materialization type. For a complete list of materialization types, refer to [materializations](https://docs.getdbt.com/docs/build/materializations.md#materializations). models/properties.yml ```yaml models: - name: events config: materialized: table ``` #### Materializations ##### View When using the `view` materialization, your model is rebuilt as a view on each run, via a `create view as` statement. * **Pros:** No additional data is stored, views on top of source data will always have the latest records in them. * **Cons:** Views that perform a significant transformation, or are stacked on top of other views, are slow to query. * **Advice:** * Generally start with views for your models, and only change to another materialization when you notice performance problems. * Views are best suited for models that do not do significant transformation, for example, renaming, or recasting columns. ##### Table When using the `table` materialization, your model is rebuilt as a table on each run, via a `create table as` statement. * **Pros:** Tables are fast to query * **Cons:** * Tables can take a long time to rebuild, especially for complex transformations * New records in underlying source data are not automatically added to the table * **Advice:** * Use the table materialization for any models being queried by BI tools, to give your end user a faster experience * Also use the table materialization for any slower transformations that are used by many downstream models ##### Incremental `incremental` models allow dbt to insert or update records into a table since the last time that model was run. * **Pros:** You can significantly reduce the build time by just transforming new records * **Cons:** Incremental models require extra configuration and are an advanced usage of dbt. Read more about using incremental models [here](https://docs.getdbt.com/docs/build/incremental-models.md). * **Advice:** * Incremental models are best for event-style data * Use incremental models when your `dbt run`s are becoming too slow (i.e. don't start with incremental models) ##### Ephemeral `ephemeral` models are not directly built into the database. Instead, dbt will interpolate the code from an ephemeral model into its dependent models using a common table expression (CTE). You can control the identifier for this CTE using a [model alias](https://docs.getdbt.com/docs/build/custom-aliases.md), but dbt will always prefix the model identifier with `__dbt__cte__`. * **Pros:** * You can still write reusable logic * Ephemeral models can help keep your data warehouse clean by reducing clutter (also consider splitting your models across multiple schemas by [using custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md)). * **Cons:** * You cannot select directly from this model. * [Operations](https://docs.getdbt.com/docs/build/hooks-operations.md#about-operations) (for example, macros called using [`dbt run-operation`](https://docs.getdbt.com/reference/commands/run-operation.md) cannot `ref()` ephemeral nodes) * Overuse of ephemeral materialization can also make queries harder to debug. * Ephemeral materialization doesn't support [model contracts](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md#where-are-contracts-supported). * **Advice:** Use the ephemeral materialization for: * Very light-weight transformations that are early on in your DAG * Are only used in one or two downstream models, and * Don't need to be queried directly ##### Materialized View The `materialized_view` materialization allows the creation and maintenance of materialized views in the target database. Materialized views are a combination of a view and a table, and serve use cases similar to incremental models. * **Pros:** * Materialized views combine the query performance of a table with the data freshness of a view * Materialized views operate much like incremental materializations, however they are usually able to be refreshed without manual interference on a regular cadence (depending on the database), forgoing the regular dbt batch refresh required with incremental materializations * `dbt run` on materialized views corresponds to a code deployment, just like views - **Cons:** * Due to the fact that materialized views are more complex database objects, database platforms tend to have fewer configuration options available; see your database platform's docs for more details * Materialized views may not be supported by every database platform * **Advice:** * Consider materialized views for use cases where incremental models are sufficient, but you would like the data platform to manage the incremental logic and refresh. ###### Configuration Change Monitoring This materialization makes use of the [`on_configuration_change`](https://docs.getdbt.com/reference/resource-configs/on_configuration_change.md) config, which aligns with the incremental nature of the namesake database object. This setting tells dbt to attempt to make configuration changes directly to the object when possible, as opposed to completely recreating the object to implement the updated configuration. Using `dbt-postgres` as an example, indexes can be dropped and created on the materialized view without the need to recreate the materialized view itself. ###### Scheduled Refreshes In the context of a `dbt run` command, materialized views should be thought of as similar to views. For example, a `dbt run` command is only needed if there is the potential for a change in configuration or sql; it's effectively a deploy action. By contrast, a `dbt run` command is needed for a table in the same scenarios *AND when the data in the table needs to be updated*. This also holds true for incremental and snapshot models, whose underlying relations are tables. In the table cases, the scheduling mechanism is either dbt or your local scheduler; there is no built-in functionality to automatically refresh the data behind a table. However, most platforms (Postgres excluded) provide functionality to configure automatically refreshing a materialized view. Hence, materialized views work similarly to incremental models with the benefit of not needing to run dbt to refresh the data. This assumes, of course, that auto refresh is turned on and configured in the model. info `dbt-snowflake` *does not* support materialized views, it uses Dynamic Tables instead. For details, refer to [Snowflake specific configurations](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#dynamic-tables). Beginning in v1.13, `dbt-snowflake` supports interactive tables, which are optimized for low-latency, interactive queries and can refresh automatically like dynamic tables. For details, refer to [Interactive tables](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#interactive-tables). #### Python materializations Python models support two materializations: * `table` * `incremental` Incremental Python models support all the same [incremental strategies](https://docs.getdbt.com/docs/build/incremental-strategy.md) as their SQL counterparts. The specific strategies supported depend on your adapter. Python models can't be materialized as `view` or `ephemeral`. Python isn't supported for non-model resource types (like tests and snapshots). For incremental models, like SQL models, you will need to filter incoming tables to only new rows of data: ##### Snowpark models/my\_python\_model.py ```python import snowflake.snowpark.functions as F def model(dbt, session): dbt.config(materialized = "incremental") df = dbt.ref("upstream_table") if dbt.is_incremental: # only new rows compared to max in current table max_from_this = f"select max(updated_at) from {dbt.this}" df = df.filter(df.updated_at >= session.sql(max_from_this).collect()[0][0]) # or only rows from the past 3 days df = df.filter(df.updated_at >= F.dateadd("day", F.lit(-3), F.current_timestamp())) ... return df ``` ##### PySpark models/my\_python\_model.py ```python import pyspark.sql.functions as F def model(dbt, session): dbt.config(materialized = "incremental") df = dbt.ref("upstream_table") if dbt.is_incremental: # only new rows compared to max in current table max_from_this = f"select max(updated_at) from {dbt.this}" df = df.filter(df.updated_at >= session.sql(max_from_this).collect()[0][0]) # or only rows from the past 3 days df = df.filter(df.updated_at >= F.date_add(F.current_timestamp(), F.lit(-3))) ... return df ``` **Note:** Incremental models are supported on BigQuery/Dataproc for the `merge` incremental strategy. The `insert_overwrite` strategy is not yet supported. --- ### MCP environment variables reference Local development This page is a condensed reference for all environment variables used by the self-hosted dbt MCP server. For full detail on each variable (including examples), see [Set up self-hosted MCP](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md). #### Self-hosted CLI These variables are required to use dbt commands through MCP. | Variable | Required | Description | | ------------------ | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | `DBT_PROJECT_DIR` | Required | Full path to your dbt project folder (the folder containing `dbt_project.yml`). | | `DBT_PATH` | Required | Full path to your dbt executable. Find it by running `which dbt` (macOS/Linux) or `where dbt` (Windows). | | `DBT_PROFILES_DIR` | Optional | Path to a custom `profiles.yml` directory. Defaults to `~/.dbt/`. | | `DBT_CLI_TIMEOUT` | Optional | Seconds before dbt commands time out. Defaults to `60`. Compile runs the whole project, so large projects may need a higher limit to avoid timeouts. | #### dbt platform These variables are required for dbt platform features (Semantic Layer, Discovery API, Admin API, SQL execution). | Variable | Required | Description | | ----------------- | ----------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `DBT_HOST` | Required | Your dbt platform [hostname](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). Accepts both `cloud.getdbt.com` (default) and `https://cloud.getdbt.com`.<br /><br />Include the full hostname with subdomain — for example, `DBT_HOST=abc123.us1.dbt.com`. You no longer need to separate the prefix using `MULTICELL_ACCOUNT_PREFIX` or `DBT_HOST_PREFIX`. | | `DBT_TOKEN` | Required | A service token or Personal Access Token (PAT).<br /><br />The `execute_sql` tool requires a PAT — service tokens *do not* work for that tool. | | `DBT_PROD_ENV_ID` | Required | Your production environment ID (numeric integer). | | `DBT_DEV_ENV_ID` | Required for `execute_sql` | Your development environment ID (numeric integer). | | `DBT_USER_ID` | Required for `execute_sql` | Your numeric user ID. | | `DBT_ACCOUNT_ID` | Required for Admin API and PAT-based auth | Your numeric account ID. Required when using a Personal Access Token (PAT) as your `DBT_TOKEN`. | Legacy prefix variables `MULTICELL_ACCOUNT_PREFIX` and `DBT_HOST_PREFIX` are legacy environment variables. They are backwards compatible — if you already have them set, they will continue to work. However, the recommended approach is to set the full hostname directly in `DBT_HOST` (for example, `DBT_HOST=abc123.us1.dbt.com`). See [Finding your IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md) for step-by-step instructions on locating each value. #### Tool configuration dbt MCP has two modes for controlling which tools are available: disable mode (default) and enable mode (allowlist). Toolsets may grow over time as new capabilities are added. The naming pattern generalizes to `DISABLE_{TOOLSET}=` and `DBT_MCP_ENABLE_{TOOLSET}=`, so new toolsets follow the same convention. For the latest list of toolset names, refer to the tables below or the [dbt-mcp README](https://github.com/dbt-labs/dbt-mcp). Which mode should I use? * **Disable mode:** Use when you want most tools available and only need to turn a few off. * **Enable mode:** Use when you want only a specific set of tools available. Do not mix both modes for the same toolset. For example avoid this pattern: ```bash # ❌ Mixing modes leads to unpredictable behavior DISABLE_SEMANTIC_LAYER=false DBT_MCP_ENABLE_SEMANTIC_LAYER=true ``` Also avoid empty `DBT_MCP_ENABLE_*=` lines as they activate enable mode and will silently disable all other toolsets. ##### Disable mode (default) All tools are available by default. To disable a toolset, set any of the following variables to `true` in your MCP client configuration: | Variable | Default | Description | | ----------------------------- | ------- | ----------------------------------------------------------------------------------------------- | | `DISABLE_DBT_CLI` | `false` | Disable local dbt CLI tools (dbt v1.x and v2). | | `DISABLE_SEMANTIC_LAYER` | `false` | Disable Semantic Layer tools. | | `DISABLE_DISCOVERY` | `false` | Disable Discovery API tools. | | `DISABLE_ADMIN_API` | `false` | Disable Admin API tools. | | `DISABLE_SQL` | `true` | SQL tools are disabled by default. Set to `false` to enable. | | `DISABLE_DBT_CODEGEN` | `true` | Codegen tools are disabled by default. Set to `false` to enable (requires dbt-codegen package). | | `DISABLE_LSP` | `false` | Disable LSP/dbt v2 tools. | | `DISABLE_MCP_SERVER_METADATA` | `true` | Server metadata tools are disabled by default. Set to `false` to enable. | | `DISABLE_TOOLS` | `""` | Comma-separated list of specific tool names to disable. | ##### Enable mode (allowlist) If any `DBT_MCP_ENABLE_*` variable is set, only the explicitly enabled toolsets are active: | Variable | Description | | ------------------------------- | -------------------------------------------------------------- | | `DBT_MCP_ENABLE_DBT_CLI` | Set to `true` to enable local dbt CLI tools (dbt v1.x and v2). | | `DBT_MCP_ENABLE_SEMANTIC_LAYER` | Set to `true` to enable Semantic Layer tools. | | `DBT_MCP_ENABLE_DISCOVERY` | Set to `true` to enable Discovery API tools. | | `DBT_MCP_ENABLE_ADMIN_API` | Set to `true` to enable Admin API tools. | | `DBT_MCP_ENABLE_SQL` | Set to `true` to enable SQL tools. | | `DBT_MCP_ENABLE_DBT_CODEGEN` | Set to `true` to enable codegen tools. | | `DBT_MCP_ENABLE_LSP` | Set to `true` to enable LSP/dbt v2 tools. | | `DBT_MCP_ENABLE_TOOLS` | Comma-separated list of specific tool names to enable. | ##### Precedence When multiple variables are set, they're evaluated in this order (highest priority first): 1. `DBT_MCP_ENABLE_TOOLS` (enable specific tools by name) 2. `DISABLE_TOOLS` (disable specific tools by name) 3. Toolset enable (`DBT_MCP_ENABLE_*=true`) 4. Toolset disable (`DISABLE_*=true`) 5. Default behavior #### Semantic Layer These variables control the behavior of Semantic Layer tools. | Variable | Default | Description | | -------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `DBT_MCP_SL_METRICS_RELATED_MAX` | `10` | Maximum number of metrics for which `list_metrics` also returns dimension and entity names inline, reducing the number of tool calls needed to answer data questions. When the metric count is at or below this value, dimensions and entities are embedded directly in the `list_metrics` response. When above this value, only metric names are returned and the LLM calls `get_dimensions`/`get_entities` separately. Set to `0` to always return metrics only and never inline dimension or entity data. | | `DBT_MCP_SL_MAX_RESPONSE_CHARS` | `16000` | Maximum character length of the CSV returned by `list_metrics`. Must be an integer `>= 0`. When the response would exceed this length *and* the metric count is above `DBT_MCP_SL_METRICS_RELATED_MAX`, the `description` and `metadata` columns are dropped from the response to save tokens. Set to `0` to disable trimming and always return the full response. | #### Logging and debugging | Variable | Default | Description | | ------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `DBT_MCP_LOG_LEVEL` | `INFO` | Log level for the MCP server. Options: `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`. Set to `DEBUG` temporarily to diagnose issues. | | `MCP_TRANSPORT` | `stdio` | Transport protocol. Set to `streamable-http` for local debugging only (for example, attaching a Python debugger). See the [dbt-mcp CONTRIBUTING guide](https://github.com/dbt-labs/dbt-mcp/blob/main/CONTRIBUTING.md). | --- ### MCP troubleshooting This page consolidates troubleshooting steps for all dbt MCP setups. For specific troubleshooting steps for your AI client, see the troubleshooting sections in the [Get started](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-oauth.md) and [Setup and config](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md) pages.  Can't find the uvx executable **Symptoms:** Error messages like `Could not connect to MCP server dbt-mcp`, `Error: spawn uvx ENOENT`, or `spawn uvx ENOENT` in your MCP client. **Cause:** Your MCP client (like Claude desktop) can't find `uvx` in its PATH because it starts with a limited environment. **Solution:** Use the full path to `uvx` in your configuration. 1. Find the full path: * macOS/Linux: Run `which uvx` in Terminal. * Windows: Run `where uvx` in Command Prompt or PowerShell. 2. Replace `"command": "uvx"` with the full path: ```json { "mcpServers": { "dbt": { "command": "/full/path/to/uvx", "args": ["dbt-mcp"], "env": { } } } } ``` Example on macOS with Homebrew: `"command": "/opt/homebrew/bin/uvx"` For VS Code (`mcp.json`), the same fix applies — replace `uvx` with its full path in the `command` field.  OAuth login not initiating **Symptoms:** The OAuth browser window never opens, or authentication appears to hang. **Cause:** dbt MCP uses a lock file to avoid repeated authentication. If a previous session left the lock file in place, it can block new authentication attempts. **Solution:** 1. Close your MCP client (Claude Desktop, Cursor, VS Code, etc.). 2. Delete the self-hosted dbt MCP config files: * macOS/Linux: `rm -f ~/.dbt/mcp.yml ~/.dbt/mcp.lock` * Windows: `Remove-Item -Force $env:USERPROFILE\.dbt\mcp.yml, $env:USERPROFILE\.dbt\mcp.lock` 3. Restart your client and try connecting again. If these steps don't resolve the issue, confirm that AI features are enabled on your account. An account admin can enable them in **Account settings** → **Edit** → toggle on **Enable account access to dbt Wizard features**. Refer to [Enable dbt Wizard](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md).  Server not starting **Symptoms:** The MCP server shows as disconnected or unavailable in your client. **Diagnosis:** Check the server logs: * **VS Code:** Open the Command Palette (`Ctrl/Cmd + Shift + P`) → `MCP: List Servers` → click the dbt server to see detailed logs. * **Claude Desktop:** Check `~/Library/Logs/Claude` (macOS) or `%APPDATA%\Claude\logs` (Windows). * **All clients:** Set `DBT_MCP_LOG_LEVEL=DEBUG` in your environment variables to get more verbose output. **Common causes:** * Missing or incorrect `DBT_PROJECT_DIR` or `DBT_PATH` — verify the paths exist and are absolute paths. * Invalid or expired authentication tokens — generate a new token and update your config. * Missing required environment variables for the toolset you're trying to use — see [Tool requirements](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#tool-requirements-at-a-glance).  Configuration not working in WSL (VS Code) **Symptoms:** VS Code MCP config works on other machines or native Windows but not in WSL. **Cause:** Local user settings are not applied in WSL environments. **Solution:** Configure MCP in the WSL-specific settings instead of local user settings: 1. Open the Command Palette → **Preferences: Open Remote Settings**. 2. Or select the **Remote** tab in the Settings editor. 3. Add your MCP server configuration there.  execute\_sql tool not working **Symptoms:** The `execute_sql` tool returns an authentication error or is unavailable. **Cause:** How you fix this depends on how you connected: * **Remote MCP with OAuth:** Sign in through your MCP client when prompted. You don't need a personal access token. * **Remote MCP with token-based auth:** You need a [personal access token (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md). Service tokens won't work. You also need `x-dbt-dev-environment-id` and `x-dbt-user-id` headers. The user ID must match the user who created the PAT. * **Local MCP:** Set `DBT_TOKEN` to a personal access token (PAT). Service tokens won't work. **Solution:** 1. If you use OAuth for remote MCP, confirm you completed sign-in and consent in your MCP client. 2. If you use token-based auth or local MCP, create a [personal access token (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) in **Account settings** → **API tokens** → **Personal tokens**. 3. For token-based remote MCP, add the PAT to the `Authorization` header as `Token YOUR_PAT`. For local MCP, set the PAT as your `DBT_TOKEN` value. 4. Also ensure `DBT_DEV_ENV_ID` and `DBT_USER_ID` are set (local MCP) or `x-dbt-dev-environment-id` and `x-dbt-user-id` headers are set (remote MCP). Refer to [Finding your IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md) for details.  Toolset unavailable or showing as disabled **Symptoms:** A toolset (Semantic Layer, Discovery, Admin API) is not available in your AI client even though you've configured credentials. **Cause:** Either the required variables are missing, or the toolset has been explicitly disabled. **Solution:** 1. Check that all required variables for the toolset are set — see [Tool requirements](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#tool-requirements-at-a-glance). 2. Check whether you have any `DISABLE_*` variables set to `true` that might be turning off the toolset. 3. If you're using enable mode (`DBT_MCP_ENABLE_*`), make sure the toolset you need is listed. 4. Set `DBT_MCP_LOG_LEVEL=DEBUG` to see which toolsets are active at startup.  Pasting full URLs instead of IDs **Symptoms:** Authentication errors, unexpected behavior, or the server failing to connect to the right environment. **Cause:** Environment variables like `DBT_PROD_ENV_ID`, `DBT_USER_ID`, and `DBT_ACCOUNT_ID` expect numeric integers, not full browser URLs. **Solution:** ```bash # ✅ Correct DBT_HOST=cloud.getdbt.com # https://cloud.getdbt.com also works DBT_PROD_ENV_ID=54321 DBT_USER_ID=123 # ❌ Wrong — IDs must be numeric, not full URLs DBT_PROD_ENV_ID=https://cloud.getdbt.com/deploy/12345/projects/67890/environments/54321 DBT_USER_ID=https://cloud.getdbt.com/settings/profile ``` See [Finding your IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md) for step-by-step instructions.  Multi-cell or static subdomain account connection issues **Symptoms:** Connection errors when your account URL includes a prefix (for example, `abc123.us1.dbt.com`). **Solution (as of v1.14.0):** Set `DBT_HOST` to the full hostname including the prefix. If you're using PAT-based auth, also set `DBT_ACCOUNT_ID`. ```bash # ✅ Correct DBT_HOST=abc123.us1.dbt.com DBT_ACCOUNT_ID=12345 # required for PAT-based auth ``` You no longer need to set `MULTICELL_ACCOUNT_PREFIX` or `DBT_HOST_PREFIX`. If you have these set from an older configuration, remove them. --- ### Measures Measures are deprecated in the new spec Heads up, measures have been deprecated in favor of simple metrics under the `metrics:` key. Migrate by converting each measure to a `type: simple` metric. For more info, check out [Migrate to the latest YAML spec](https://docs.getdbt.com/docs/build/latest-metrics-spec.md) and [upgrade to v2](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md). Measures are aggregations performed on columns in your model. They can be used as final metrics or as building blocks for more complex metrics. Measures have several inputs, which are described in the following table along with their field types. | Parameter | Description | Required | Type | | -------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------- | | [`name`](https://docs.getdbt.com/docs/build/measures.md#name) | Provide a name for the measure, which must be unique and can't be repeated across all semantic models in your dbt project. | Required | String | | [`description`](https://docs.getdbt.com/docs/build/measures.md#description) | Describes the calculated measure. | Optional | String | | [`agg`](https://docs.getdbt.com/docs/build/measures.md#aggregation) | dbt supports the following aggregations: `sum`, `max`, `min`, `average`, `median`, `count_distinct`, `percentile`, and `sum_boolean`. | Required | String | | [`expr`](https://docs.getdbt.com/docs/build/measures.md#expr) | Either reference an existing column in the table or use a SQL expression to create or derive a new one. | Optional | String | | [`non_additive_dimension`](https://docs.getdbt.com/docs/build/measures.md#non-additive-dimensions) | Non-additive dimensions can be specified for measures that cannot be aggregated over certain dimensions, such as bank account balances, to avoid producing incorrect results. | Optional | String | | `agg_params` | Specific aggregation properties, such as a percentile. | Optional | Dict | | `agg_time_dimension` | The time field. Defaults to the default agg time dimension for the semantic model. | Optional | String | | `label` | String that defines the display value in downstream tools. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). Available in dbt version 1.7 or higher. | Optional | String | | `create_metric` | Create a `simple` metric from a measure by setting `create_metric: True`. The `label` and `description` attributes will be automatically propagated to the created metric. Available in dbt version 1.7 or higher. | Optional | Boolean | | `config` | Use the [`config`](https://docs.getdbt.com/reference/resource-properties/config.md) property to specify configurations for your metric. Supports the [`meta`](https://docs.getdbt.com/reference/resource-configs/meta.md) property, nested under `config`. | Optional | | #### Measure spec An example of the complete YAML measures spec is below. The actual configuration of your measures will depend on the aggregation you're using. ```yaml semantic_models: - name: semantic_model_name ..rest of the semantic model config measures: - name: The name of the measure description: 'same as always' ## Optional agg: the aggregation type. expr: the field agg_params: 'specific aggregation properties such as a percentile' ## Optional agg_time_dimension: The time field. Defaults to the default agg time dimension for the semantic model. ## Optional non_additive_dimension: 'Use these configs when you need non-additive dimensions.' ## Optional config: Use the config property to specify configurations for your measure. ## Optional meta: {<dictionary>} Set metadata for a resource and organize resources. Accepts plain text, spaces, and quotes. ## Optional ``` ##### Name When you create a measure, you can either give it a custom name or use the `name` of the data platform column directly. If the measure's `name` differs from the column name, you need to add an `expr` to specify the column name. The `name` of the measure is used when creating a metric. Measure names must be unique across all semantic models in a project and can not be the same as an existing `entity` or `dimension` within that same model. ##### Description The description describes the calculated measure. It's strongly recommended you create verbose and human-readable descriptions in this field. ##### Aggregation The aggregation determines how the field will be aggregated. For example, a `sum` aggregation type over a granularity of `day` would sum the values across a given day. Supported aggregations include: | Aggregation types | Description | | ----------------- | ------------------------------------------ | | sum | Sum across the values | | min | Minimum across the values | | max | Maximum across the values | | average | Average across the values | | sum\_boolean | A sum for a boolean type | | count\_distinct | Distinct count of values | | median | Median (p50) calculation across the values | | percentile | Percentile calculation across the values. | ###### Percentile aggregation example If you're using the `percentile` aggregation, you must use the `agg_params` field to specify details for the percentile aggregation (such as what percentile to calculate and whether to use discrete or continuous calculations). ```yaml name: p99_transaction_value description: The 99th percentile transaction value expr: transaction_amount_usd agg: percentile agg_params: percentile: .99 use_discrete_percentile: False # False calculates the continuous percentile, True calculates the discrete percentile. ``` ###### Percentile across supported engine types The following table lists which SQL engine supports continuous, discrete, approximate, continuous, and approximate discrete percentiles. | | Cont. | Disc. | Approx. cont | Approx. disc | | ------------------------------------------------------------------------ | ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | | Snowflake | [Yes](https://docs.snowflake.com/en/sql-reference/functions/percentile_cont.html) | [Yes](https://docs.snowflake.com/en/sql-reference/functions/percentile_disc.html) | [Yes](https://docs.snowflake.com/en/sql-reference/functions/approx_percentile.html) (t-digest) | No | | Bigquery | No (window) | No (window) | [Yes](https://cloud.google.com/bigquery/docs/reference/standard-sql/functions-and-operators#approx_quantiles) | No | | Databricks | [Yes](https://docs.databricks.com/sql/language-manual/functions/percentile_cont.html) | [No](https://docs.databricks.com/sql/language-manual/functions/percentile_disc.html) | No | [Yes](https://docs.databricks.com/sql/language-manual/functions/approx_percentile.html) | | Redshift | [Yes](https://docs.aws.amazon.com/redshift/latest/dg/r_PERCENTILE_CONT.html) | No (window) | No | [Yes](https://docs.aws.amazon.com/redshift/latest/dg/r_APPROXIMATE_PERCENTILE_DISC.html) | | [Postgres](https://www.postgresql.org/docs/9.4/functions-aggregate.html) | Yes | Yes | No | No | | [DuckDB](https://duckdb.org/docs/sql/aggregates.html) | Yes | Yes | Yes (t-digest) | No | ##### Expr If the `name` you specified for a measure doesn't match a column name in your model, you can use the `expr` parameter instead. This allows you to use any valid SQL to manipulate an underlying column name into a specific output. The `name` parameter then serves as an alias for your measure. **Notes**: When using SQL functions in the `expr` parameter, **always use data platform-specific SQL**. This is because outputs may differ depending on your specific data platform. For Snowflake users For Snowflake users, if you use a week-level function in the `expr` parameter, it'll now return Monday as the default week start day based on ISO standards. If you have any account or session level overrides for the `WEEK_START` parameter that fixes it to a value other than 0 or 1, you will still see Monday as the week starts. If you use the `dayofweek` function in the `expr` parameter with the legacy Snowflake default of `WEEK_START = 0`, it will now return ISO-standard values of 1 (Monday) through 7 (Sunday) instead of Snowflake's legacy default values of 0 (Monday) through 6 (Sunday). ##### Model with different aggregations ```yaml semantic_models: - name: transactions description: A record of every transaction that takes place. Carts are considered multiple transactions for each sku. model: ref('schema.transactions') defaults: agg_time_dimension: transaction_date # --- entities --- entities: - name: transaction_id type: primary - name: customer_id type: foreign - name: store_id type: foreign - name: product_id type: foreign # --- measures --- measures: - name: transaction_amount_usd description: Total usd value of transactions expr: transaction_amount_usd agg: sum config: meta: used_in_reporting: true - name: transaction_amount_usd_avg description: Average usd value of transactions expr: transaction_amount_usd agg: average - name: transaction_amount_usd_max description: Maximum usd value of transactions expr: transaction_amount_usd agg: max - name: transaction_amount_usd_min description: Minimum usd value of transactions expr: transaction_amount_usd agg: min - name: quick_buy_transactions description: The total transactions bought as quick buy expr: quick_buy_flag agg: sum_boolean - name: distinct_transactions_count description: Distinct count of transactions expr: transaction_id agg: count_distinct - name: transaction_amount_avg description: The average value of transactions expr: transaction_amount_usd agg: average - name: transactions_amount_usd_valid # Notice here how we use expr to compute the aggregation based on a condition description: The total usd value of valid transactions only expr: case when is_valid = True then transaction_amount_usd else 0 end agg: sum - name: transactions description: The average value of transactions. expr: transaction_amount_usd agg: average - name: p99_transaction_value description: The 99th percentile transaction value expr: transaction_amount_usd agg: percentile agg_params: percentile: .99 use_discrete_percentile: False # False calculates the continuous percentile, True calculates the discrete percentile. - name: median_transaction_value description: The median transaction value expr: transaction_amount_usd agg: median # --- dimensions --- dimensions: - name: transaction_date type: time expr: date_trunc('day', ts) # expr refers to underlying column ts type_params: time_granularity: day - name: is_bulk_transaction type: categorical expr: case when quantity > 10 then true else false end ``` ##### Non-additive dimensions Some measures cannot be aggregated over certain dimensions, like time, because it could result in incorrect outcomes. Examples include bank account balances where it does not make sense to carry over balances month-to-month, and monthly recurring revenue where daily recurring revenue cannot be summed up to achieve monthly recurring revenue. You can specify non-additive dimensions to handle this, where certain dimensions are excluded from aggregation. To demonstrate the configuration for non-additive measures, consider a subscription table that includes one row per date of the registered user, the user's active subscription plan(s), and the plan's subscription value (revenue) with the following columns: * `date_transaction`: The daily date-spine. * `user_id`: The ID of the registered user. * `subscription_plan`: A column to indicate the subscription plan ID. * `subscription_value`: A column to indicate the monthly subscription value (revenue) of a particular subscription plan ID. Parameters under the `non_additive_dimension` will specify dimensions that the measure should not be aggregated over. | Parameter | Description | Field type | | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | | `name` | This will be the name of the time dimension (that has already been defined in the data source) that the measure should not be aggregated over. | Required | | `window_choice` | Choose either `min` or `max`, where `min` reflects the beginning of the time period and `max` reflects the end of the time period. | Required | | `window_groupings` | Provide the entities that you would like to group by. | Optional | ```yaml semantic_models: - name: subscriptions description: A subscription table with one row per date for each active user and their subscription plans. model: ref('your_schema.subscription_table') defaults: agg_time_dimension: subscription_date entities: - name: user_id type: foreign primary_entity: subscription dimensions: - name: subscription_date type: time expr: date_transaction type_params: time_granularity: day measures: - name: count_users description: Count of users at the end of the month expr: user_id agg: count_distinct non_additive_dimension: name: subscription_date window_choice: max - name: mrr description: Aggregate by summing all users' active subscription plans expr: subscription_value agg: sum non_additive_dimension: name: subscription_date window_choice: max - name: user_mrr description: Group by user_id to achieve each user's MRR expr: subscription_value agg: sum non_additive_dimension: name: subscription_date window_choice: max window_groupings: - user_id metrics: - name: mrr_metrics type: simple type_params: measure: mrr ``` We can query the semi-additive metrics using the following syntax: For dbt: ```bash dbt sl query --metrics mrr_by_end_of_month --group-by subscription__subscription_date__month --order subscription__subscription_date__month dbt sl query --metrics mrr_by_end_of_month --group-by subscription__subscription_date__week --order subscription__subscription_date__week ``` For dbt v1: ```bash mf query --metrics mrr_by_end_of_month --group-by subscription__subscription_date__month --order subscription__subscription_date__month mf query --metrics mrr_by_end_of_month --group-by subscription__subscription_date__week --order subscription__subscription_date__week ``` #### Dependencies (Applies to dbt v1.12 and later) Metric nodes will reflect dependencies on semantic models based on their simple metrics. However, dependencies based on filters should not be reflected in: * [dbt selection syntax](https://docs.getdbt.com/reference/node-selection/syntax.md) * Visualization of the DAG in dbt-docs and the [integrated development environment](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) (IDE). This is because metrics need to source nodes for their `depends_on` attribute from a few different places: * `RATIO` metrics depend on `numerator` and `denominator` metrics, while `DERIVED` metrics depend on `input_metrics`. * `SIMPLE` type metrics depend on their semantic\_model. For example, when you run the command `dbt list --select my_semantic_model+`, it will show you the metrics that belong to the specified semantic model. But there's a condition: Only the metrics that actually use simple metrics or derived metrics from that semantic model will be included in the list. In other words, if a metric only uses a dimension from the semantic model in its filters, it won't be considered as part of that semantic model. --- ### Merge jobs in dbt dbt platform | Starter, Enterprise You can set up a merge job to implement a continuous deployment (CD) workflow in dbt. The merge job triggers a dbt job to run when someone merges Git pull requests into production. This workflow creates a seamless development experience where changes made in code will automatically update production data. You can also use a merge job to refresh an environment’s `manifest.json` so downstream [CI jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md) stay fast. For example, run [`dbt compile`](https://docs.getdbt.com/reference/commands/compile.md) or, to avoid warehouse work entirely, [`dbt parse --no-partial-parse`](https://docs.getdbt.com/reference/commands/parse.md) in a dedicated environment—the same one your CI jobs defer to under **Compare changes against**, so the manifest those pull request runs compare against updates quickly. Use the same base branch that environment tracks (for example, `main` or a [custom branch](https://docs.getdbt.com/faqs/Environments/custom-branch-settings.md) such as `develop`). If merge also kicks off a long `dbt build`, consider a separate merge triggered job that runs *only* `dbt parse --no-partial-parse` so the manifest updates before the build finishes. By using CD in dbt, you can take advantage of deferral to build only the edited model and any downstream changes. With merge jobs, state will be updated almost instantly, always giving the most up-to-date state information in [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md). Triggering merge jobs in monorepos If you have a monorepo with several dbt projects, merging a single pull request in one of your projects will trigger jobs for all projects connected to the monorepo. To address this, you can use separate target branches per project (for example, `main-project-a`, `main-project-b`) to separate CI triggers. #### Prerequisites * You have a dbt account. * You have set up a [connection with your Git provider](https://docs.getdbt.com/docs/platform/git/configure-git.md). This integration lets dbt run jobs on your behalf for job triggering. * If you're using a native [GitLab](https://docs.getdbt.com/docs/platform/git/connect-gitlab.md) integration, you need a paid or self-hosted account that includes support for GitLab webhooks and [project access tokens](https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html). If you're using GitLab Free, merge requests will trigger CI jobs but CI job status updates (success or failure of the job) will not be reported back to GitLab. * For deferral (which is the default), make sure there has been at least one successful job run in the environment you defer to. #### Set up job trigger on Git merge 1. On your deployment environment page, click **Create job** > **Merge job**. 2. Options in the **Job settings** section: * **Job name** — Specify the name for the merge job. * **Description** — Provide a description about the job. * **Environment** — By default, it’s set to the environment you created the job from. 3. In the **Git trigger** section, the **Run on merge** option is enabled by default. Every time a PR merges (to a base branch configured in the environment) in your Git repo, this job will get triggered to run. 4. Options in the **Execution settings** section: * **Commands** — By default, it includes the `dbt build --select state:modified+` command. This informs dbt to build only new or changed models and their downstream dependents. Importantly, state comparison can only happen when there is a deferred environment selected to compare state to. Click **Add command** to add more [commands](https://docs.getdbt.com/docs/deploy/job-commands.md) that you want to be invoked when this job runs. * **Compare changes against** — By default, it's set to compare changes against the environment you created the job from. This option allows dbt to check the state of the code in the PR against the code running in the deferred environment, so as to only check the modified code, instead of building the full table or the entire DAG. To change the default settings, you can select **No deferral**, **This job** for self-deferral, or choose a different environment. * **Enable dbt State** [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") — [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) reduces unnecessary model rebuilds by reusing nodes when neither the logic nor the data has changed. For more details, refer to [Setting up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) and [Enabling dbt State on individual jobs](https://docs.getdbt.com/docs/deploy/dbt-state-enable-jobs.md). 5. (optional) Options in the **Advanced settings** section: * **Environment variables** — Define [environment variables](https://docs.getdbt.com/docs/build/environment-variables.md) to customize the behavior of your project when this job runs. * **Target name** — Define the [target name](https://docs.getdbt.com/docs/build/custom-target-names.md). Similar to environment variables, this option lets you customize the behavior of the project. * **Run timeout** — Cancel this job if the run time exceeds the timeout value. * **dbt version** — By default, it’s set to inherit the [dbt version](https://docs.getdbt.com/docs/dbt-versions.md) from the environment. dbt Labs strongly recommends that you don't change the default setting. This option to change the version at the job level is useful only when you upgrade a project to the next dbt version; otherwise, mismatched versions between the environment and job can lead to confusing behavior. * **Threads** — By default, it’s set to 4 [threads](https://docs.getdbt.com/docs/local/profiles.yml.md#understanding-threads). Increase the thread count to increase model execution concurrency. [![Example of creating a merge job](/img/docs/dbt-platform/using-dbt-platform/example-create-merge-job.png?v=2 "Example of creating a merge job")](#)Example of creating a merge job #### Verify push events in Git Merge jobs require push events so make sure they've been enabled in your Git provider, especially if you have an already-existing Git integration. However, for a new integration setup, you can skip this check since push events are typically enabled by default.  GitHub example The following is a GitHub example of when the push events are already set: [![Example of the Pushes option enabled in the GitHub settings](/img/docs/dbt-platform/using-dbt-platform/example-github-push-events.png?v=2 "Example of the Pushes option enabled in the GitHub settings")](#)Example of the Pushes option enabled in the GitHub settings  GitLab example The following is a GitLab example of when the push events are already set: [![Example of the Push events option enabled in the GitLab settings](/img/docs/dbt-platform/using-dbt-platform/example-gitlab-push-events.png?v=2 "Example of the Push events option enabled in the GitLab settings")](#)Example of the Push events option enabled in the GitLab settings  Azure DevOps example The following is an example of creating a new **Code pushed** trigger in Azure DevOps. Create a new service hooks subscription when code pushed events haven't been set: [![Example of creating a new trigger to push events in Azure Devops](/img/docs/dbt-platform/using-dbt-platform/example-azuredevops-new-event.png?v=2 "Example of creating a new trigger to push events in Azure Devops")](#)Example of creating a new trigger to push events in Azure Devops --- ### MetricFlow commands MetricFlow allows you to define and query metrics in your dbt project in [dbt platform](https://docs.getdbt.com/docs/platform/about-develop-dbt.md) or [self-hosted dbt](https://docs.getdbt.com/docs/local/install-dbt.md) locally. To experience the power of the universal [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) and dynamically query those metrics in downstream tools, you'll need a dbt [Starter, Enterprise, or Enterprise+](https://www.getdbt.com/pricing/) account. Once you define metrics in your dbt project, you can query metrics, dimensions, and dimension values, and validate your configs using the MetricFlow commands, available across [dbt versions](https://docs.getdbt.com/docs/introduction.md). MetricFlow is compatible with Python versions 3.8, 3.9, 3.10, 3.11, and 3.12. #### MetricFlow (Applies to dbt v1.12 and later) MetricFlow is a Python library that allows you to define and query metrics in your dbt project. You can use MetricFlow to query metrics in your dbt project two ways: in the dbt platform or locally with self-hosted dbt, where you install and manage the engine yourself. ##### MetricFlow in dbt platform This section applies to dbt platform users running either the v1 or v2 engine, where commands and validations execute remotely in dbt platform. * Run MetricFlow commands using the `dbt sl` prefix in the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) or [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) or using the [VS Code extension](https://docs.getdbt.com/docs/install-dbt-extension.md). * For CLI or VS Code/Cursor users, MetricFlow commands are embedded, which means you can immediately run them once you install the [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) or [VS Code extension](https://docs.getdbt.com/docs/install-dbt-extension.md) and don't need to install MetricFlow separately. * Using MetricFlow with dbt platform doesn't require you to manage versioning — your dbt account will automatically manage the versioning. * dbt jobs support the `dbt sl validate` command to [automatically test your semantic nodes](https://docs.getdbt.com/docs/deploy/ci-jobs.md#semantic-validations-in-ci). You can also add MetricFlow validations with your Git provider (such as GitHub Actions) by installing MetricFlow (`python -m pip install metricflow`). This allows you to run MetricFlow commands as part of your continuous integration checks on PRs. ##### MetricFlow with self-hosted dbt This section applies to local self-hosted dbt users who aren't on dbt platform. You can install [MetricFlow](https://github.com/dbt-labs/metricflow#getting-started) from [PyPI](https://pypi.org/project/dbt-metricflow/). You need to use `pip` to install MetricFlow on Windows or Linux operating systems: 1. Create or activate your virtual environment `python -m venv venv`. 2. Run `pip install dbt-metricflow`. * You can install MetricFlow using PyPI as an extension of your dbt adapter in the command line. To install the adapter, run `python -m pip install "dbt-metricflow[adapter_package_name]"` and add the adapter name at the end of the command. For example, for a Snowflake adapter, run `python -m pip install "dbt-metricflow[dbt-snowflake]"`. **Note**, you'll need to manage versioning yourself: between dbt, your adapter, and MetricFlow if you execute with v1, or between dbt and MetricFlow if you execute with v2. Something to note, MetricFlow `mf` commands return an error if you have a Metafont latex package installed. To run `mf` commands, uninstall the package. #### MetricFlow commands Use MetricFlow commands to retrieve metadata and query metrics. The following table lists the compatibility matrix for MetricFlow commands and where you can run them. | Development setup | Engine | Hosted on | Prefix | Notes | | ------------------------------------------------ | ------------ | ------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Studio IDE/dbt platform CLI or VS Code extension | dbt v2 or v1 | dbt platform | `dbt sl` | Remote execution; Platform manages versions.<br />VS Code extension users must have a [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md) file with a valid token to connect to dbt platform | | Self-hosted<br />(no dbt platform project) | dbt v2 or v1 | Local machine | `mf` | Install and manage MetricFlow locally. | * If you’re using v2 with dbt platform and have a [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md) file with a valid token to connect to dbt platform, run MetricFlow commands using the `dbt sl` prefix. * This allows you to interact with metrics that are executed remotely on dbt platform (for example, from the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) or [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md)). * If you’re using [v2 CLI](https://docs.getdbt.com/docs/local/install-dbt.md?version=2) and aren't connected to dbt platform, install MetricFlow separately and use the `mf` prefix to run commands locally. * If you’re using v1 locally without v2, run MetricFlow commands using the `mf` prefix. ##### Commands for dbt platform This section applies to dbt platform users running the v2 or v1 engine where commands and validations execute remotely in dbt platform. * Use the `dbt sl` prefix before the command name to execute them in the dbt platform (Studio IDE, VS Code/Cursor, dbt platform CLI) (like `dbt sl list metrics` to list all metrics). * For dbt platform users developing with a CLI or an editor (like VS Code), run the `dbt sl --help` command in the terminal to view a complete list of the MetricFlow commands and flags. * The following table lists the commands compatible with dbt platform (Studio IDE, VS Code/Cursor, dbt platform CLI) powered by the v2 or v1 engine: | Command | Description | Studio IDE | dbt platform CLI | VS Code/Cursor | | ------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ---------------- | -------------- | | [`list metrics`](#list-metrics) | Lists metrics with dimensions. | ✅ | ✅ | ✅ | | [`list dimensions`](#list) | Lists unique dimensions for metrics. | ✅ | ✅ | ✅ | | [`list dimension-values`](#list-dimension-values) | List dimensions with metrics. | ✅ | ✅ | ✅ | | [`list entities`](#list-entities) | Lists all unique entities. | ✅ | ✅ | ✅ | | [`list saved-queries`](#list-saved-queries) | Lists available saved queries. Use the `--show-exports` flag to display each export listed under a saved query or `--show-parameters` to show the full query parameters each saved query uses. | ✅ | ✅ | ✅ | | [`query`](#query) | Query metrics, saved queries, and dimensions you want to see in the command line interface. Refer to [query examples](#query-examples) to query metrics and dimensions (such as querying metrics, using the `where` filter, adding an `order`, and more). | ✅ | ✅ | ✅ | | [`validate`](#validate) | Validates semantic model configurations. | ✅ | ✅ | ✅ | | [`export`](#export) | Runs exports for a singular saved query for testing and generating exports in your development environment. You can also use the `--select` flag to specify particular exports from a saved query. | ❌ | ✅ | ✅ | | [`export-all`](#export-all) | Runs exports for multiple saved queries at once, saving time and effort. | ❌ | ✅ | ✅ | Run dbt parse to reflect metric changes When you make changes to metrics, make sure to run `dbt parse` at a minimum to update the Semantic Layer. This updates the `semantic_manifest.json` file(Applies to dbt v1.12 and later) and `osi_document.json`, reflecting your changes when querying metrics. By running `dbt parse`, you won't need to rebuild all the models.  How can I query or preview metrics with the dbt platform CLI? Check out the following video for a short video demo of how to query or preview metrics with the dbt platform CLI: ##### Commands for local self-hosted dbt This section applies to local self-hosted users who aren't on dbt platform, whether you execute with v1 or v2. Commands and validations execute locally and use the `mf` prefix before the command name to execute them. For example, to list all metrics, run `mf list metrics`. * [`list metrics`](#list-metrics) — Lists metrics with dimensions. * [`list dimensions`](#list) — Lists unique dimensions for metrics. * [`list dimension-values`](#list-dimension-values) — List dimensions with metrics. * [`list entities`](#list-entities) — Lists all unique entities. * [`validate-configs`](#validate-configs) — Validates semantic model configurations. * [`health-checks`](#health-checks) — Performs data platform health check. * [`tutorial`](#tutorial) — Dedicated MetricFlow tutorial to help get you started. * [`query`](#query) — Query metrics and dimensions you want to see in the command line interface. Refer to [query examples](#query-examples) to help you get started. #### List metrics This command lists the metrics with their available dimensions: ```bash dbt sl list metrics <metric_name> # For dbt platform users (dbt v1 or v2) mf list metrics <metric_name> # For self-hosted users (dbt v1 or v2) Options: --search TEXT Filter available metrics by this search term --show-all-dimensions Show all dimensions associated with a metric. --help Show this message and exit. ``` #### List dimensions This command lists all unique dimensions for a metric or multiple metrics. It displays only common dimensions when querying multiple metrics: ```bash dbt sl list dimensions --metrics <metric_name> # For dbt platform users (dbt v1 or v2) mf list dimensions --metrics <metric_name> # For self-hosted users (dbt v1 or v2) Options: --metrics SEQUENCE List dimensions by given metrics (intersection). Ex. --metrics bookings,messages --help Show this message and exit. ``` #### List dimension-values This command lists all dimension values with the corresponding metric: ```bash dbt sl list dimension-values --metrics <metric_name> --dimension <dimension_name> # For dbt platform users (dbt v1 or v2) mf list dimension-values --metrics <metric_name> --dimension <dimension_name> # For self-hosted users (dbt v1 or v2) Options: --dimension TEXT Dimension to query values from [required] --metrics SEQUENCE Metrics that are associated with the dimension [required] --end-time TEXT Optional iso8601 timestamp to constraint the end time of the data (inclusive) *Not available in the dbt platform/dbt v2 yet --start-time TEXT Optional iso8601 timestamp to constraint the start time of the data (inclusive) *Not available in the dbt platform/dbt v2 yet --help Show this message and exit. ``` #### List entities This command lists all unique entities: ```bash dbt sl list entities --metrics <metric_name> # For dbt platform users (dbt v1 or v2) mf list entities --metrics <metric_name> # For self-hosted users (dbt v1 or v2) Options: --metrics SEQUENCE List entities by given metrics (intersection). Ex. --metrics bookings,messages --help Show this message and exit. ``` #### List saved queries This command lists all available saved queries: ```bash dbt sl list saved-queries # For dbt platform users (dbt v1 or v2) ``` You can also add the `--show-exports` flag (or option) to show each export listed under a saved query: ```bash dbt sl list saved-queries --show-exports # For dbt platform users (dbt v1 or v2) ``` **Output** ```bash dbt sl list saved-queries --show-exports The list of available saved queries: - new_customer_orders exports: - Export(new_customer_orders_table, exportAs=TABLE) - Export(new_customer_orders_view, exportAs=VIEW) - Export(new_customer_orders, alias=orders, schemas=customer_schema, exportAs=TABLE) ``` #### Validate The following command performs validations against the defined semantic model configurations. * For dbt v2 and dbt platform CLI users with a valid [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md): ```bash dbt sl validate ``` * For self-hosted users (dbt v1 or v2): ```bash mf validate-configs ``` ```bash Options: --timeout # dbt platform only Optional timeout for data warehouse validation in dbt platform. --dw-timeout INTEGER # dbt v1 only Optional timeout for data warehouse validation steps. Default None. --skip-dw # dbt v1 only Skips the data warehouse validations. --show-all # dbt v1 only Prints warnings and future errors. --verbose-issues # dbt v1 only Prints extra details about issues. --semantic-validation-workers INTEGER # dbt v1 only Uses specified number of workers for large configs. --help Show this message and exit. ``` #### Health checks The following command performs a health check against the data platform you provided in the configs. Note, in dbt, the `health-checks` command isn't required since it uses dbt's credentials to perform the health check. ```bash mf health-checks # For self-hosted users (dbt v1 or v2) ``` #### Tutorial Follow the dedicated MetricFlow tutorial to help you get started: ```bash mf tutorial # For self-hosted users (dbt v1 or v2) ``` #### Query Create a new query with MetricFlow and execute it against your data platform. The query returns the following result: ```bash dbt sl query --metrics <metric_name> --group-by <dimension_name> # For dbt platform users (dbt v1 or v2) dbt sl query --saved-query <name> # For dbt platform users (dbt v1 or v2) mf query --metrics <metric_name> --group-by <dimension_name> # For self-hosted users (dbt v1 or v2) Options: --metrics SEQUENCE Syntax to query single metrics: --metrics metric_name For example, --metrics bookings To query multiple metrics, use --metrics followed by the metric names, separated by commas without spaces. For example, --metrics bookings,messages --group-by SEQUENCE Syntax to group by single dimension/entity: --group-by dimension_name For example, --group-by ds For multiple dimensions/entities, use --group-by followed by the dimension/entity names, separated by commas without spaces. For example, --group-by ds,org --end-time TEXT Optional iso8601 timestamp to constraint the end time of the data (inclusive). *Not available in the dbt platform/dbt v2 yet --start-time TEXT Optional iso8601 timestamp to constraint the start time of the data (inclusive) *Not available in the dbt platform/dbt v2 yet --where TEXT SQL-like where statement provided as a string and wrapped in quotes. All filter items must explicitly reference fields or dimensions that are part of your model. To query a single statement: ---where "{{ Dimension('order_id__revenue') }} > 100" To query multiple statements: --where "{{ Dimension('order_id__revenue') }} > 100" --where "{{ Dimension('user_count') }} < 1000" # make sure to wrap each statement in quotes To add a dimension filter, use the `Dimension()` template wrapper to indicate that the filter item is part of your model. Refer to the FAQ for more info on how to do this using a template wrapper. --limit TEXT Limit the number of rows out using an int or leave blank for no limit. For example: --limit 100 --order-by SEQUENCE Specify metrics, dimension, or group bys to order by. Add the `-` prefix to sort query in descending (DESC) order. Leave blank for ascending (ASC) order. For example, to sort metric_time in DESC order: --order-by -metric_time To sort metric_time in ASC order and revenue in DESC order: --order-by metric_time,-revenue --csv FILENAME Provide filepath for data frame output to csv --compile (dbt platform/dbt v2) In the query output, show the query that was --explain (dbt v1) executed against the data warehouse --show-dataflow-plan Display dataflow plan in explain output --display-plans Display plans (such as metric dataflow) in the browser --decimals INTEGER Choose the number of decimal places to round for the numerical values --show-sql-descriptions Shows inline descriptions of nodes in displayed SQL --help Show this message and exit. ``` #### Query examples This section shares various types of query examples that you can use to query metrics and dimensions. The query examples listed are: * [Query metrics](#query-metrics) * [Query dimensions](#query-dimensions) * [Add `order`/`limit` function](#add-orderlimit) * [Add `where` clause](#add-where-clause) * [Filter by time](#filter-by-time) * [Query saved queries](#query-saved-queries) ##### Query metrics Use the example to query multiple metrics by dimension and return the `order_total` and `users_active` metrics by `metric_time.` **Query** ```bash dbt sl query --metrics order_total,users_active --group-by metric_time # For dbt platform users (dbt v1 or v2) mf query --metrics order_total,users_active --group-by metric_time # For self-hosted users (dbt v1 or v2) ``` **Result** ```bash ✔ Success 🦄 - query completed after 1.24 seconds | METRIC_TIME | ORDER_TOTAL | |:--------------|---------------:| | 2017-06-16 | 792.17 | | 2017-06-17 | 458.35 | | 2017-06-18 | 490.69 | | 2017-06-19 | 749.09 | | 2017-06-20 | 712.51 | | 2017-06-21 | 541.65 | ``` ##### Query dimensions You can include multiple dimensions in a query. For example, you can group by the `is_food_order` dimension to confirm if orders were for food or not. Note that when you query a dimension, you need to specify the primary entity for that dimension. In the following example, the primary entity is `order_id`. **Query** ```bash dbt sl query --metrics order_total --group-by order_id__is_food_order # For dbt platform users (dbt v1 or v2) mf query --metrics order_total --group-by order_id__is_food_order # For self-hosted users (dbt v1 or v2) ``` **Result** ```bash Success 🦄 - query completed after 1.70 seconds | METRIC_TIME | IS_FOOD_ORDER | ORDER_TOTAL | |:--------------|:----------------|---------------:| | 2017-06-16 | True | 499.27 | | 2017-06-16 | False | 292.90 | | 2017-06-17 | True | 431.24 | | 2017-06-17 | False | 27.11 | | 2017-06-18 | True | 466.45 | | 2017-06-18 | False | 24.24 | | 2017-06-19 | False | 300.98 | | 2017-06-19 | True | 448.11 | ``` ##### Add order/limit You can add order and limit functions to filter and present the data in a readable format. The following query limits the data set to 10 records and orders them by `metric_time`, descending. Note that using the `-` prefix will sort the query in descending order. Without the `-` prefix sorts the query in ascending order. Note that when you query a dimension, you need to specify the primary entity for that dimension. In the following example, the primary entity is `order_id`. **Query** ```bash # For dbt platform users (dbt v1 or v2) dbt sl query --metrics order_total --group-by order_id__is_food_order --limit 10 --order-by -metric_time # For self-hosted users (dbt v1 or v2) mf query --metrics order_total --group-by order_id__is_food_order --limit 10 --order-by -metric_time ``` **Result** ```bash ✔ Success 🦄 - query completed after 1.41 seconds | METRIC_TIME | IS_FOOD_ORDER | ORDER_TOTAL | |:--------------|:----------------|---------------:| | 2017-08-31 | True | 459.90 | | 2017-08-31 | False | 327.08 | | 2017-08-30 | False | 348.90 | | 2017-08-30 | True | 448.18 | | 2017-08-29 | True | 479.94 | | 2017-08-29 | False | 333.65 | | 2017-08-28 | False | 334.73 | ``` ##### Add where clause You can further filter the data set by adding a `where` clause to your query. The following example shows you how to query the `order_total` metric, grouped by `is_food_order` with multiple `where` statements (orders that are food orders and orders from the week starting on or after Feb 1st, 2024). **Query** ```bash # For dbt platform users (dbt v1 or v2) dbt sl query --metrics order_total --group-by order_id__is_food_order --where "{{ Dimension('order_id__is_food_order') }} = True" --where "{{ TimeDimension('metric_time', 'week') }} >= '2024-02-01'" # For self-hosted users (dbt v1 or v2) mf query --metrics order_total --group-by order_id__is_food_order --where "{{ Dimension('order_id__is_food_order') }} = True" --where "{{ TimeDimension('metric_time', 'week') }} >= '2024-02-01'" ``` Notes: * The type of dimension changes the syntax you use. So if you have a date field, use `TimeDimension` instead of `Dimension`. * When you query a dimension, you need to specify the primary entity for that dimension. In the example just shared, the primary entity is `order_id`. **Result** ```bash ✔ Success 🦄 - query completed after 1.06 seconds | METRIC_TIME | IS_FOOD_ORDER | ORDER_TOTAL | |:--------------|:----------------|---------------:| | 2017-08-31 | True | 459.90 | | 2017-08-30 | True | 448.18 | | 2017-08-29 | True | 479.94 | | 2017-08-28 | True | 513.48 | | 2017-08-27 | True | 568.92 | | 2017-08-26 | True | 471.95 | | 2017-08-25 | True | 452.93 | | 2017-08-24 | True | 384.40 | | 2017-08-23 | True | 423.61 | | 2017-08-22 | True | 401.91 | ``` ##### Filter by time To filter by time, there are dedicated start and end time options. Using these options to filter by time allows MetricFlow to further optimize query performance by pushing down the where filter when appropriate. Note that when you query a dimension, you need to specify the primary entity for that dimension. In the following example, the primary entity is `order_id`. **Query** ```bash # For self-hosted users (dbt v1 or v2) mf query --metrics order_total --group-by order_id__is_food_order --limit 10 --order-by -metric_time --where "is_food_order = True" --start-time '2017-08-22' --end-time '2017-08-27' ``` **Result** ```bash ✔ Success 🦄 - query completed after 1.53 seconds | METRIC_TIME | IS_FOOD_ORDER | ORDER_TOTAL | |:--------------|:----------------|---------------:| | 2017-08-27 | True | 568.92 | | 2017-08-26 | True | 471.95 | | 2017-08-25 | True | 452.93 | | 2017-08-24 | True | 384.40 | | 2017-08-23 | True | 423.61 | | 2017-08-22 | True | 401.91 | ``` ##### Query saved queries You can use this for frequently used queries. Replace `<name>` with the name of your [saved query](https://docs.getdbt.com/docs/build/saved-queries.md). **Query** ```bash dbt sl query --saved-query <name> # For dbt platform users (dbt v1 or v2) mf query --saved-query <name> # For self-hosted users (dbt v1 or v2) ``` For example, if you use dbt and have a saved query named `new_customer_orders`, you would run `dbt sl query --saved-query new_customer_orders`. A note on querying saved queries When querying [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md), you can use parameters such as `where`, `limit`, `order`, `compile`, and so on. However, keep in mind that you can't access `metric` or `group_by` parameters in this context. This is because they are predetermined and fixed parameters for saved queries, and you can't change them at query time. If you would like to query more metrics or dimensions, you can build the query using the standard format. #### Additional query examples The following tabs present additional query examples, like exporting to a CSV. Select the tab that best suits your needs: ##### --compile/--explain flag Add `--compile` (or `--explain` for v1 users) to your query to view the SQL generated by MetricFlow. **Query** ```bash # For dbt platform users (dbt v1 or v2) dbt sl query --metrics order_total --group-by metric_time,is_food_order --limit 10 --order-by -metric_time --where "is_food_order = True" --start-time '2017-08-22' --end-time '2017-08-27' --compile # For self-hosted users (dbt v1 or v2) mf query --metrics order_total --group-by metric_time,is_food_order --limit 10 --order-by -metric_time --where "is_food_order = True" --start-time '2017-08-22' --end-time '2017-08-27' --explain ``` **Result** ```bash ✔ Success 🦄 - query completed after 0.28 seconds 🔎 SQL (remove --compile to see data or add --show-dataflow-plan to see the generated dataflow plan): select metric_time , is_food_order , sum(order_cost) as order_total from ( select cast(ordered_at as date) as metric_time , is_food_order , order_cost from analytics.js_dbt_sl_demo.orders orders_src_1 where cast(ordered_at as date) between cast('2017-08-22' as timestamp) and cast('2017-08-27' as timestamp) ) subq_3 where is_food_order = True group by metric_time , is_food_order order by metric_time desc limit 10 ``` ##### Export to CSV Add the `--csv file_name.csv` flag to export the results of your query to a CSV. The `--csv` flag is available to local self-hosted users only (with the `mf` prefix) and isn't supported by `dbt sl` in dbt platform. **Query** ```bash # For self-hosted users (dbt v1 or v2) mf query --metrics order_total --group-by metric_time,is_food_order --limit 10 --order-by -metric_time --where "is_food_order = True" --start-time '2017-08-22' --end-time '2017-08-27' --csv query_example.csv ``` **Result** ```bash ✔ Success 🦄 - query completed after 0.83 seconds 🖨 Successfully written query output to query_example.csv ``` #### Time granularity Optionally, you can specify the time granularity you want your data to be aggregated at by appending two underscores and the unit of granularity you want to `metric_time`, the global time dimension. You can group the granularity by: `day`, `week`, `month`, `quarter`, and `year`. Below is an example for querying metric data at a monthly grain: ```bash dbt sl query --metrics revenue --group-by metric_time__month # For dbt platform users (dbt v1 or v2) mf query --metrics revenue --group-by metric_time__month # For self-hosted users (dbt v1 or v2) ``` #### Export Run [exports for a specific saved query](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md#exports-for-single-saved-query). Use this command to test and generate exports in your development environment. You can also use the `--select` flag to specify particular exports from a saved query. Refer to [exports in development](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md#exports-in-development) for more info. Export is available in dbt. ```bash dbt sl export # For dbt platform users (dbt v1 or v2) ``` #### Export-all Run [exports for multiple saved queries](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md#exports-for-multiple-saved-queries) at once. This command provides a convenient way to manage and execute exports for several queries simultaneously, saving time and effort. Refer to [exports in development](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md#exports-in-development) for more info. Export is available in dbt. ```bash dbt sl export-all # For dbt platform users (dbt v1 or v2) ``` #### FAQs  How can I add a dimension filter to a where filter? To add a dimension filter to a where filter, you have to indicate that the filter item is part of your model and use a template wrapper: `{{Dimension('primary_entity__dimension_name')}}`. Here's an example query: `dbt sl query --metrics order_total --group-by metric_time --where "{{Dimension('order_id__is_food_order')}} = True"`. Before using the template wrapper, however, set up your terminal to escape curly braces for the filter template to work. Details How to set up your terminal to escape curly braces? To configure your `.zshrc`profile to escape curly braces, you can use the `setopt` command to enable the `BRACECCL` option. This option will cause the shell to treat curly braces as literals and prevent brace expansion. Refer to the following steps to set it up: <br /> 1. Open your terminal. 2. Open your `.zshrc` file using a text editor like `nano`, `vim`, or any other text editor you prefer. You can use the following command to open it with `nano`: ```bash nano ~/.zshrc ``` 3. Add the following line to the file: ```bash setopt BRACECCL ``` 4. Save and exit the text editor (in `nano`, press Ctrl + O to save, and Ctrl + X to exit). 5. Source your `.zshrc` file to apply the changes: ```bash source ~/.zshrc ``` 6. After making these changes, your Zsh shell will treat curly braces as literal characters and will not perform brace expansion. This means that you can use curly braces without worrying about unintended expansions. Keep in mind that modifying your shell configuration files can have an impact on how your shell behaves. If you're not familiar with shell configuration, it's a good idea to make a backup of your `.zshrc` file before making any changes. If you encounter any issues or unexpected behavior, you can revert to the backup.  Why is my query limited to 100 rows in the dbt platform CLI? The default `limit` for query issues from the dbt platform CLI is 100 rows. We set this default to prevent returning unnecessarily large data sets as the dbt platform CLI is typically used to query the dbt Semantic Layer during the development process, not for production reporting or to access large data sets. For most workflows, you only need to return a subset of the data. However, you can change this limit if needed by setting the `--limit` option in your query. For example, to return 1000 rows, you can run `dbt sl list metrics --limit 1000`.  How can I query multiple metrics, group bys, or where statements? To query multiple metrics, group bys, or where statements in your command, follow this guidance: * To query multiple metrics and group bys, use the `--metrics` or `--group-by` syntax followed by the metric or dimension/entity names, separated by commas without spaces: * Multiple metrics example: `dbt sl query --metrics accounts_active,users_active` * Multiple dimension/entity example: `dbt sl query --metrics accounts_active,users_active --group-by metric_time__week,accounts__plan_tier` * To query multiple where statements, use the `--where` syntax and wrap the statement in quotes: * Multiple where statement example: `dbt sl query --metrics accounts_active,users_active --group-by metric_time__week,accounts__plan_tier --where "metric_time__week >= '2024-02-01'" --where "accounts__plan_tier = 'coco'"`  How can I sort my query in ascending or descending order? When you query metrics, use `--order-by` to specify metrics or groupings to order by. The `order_by` option applies to metrics, dimensions, and group bys. Add the `-` prefix to sort your query in descending (DESC) order. Leave blank for ascending (ASC) order: * For example, to query a metric and sort `metric_time` in descending order, run `dbt sl query --metrics order_total --group-by metric_time --order-by -metric_time`. Note that the `-` prefix in `-metric_time` sorts the query in descending order. * To query a metric and sort `metric_time` in ascending order and `revenue` in descending order, run `dbt sl query --metrics order_total --order-by metric_time,-revenue`. Note that `metric_time` without a prefix is sorted in ascending order and `-revenue` with a `-` prefix sorts the query in descending order. --- ### MetricFlow time spine (Applies to dbt v1.9 and later) It's common in analytics engineering to have a date dimension or "time spine" table as a base table for different types of time-based joins and aggregations. The structure of this table is typically a base column of daily or hourly dates, with additional columns for other time grains, like fiscal quarters, defined based on the base column. You can join other tables to the time spine on the base column to calculate metrics like revenue at a point in time, or to aggregate to a specific time grain. To use MetricFlow with time-based metrics and dimensions, you *must* provide a time spine. This table serves as the foundation for time-based joins and aggregations. You can either: * Create a time spine from scratch (check out the [example time spine](#example-time-spine-tables) section for examples), or * Use an existing table in your project, like a `dim_date` table And once you have a time spine, you need to configure it in YAML to tell MetricFlow how to use it. #### Prerequisites MetricFlow requires you to define at least one dbt model which provides a time-spine, and then specify (in YAML) the columns to be used for time-based joins. This means you need to: * Define at least one [time spine](#example-time-spine-tables) at whichever granularity needed for your metrics (like daily or hourly). You can optionally define additional tables for coarser grains (like monthly or yearly). * [Configure each time spine in a properties YAML file](#configuring-time-spine-in-yaml) to define how MetricFlow recognizes and uses its columns. Note that you can't have overlapping time spines. MetricFlow will then join against the time spine model for the following types of metrics and dimensions: * [Cumulative metrics](https://docs.getdbt.com/docs/build/cumulative.md) * [Metric offsets](https://docs.getdbt.com/docs/build/derived.md#derived-metric-offset) * [Conversion metrics](https://docs.getdbt.com/docs/build/conversion.md) * [Slowly Changing Dimensions](https://docs.getdbt.com/docs/build/dimensions.md#scd-type-ii) * [Metrics](https://docs.getdbt.com/docs/build/metrics-overview.md) with the `join_to_timespine` configuration set to true To see the generated SQL for the metric and dimension types that use time spine joins, refer to the respective documentation or add the `compile=true` flag when querying the Semantic Layer to return the compiled SQL. #### Configuring time spine in YAML Use our mini guide to create a time spine table For a quick start guide on how to create a time spine table, check out our [MetricFlow time spine mini guide](https://docs.getdbt.com/guides/mf-time-spine.md)! Time spine models are normal dbt models with extra configurations that tell dbt and MetricFlow how to use specific columns by defining their properties. Add the [`models` key](https://docs.getdbt.com/reference/model-properties.md) for the time spine in your `models/` directory. If your project already includes a calendar table or date dimension, you can configure that table as a time spine. Otherwise, review the [example time-spine tables](#example-time-spine-tables) to create one. If the relevant model file doesn't exist, create it and add the configuration mentioned in the [next section](#creating-a-time-spine-table). Some things to note when configuring time spine models: * Make sure you already have a time spine SQL table defined in your project. * Add the configurations under the `time_spine` key for that [model's properties](https://docs.getdbt.com/reference/model-properties.md), just as you would add a description or tests. * You only need to configure time-spine models that the Semantic Layer should recognize. * At a minimum, define a time-spine table for a daily grain. * You can optionally define additional time-spine tables for different granularities, like hourly. Review the [granularity considerations](#granularity-considerations) when deciding which tables to create. * If you're looking to specify the grain of a time dimension so that MetricFlow can transform the underlying column to the required granularity, refer to the [Time granularity documentation](https://docs.getdbt.com/docs/build/dimensions.md?dimension=time_gran) tip * If you previously used a `metricflow_time_spine.sql` model, you can delete it after configuring the `time_spine` property in YAML. The Semantic Layer automatically recognizes the new configuration. No additional `.yml` files are needed. * You can also configure MetricFlow to use any date dimension or time spine table already in your project by updating the `model` setting in the Semantic Layer. * If you don’t have a date dimension table, you can still create one by using the code snippet in the [next section](#creating-a-time-spine-table) to build your time spine model. ##### Creating a time spine table MetricFlow supports granularities ranging from milliseconds to years. Refer to the [Dimensions page](https://docs.getdbt.com/docs/build/dimensions.md?dimension=time_gran#time) (time\_granularity tab) to find the full list of supported granularities. To create a time spine table from scratch, you can do so by adding the following code to your dbt project. This example creates a time spine at an hourly grain and a daily grain: `time_spine_hourly` and `time_spine_daily`. (Applies to dbt v1.9 and later) models/\_models.yml ```yaml models: # Hourly time spine - name: time_spine_hourly description: my favorite time spine time_spine: standard_granularity_column: date_hour # column for the standard grain of your table, must be date time type. custom_granularities: - name: fiscal_year column_name: fiscal_year_column # must refer to a column defined in the model columns: - name: date_hour granularity: hour # set granularity at column-level for standard_granularity_column # Daily time spine - name: time_spine_daily time_spine: standard_granularity_column: date_day # column for the standard grain of your table columns: - name: date_day granularity: day # set granularity at column-level for standard_granularity_column ``` * This example configuration shows a time spine model called `time_spine_hourly` and `time_spine_daily`. It sets the time spine configurations under the `time_spine` key. * The `standard_granularity_column` is the column that maps to one of our [standard granularities](https://docs.getdbt.com/docs/build/dimensions.md?dimension=time_gran). This column must be set under the `columns` key and should have a grain that is finer or equal to any custom granularity columns defined in the same model. * It needs to reference a column defined under the `columns` key, in this case, `date_hour` and `date_day`, respectively. * It sets the granularity at the column-level using the `granularity` key, in this case, `hour` and `day`, respectively. * MetricFlow will use the `standard_granularity_column` as the join key when joining the time spine table to another source table. * [The `custom_granularities` field](#custom-calendar), (available in dbt Latest and dbt v1.9 and higher) lets you specify non-standard time periods like `fiscal_year` or `retail_month` that your organization may use. * The `column_name` field must reference a column that exists in the same model. For an example project, refer to our [Jaffle shop](https://github.com/dbt-labs/jaffle-sl-template/blob/main/models/marts/_models.yml) example. ##### Migrating from SQL to YAML If you already have a SQL model that defines your time spine, you can reference that model directly in the YAML file. If you don't have a SQL model that defines your time spine, add one before proceeding to the following steps. 1. Add the following configuration to a new or existing properties YAML file using the [`models` key](https://docs.getdbt.com/reference/model-properties.md) for the time spine in your `models/` directory. Name the properties YAML file whatever you want (for example, `util/_models.yml`): models/\_models.yml ```yaml models: - name: all_days description: A time spine with one row per day, ranging from 2020-01-01 to 2039-12-31. time_spine: standard_granularity_column: date_day # Column for the standard grain of your table columns: - name: date_day granularity: day # Set the granularity of the column ``` 2. After adding the YAML configuration and ensuring you have a SQL model that defines the time spine, you can delete the existing `metricflow_time_spine.sql` file from your project to avoid any deprecation warnings or errors. 3. Test the configuration to ensure compatibility with your production jobs. Note that if you're migrating from a `metricflow_time_spine.sql` file: * Replace its functionality by adding the `time_spine` property to YAML as shown in the previous example. * Once configured, MetricFlow will recognize the YAML settings, and then the SQL model file can be safely removed. ##### Considerations when choosing which granularities to create * MetricFlow will use the time spine with the largest compatible granularity for a given query to ensure the most efficient query possible. For example, if you have a time spine at a monthly grain, and query a dimension at a monthly grain, MetricFlow will use the monthly time spine. If you only have a daily time spine, MetricFlow will use the daily time spine and `date_trunc` to month. * You can add a time spine for each granularity you intend to use if query efficiency is more important to you than configuration time, or storage constraints. For most engines, the query performance difference should be minimal and transforming your time spine to a coarser grain at query time shouldn't add significant overhead to your queries. * We recommend having a time spine at the finest grain used in any of your dimensions to avoid unexpected errors. For example, if you have dimensions at an hourly grain, you should have a time spine at an hourly grain. #### Example time spine tables The following examples show how to create time spine tables at different granularities: * [Seconds](#seconds) * [Minutes](#minutes) * [Daily](#daily) * [Daily (BigQuery)](#daily-bigquery) * [Hourly](#hourly) ##### Seconds metricflow\_time\_spine.sql ```sql {{ config(materialized='table') }} with seconds as ( {{ dbt.date_spine( 'second', "date_trunc('second', dateadd(second, -10, current_timestamp()))", "date_trunc('second', current_timestamp())" ) }} ), final as ( select cast(date_second as timestamp) as second_timestamp from seconds ) select * from final ``` ##### Minutes metricflow\_time\_spine.sql ```sql {{ config(materialized='table') }} with minutes as ( {{ dbt.date_spine( 'minute', "date_trunc('minute', dateadd(minute, -5, current_timestamp()))", "date_trunc('minute', current_timestamp())" ) }} ), final as ( select cast(date_minute as timestamp) as minute_timestamp from minutes ) select * from final ``` ##### Daily metricflow\_time\_spine.sql ```sql {{ config( materialized = 'table', ) }} with days as ( {{ dbt.date_spine( 'day', "to_date('01/01/2000','mm/dd/yyyy')", "to_date('01/01/2025','mm/dd/yyyy')" ) }} ), final as ( select cast(date_day as date) as date_day from days ) select * from final where date_day > dateadd(year, -4, current_timestamp()) and date_day < dateadd(day, 30, current_timestamp()) ``` ##### Daily (BigQuery) Use this model if you're using BigQuery. BigQuery supports `DATE()` instead of `TO_DATE()`: metricflow\_time\_spine.sql ```sql {{config(materialized='table')}} with days as ( {{dbt.date_spine( 'day', "DATE(2000,01,01)", "DATE(2025,01,01)" ) }} ), final as ( select cast(date_day as date) as date_day from days ) select * from final -- filter the time spine to a specific range where date_day > date_add(DATE(current_timestamp()), INTERVAL -4 YEAR) and date_day < date_add(DATE(current_timestamp()), INTERVAL 30 DAY) ``` ##### Hourly time\_spine\_hourly.sql ```sql {{ config( materialized = 'table', ) }} with hours as ( {{ dbt.date_spine( 'hour', "to_date('01/01/2000','mm/dd/yyyy')", "to_date('01/01/2025','mm/dd/yyyy')" ) }} ), final as ( select cast(date_hour as timestamp) as date_hour from hours ) select * from final -- filter the time spine to a specific range where date_day > dateadd(year, -4, current_timestamp()) and date_hour < dateadd(day, 30, current_timestamp()) ``` #### Custom calendar [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") tip Check out our mini guide on [how to create a time spine table](https://docs.getdbt.com/guides/mf-time-spine.md) to get started! (Applies to dbt v1.9 and later) Custom date transformations can be complex, and organizations often have unique needs that can’t be easily generalized. Creating a custom calendar model allows you to define these transformations in SQL, offering more flexibility than native transformations in MetricFlow. This approach lets you map custom columns back to MetricFlow granularities, ensuring consistency while giving you control over the transformations. For example, if you use a custom calendar in your organization, such as a fiscal calendar, you can configure it in MetricFlow using its date and time operations. * This is useful for calculating metrics based on a custom calendar, such as fiscal quarters or weeks. * Use the `custom_granularities` key to define a non-standard time period for querying data, such as a `retail_month` or `fiscal_week`, instead of standard options like `day`, `month`, or `year`. * This feature provides more control over how time-based metrics are calculated.  Data types and time zone considerations When working with custom calendars in MetricFlow, it's important to ensure: * Consistent data types — Both your dimension column and the time spine column should use the same data type to allow accurate comparisons. Functions like `DATE_TRUNC` don't change the data type of the input in some databases (like Snowflake). Using different data types can lead to mismatches and inaccurate results. We recommend using `DATETIME` or `TIMESTAMP` data types for your time dimensions and time spine, as they support all granularities. The `DATE` data type may not support smaller granularities like hours or minutes. * Time zones — MetricFlow currently doesn't perform any timezone manipulation. When working with timezone-aware data, inconsistent time zones may lead to unexpected results during aggregations and comparisons. For example, if your time spine column is `TIMESTAMP` type and your dimension column is `DATE` type, comparisons between these columns might not work as intended. To fix this, convert your `DATE` column to `TIMESTAMP`, or make sure both columns are the same data type. ##### Add custom granularities To add custom granularities, the Semantic Layer supports custom calendar configurations that allow users to query data using non-standard time periods like `fiscal_year` or `retail_month`. You can define these custom granularities (all lowercased) by modifying your model's YAML configuration like this: models/\_models.yml ```yaml models: - name: my_time_spine description: my favorite time spine time_spine: standard_granularity_column: date_day custom_granularities: - name: fiscal_year column_name: fiscal_year_column # must refer to a column defined in the model ``` ###### Coming soon Note that features like calculating offsets and period-over-period will be supported soon! #### Related docs * [MetricFlow time granularity](https://docs.getdbt.com/docs/build/dimensions.md?dimension=time_gran#time) * [MetricFlow time spine mini guide](https://docs.getdbt.com/guides/mf-time-spine.md) --- ### Metrics as dimensions with metric filters [Metrics](https://docs.getdbt.com/docs/build/metrics-overview.md) provide users with valuable insights into their data, like number of active users and overall performance trends to inform business decisions. [Dimensions](https://docs.getdbt.com/docs/build/dimensions.md), on the other hand, help categorize data through attributes, like user type or number of orders placed by a customer. To make informed business decisions, some metrics need the value of another metric as part of the metric definition, leading us to "metrics as dimensions". This document explains how you can use metrics as dimensions with metric filters, enabling you to create more complex metrics and gain more insights. #### Reference a metric in a filter Use the `Metric()` object syntax to reference a metric in the `where` filter for another metric. The function for referencing a metric accepts a metric name and exactly one entity: ```yaml {{ Metric('metric_name', group_by=['entity_name']) }} ``` ##### Usage example As an example, a Software as a service (SaaS) company wants to count activated accounts. In this case, the definition of an activated account is an account with more than five data model runs. To express this metric in SQL, the company will: * Write a query to calculate the number of data model runs per account. * Then count the number of accounts who have more than five data model runs. models/model\_name.sql ```sql with data_models_per_user as ( select account_id as account, count(model_runs) as data_model_runs from {{ ref('fct_model_runs') }} group by account_id ), activated_accounts as ( select count(distinct account_id) as activated_accounts from {{ ref('dim_accounts') }} left join data_models_per_user on {{ ref('dim_accounts') }}.account_id = data_models_per_user.account where data_models_per_user.data_model_runs > 5 ) select * from activated_accounts ``` This SQL query calculates the number of `activated_accounts` by using the `data_model_runs` metric as a dimension for the user entity. It filters based on the metric value scoped to the account entity. You can express this logic at the query level or in the metric's YAML configuration. ###### YAML configuration Using the same `activated_accounts` example mentioned in [the usage example](#usage-example), the following YAML example explains how a company can create [semantic models](https://docs.getdbt.com/docs/build/semantic-models.md) and [metrics](https://docs.getdbt.com/docs/build/metrics-overview.md), and use the `Metric()` object to reference the `data_model_runs` metric in the `activated_accounts` metric filter: (Applies to dbt v1.12 and later) * Create two semantic models: `model_runs` and `accounts`. * Create metrics to count data model runs and users. * Specify the foreign entity `account` in the `model_runs` semantic model. * Then create the `Activated Accounts` metric by filtering accounts that have more than five data model runs. models/model\_runs.yml ```yaml models: - name: model_runs semantic_model: enabled: true # ... other configurations ... columns: - name: model_run_id entity: type: primary name: model_run - name: account_id entity: type: foreign name: account metrics: - name: data_model_runs type: simple agg: sum expr: 1 - name: accounts semantic_model: enabled: true # ... other configurations ... columns: - name: account_id entity: type: primary name: account metrics: - name: accounts type: simple agg: sum expr: 1 # Metric that filters accounts based on another metric - name: activated_accounts label: Activated Accounts type: simple agg: sum expr: 1 filter: | {{ Metric('data_model_runs', group_by=['account']) }} > 5 ``` Let’s break down the SQL the system generates based on the metric definition when you run `dbt sl query --metrics activated_accounts` from the command line interface: * The filter `{{ Metric('data_model_runs', group_by=['account']) }}` generates SQL similar to the `data_models_per_user` sub-query shown earlier: ```sql select sum(1) as data_model_runs, account from data_model_runs group by account ``` * MetricFlow joins this query to the query generated by `accounts` on the group by elements and applies the filter conditions: ```sql select sum(1) as activated_accounts from accounts left join ( select sum(1) as data_model_runs, account from data_model_runs group by account ) as subq on accounts.account = subq.account where data_model_runs > 5 ``` The intermediate tables used to create this metric is: Accounts with the `data_model_runs` dimension | account | data\_model runs | | ------- | ---------------- | | 1 | 4 | | 2 | 7 | | 3 | 9 | | 4 | 1 | MetricFlow then filters this table to accounts with more than 5 data model runs and counts the number of accounts that meet this criteria: | activated\_accounts | | ------------------- | | 2 | ###### Query filter You can also use metrics in filters at the query level. Run this command in the command line interface (CLI) to generate the same SQL query referenced earlier: `dbt sl query --metrics accounts --where "{{ Metric('data_model_runs', group_by=['account']) }} > 5"` The resulting SQL and data will be the same, except with the `accounts` metric name instead of `activated_accounts`. #### Considerations * When using a metric filter, ensure the sub-query can join to the outer query without fanning out the result (unexpectedly increasing the number of rows). * The example that filters accounts using `{{ Metric('data_model_runs', group_by=['account']) }}` is valid because it aggregates the model runs to the account level. * However, filtering 'accounts' by `{{ Metric('data_model_runs', group_by=['model']) }}` isn't valid due to a one-to-many relationship between accounts and model runs, leading to duplicate data. * You can only group a metric by one entity. The ability to support grouping by multiple entities and dimensions is pending. * In the future, you can use metrics as dimensions for some of the following example use cases: * User segments: Segment users by using the number of orders placed by a user in the last 7 days as a dimension. * Churn prediction: Use the number of support tickets an account submitted in the first 30 days to predict potential churn. * Activation tracking: Define account or user activation based on the specific actions taken within a certain number of days after signing up. * Support for metric filters requiring multi-hop joins is pending. --- ### Microsoft Excel dbt platform | Starter, Enterprise, Enterprise+ The Semantic Layer offers a seamless integration with Excel Online and Desktop through a custom menu. This add-on allows you to build Semantic Layer queries and return data on your metrics directly within Excel. #### Prerequisites * You have [configured the Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md) and are using dbt v1.6 or higher. * You need a Microsoft Excel account with access to install add-ons. * You have a [dbt Environment ID](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md#set-up-dbt-semantic-layer). * You have a [service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) or a [personal access token](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) to authenticate with from a dbt account. * You must have a dbt Starter, Enterprise, or Enterprise+ [account](https://www.getdbt.com/pricing). Suitable for both Multi-tenant and Single-tenant deployment. tip 📹 For on-demand video learning, explore the [Querying the Semantic Layer with Excel](https://learn.getdbt.com/courses/querying-the-semantic-layer-with-excel) course to learn how to query metrics with Excel. #### Installing the add-on The Semantic Layer Microsoft Excel integration is available to download directly on [Microsoft AppSource](https://appsource.microsoft.com/en-us/product/office/WA200007100?tab=Overview). You can choose to download this add-on in for both [Excel Desktop](https://pages.store.office.com/addinsinstallpage.aspx?assetid=WA200007100\&rs=en-US\&correlationId=4132ecd1-425d-982d-efb4-de94ebc83f26) and [Excel Online](https://pages.store.office.com/addinsinstallpage.aspx?assetid=WA200007100\&rs=en-US\&correlationid=4132ecd1-425d-982d-efb4-de94ebc83f26\&isWac=True) 1. In Excel, authenticate with your Host, dbt Environment ID, and service token. * Access your Environment ID, Host, and URLs in your Semantic Layer settings. Generate a service token in the Semantic Layer settings or **API tokens** settings. Alternatively, you can also create a personal access token by going to **API tokens** > **Personal tokens**. [![Access your Environment ID, Host, and URLs in your dbt Semantic Layer settings. Generate a service token in the Semantic Layer settings or API tokens settings](/img/docs/dbt-platform/semantic-layer/sl-and-gsheets.png?v=2 "Access your Environment ID, Host, and URLs in your dbt Semantic Layer settings. Generate a service token in the Semantic Layer settings or API tokens settings")](#)Access your Environment ID, Host, and URLs in your dbt Semantic Layer settings. Generate a service token in the Semantic Layer settings or API tokens settings 2. Start querying your metrics using the **Query Builder**. For more info on the menu functions, refer to [Query Builder functions](#query-builder-functions). To cancel a query while running, press the **Cancel** button. When querying your data with Microsoft Excel: * It returns the data to the cell you clicked on. * Results that take longer than one minute to load into Excel will fail. This limit only applies to the loading process, not the time it takes for the data platform to run the query. * If you're using this extension, make sure you're signed into Microsoft with the same Excel profile you used to set up the Add-In. Log in with one profile at a time as using multiple profiles at once might cause issues. * Note that only standard granularities are currently available, custom time granularities aren't currently supported for this integration. #### Query Builder functions The Microsoft Excel **Query Builder** custom menu has the following capabilities: | Menu items | Description | | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Metrics | Search and select metrics. | | Group By | Search and select dimensions or entities to group by. Dimensions are grouped by the entity of the semantic model they come from. You may choose dimensions on their own without metrics. | | Time Range | Quickly select time ranges to look at the data, which applies to the main time series for the metrics (metric time), or do more advanced filter using the "Custom" selection. | | Where | Filter your data. This includes categorical and time filters. | | Order By | Return your data order. | | Limit | Set a limit for the rows of your output. | Note: Click the **info** button next to any metric or dimension to see its defined description from your dbt project. ###### Modifying time granularity When you select time dimensions in the **Group By** menu, you'll see a list of available time granularities. The lowest granularity is selected by default. Metric time is the default time dimension for grouping your metrics. info Note: [Custom time granularities](https://docs.getdbt.com/docs/build/metricflow-time-spine.md#add-custom-granularities) (like fiscal year) aren't currently supported or accessible in this integration. Only [standard granularities](https://docs.getdbt.com/docs/build/dimensions.md?dimension=time_gran#time) (like day, week, month, and so on) are available. If you'd like to access custom granularities, consider using the [Semantic Layer APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md). ###### Filtering data To use the filter functionality, choose the [dimension](https://docs.getdbt.com/docs/build/dimensions.md) you want to filter by and select the operation you want to filter on. * For categorical dimensions, you can type a value into search or select from a populated list. * For entities, you must type the value you are looking for as we do not load all of them given the large number of values. * Continue adding additional filters as needed with AND and OR. * For time dimensions, you can use the time range selector to filter on presets or custom options. The time range selector applies only to the primary time dimension (`metric_time`). For all other time dimensions that aren't `metric_time`, you can use the "Where" option to apply filters. ###### Other settings If you would like to just query the data values without the headers, you can optionally select the **Exclude column names** box. To return your results and keep any previously selected data below it intact, un-select the **Clear trailing rows** box. By default, we'll clear all trailing rows if there's stale data. [![Run a query in the Query Builder. Use the arrow next to the Query button to select additional settings.](/img/docs/dbt-platform/semantic-layer/query-builder.png?v=2 "Run a query in the Query Builder. Use the arrow next to the Query button to select additional settings.")](#)Run a query in the Query Builder. Use the arrow next to the Query button to select additional settings. #### Using saved selections Saved selections allow you to save the inputs you've created in the Microsoft Excel **Query Builder** and easily access them again so you don't have to continuously build common queries from scratch. To create a saved selection: 1. Run a query in the **Query Builder**. 2. Save the selection by selecting the arrow next to the **Query** button and then select **Query & Save Selection**. 3. The application saves these selections, allowing you to view and edit them from the hamburger menu under **Saved Selections**. ##### Refreshing selections Set your saved selections to automatically refresh every time you load the addon. You can do this by selecting **Refresh on Load** when creating the saved selection. When you access the addon and have saved selections that should refresh, you'll see "Loading..." in the cells that are refreshing. Public saved selections will refresh for anyone who edits the sheet. What's the difference between saved selections and saved queries? * Saved selections are saved components that you can create only when using the application. * Saved queries, explained in the next section, are code-defined sections of data you create in your dbt project that you can easily access and use for building selections. You can also use the results from a saved query to create a saved selection. #### Using saved queries Access [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md), powered by MetricFlow, in Microsoft Excel to quickly get results from pre-defined sets of data. To access the saved queries in Microsoft Excel: 1. Open the hamburger menu in Microsoft Excel. 2. Navigate to **Saved Queries** to access the ones available to you. 3. You can also select **Build Selection**, which allows you to explore the existing query. This won't change the original query defined in the code. * If you use a `WHERE` filter in a saved query, Microsoft Excel displays the advanced syntax for this filter. #### FAQs I'm receiving an \`Failed ALPN\` error when trying to connect to the dbt Semantic Layer. If you're receiving a `Failed ALPN` error when trying to connect the dbt Semantic Layer with the various [data integration tools](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) (such as Tableau, DBeaver, Datagrip, ADBC, or JDBC), it typically happens when connecting from a computer behind a corporate VPN or Proxy (like Zscaler or Check Point). The root cause is typically the proxy interfering with the TLS handshake as the Semantic Layer uses gRPC/HTTP2 for connectivity. To resolve this: * If your proxy supports gRPC/HTTP2 but isn't configured to allow ALPN, adjust its settings accordingly to allow ALPN. Or create an exception for the dbt domain. * If your proxy does not support gRPC/HTTP2, add an SSL interception exception for the dbt domain in your proxy settings This should help in successfully establishing the connection without the Failed ALPN error. --- ### Migrate to dbt Wizard Local development Move from Claude Code to dbt Wizard while keeping your project conventions. dbt Wizard automatically imports Claude Code instructions, skills, and settings from your repo. Use this page if you already use Claude Code on dbt projects and want to bring over project context, skills, or model settings. If you're new to AI agents, start with [Use dbt Wizard locally](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md). #### Prerequisites You'll need: * [dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-cli.md) installed or dbt Wizard enabled in the [dbt platform](https://docs.getdbt.com/docs/platform/wizard-platform.md) * [BYOK](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) configured for a supported CLI provider (OpenAI, Anthropic, AWS Bedrock, or Snowflake Cortex in preview) * Any existing Claude Code files you want to migrate, such as `CLAUDE.md`, `.claude/CLAUDE.md`, or `.claude/skills/` #### Review project context and skills dbt Wizard automatically reads common agent instruction files and skills from your repo. You usually don't need to move existing project context before using dbt Wizard. | File or folder | What Wizard does | Migration needed? | | ---------------------------------------- | -------------------------------------------------------------- | ----------------- | | `AGENTS.override.md` at the project root | Checks this first for local override instructions. | No | | `AGENTS.md` at the project root | Uses this as the primary project instruction file. | No | | `CLAUDE.md` at the project root | Uses this as a built-in fallback when `AGENTS.md` is absent. | No | | `.claude/CLAUDE.md` | Uses this as another fallback instruction file. | No | | `.claude/skills/NAME/SKILL.md` | Auto-discovers these as skills. | No | | Reusable prompts or new skills | Place them in `.agents/skills/NAME/SKILL.md` at the repo root. | Yes | dbt Wizard also walks the directory tree from your project root to your current working directory, picks up instruction files at each level, and combines them for the session. For the full skill format, refer to [Skills](https://docs.getdbt.com/docs/dbt-ai/wizard-skills.md). ##### Convert reusable prompts to skills Use skills for reusable workflows or specialized instructions, not for general project context that already lives in `AGENTS.md` or `CLAUDE.md`. Each repo-level skill lives in `.agents/skills/SKILL_NAME/SKILL.md`. ```bash mkdir -p .agents/skills/my-team-style touch .agents/skills/my-team-style/SKILL.md ``` Then add frontmatter and instructions: ```markdown --- name: my-team-style description: Apply team-specific dbt modeling conventions when creating, editing, refactoring, testing, or documenting models in this project. --- # Team style (add your reusable instructions here) ``` Write a specific `description:` because dbt Wizard uses it to decide when to load the skill. #### Configure model settings Set provider credentials in [Configure BYOK](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md). To set a default model across sessions, add this to `~/.dbt/wizard/config.toml`: ```toml model = "claude-sonnet-4-6" # use a model ID from `wizard debug models` ``` You can also pick a model in the TUI with `/model` without editing config or the `--config` flag. Run `wizard --help` to see all available flags. #### Verify the migration From your dbt project root, start a new session: ```bash wizard ``` Ask dbt Wizard to do something that should use your migrated skill, such as: ```text create a new staging model for raw_invoices ``` And if your conventions aren't applied, check: * General project context is in an instruction file such as `AGENTS.md` or `CLAUDE.md` * Reusable skills are under `.agents/skills/` at the project root or `~/.agents/skills/` for CLI-global skills * The skill has a clear `description:` * You started a new session after adding the skill #### Related docs * [Use dbt Wizard locally](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) * [Skills](https://docs.getdbt.com/docs/dbt-ai/wizard-skills.md) * [Configure BYOK](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) * [Use cases and examples](https://docs.getdbt.com/docs/dbt-ai/wizard-use-cases.md) * [How to use dbt Wizard in your dbt project](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-1-intro.md) for recommended workflows --- ### Migrate to the latest YAML spec The latest Semantic Layer specification creates an open standard for defining metrics and dimensions that works across multiple platforms. It simplifies authorship by embedding semantic annotations alongside each model, replacing measures with simple metrics, and promoting frequently used options to top-level keys. With the new spec, you get simpler configuration without losing flexibility, faster onboarding for new contributors, and a clearer path to consistent, governed metrics across your organization. Availability The latest YAML spec is supported in the following environments: * **dbt platform (v1 Latest release track)** * **dbt v2** * **dbt v1.12** #### Changes in the latest spec This section highlights the key updates in the latest metrics spec and compares them to the legacy spec. * [Semantic models](#semantic-models): These define the business logic for your metrics by specifying entities, dimensions, and how they relate to your data models. In the new spec, `semantic_model` is nested directly under each model in `models:` instead of being a top-level key. * [Entities and dimensions](#entities-and-dimensions): Entities are the people, places, or things you want to group or join your metrics by (like `user_id` or `order_id`), while dimensions are the attributes you use to filter or slice your data (like `status` or `region`). In the new spec, both are defined directly under `columns:`. * [Time dimension](#time-dimension): Time dimensions are the date or timestamp columns that let you analyze metrics over time (like `order_date` or `created_at`). In the new spec, set `agg_time_dimension` at the model level as the default time dimension for all metrics, with the option to override per metric. Define `granularity` at the column level instead of using the deprecated `time_granularity`. * [Simple metrics](#simple-metrics): Metrics that directly reference a single column expression within a semantic model, without any additional columns involved. Simple metrics replace measures in the new spec. Use `type: simple` metrics defined directly within the model to replace measures. * [Advanced metrics](#advanced-metrics): These are metrics that combine or build upon other metrics, such as ratios, conversions, or derived calculations. In the new spec, define simple metrics inside the model, and create cross‑model metrics under a top‑level `metrics` block. Top-level key is required for any metric that depends on metrics or dimensions defined in a different semantic model. * [`type_params`](#type_params): This is a wrapper key in the legacy spec that contains metric-specific configurations (for example, `expr`, `join_to_timespine`). `type_params` is deprecated in the new spec and these parameters are promoted to top-level keys within each metric definition. ##### Semantic models The `semantic_model` key is embedded under `models`. ###### New spec ```yml models: - name: fct_orders semantic_model: enabled: true # required name: fct_orders_semantic_model # optional override; defaults to value of model.name ``` ###### Legacy spec ```yml semantic_models: - name: orders model: ref('orders') ``` Cross-project refs unsupported in latest SL YAML spec When using [dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md) with the [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md), [referencing models from different projects](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models) is only supported in the legacy YAML spec, where semantic models are defined as top-level resources and can reference models across projects. In the [latest YAML spec](https://docs.getdbt.com/docs/build/latest-metrics-spec.md), semantic models are defined within model YAML files, and cross-project references are not yet supported. Support for this capability in the latest spec is planned for a future release. ##### Entities and dimensions Entities and dimensions are defined directly under columns, creating a 1:1 relationship between the physical columns and their semantic definitions. ###### New spec ```yml models: - name: orders semantic_model: enabled: true agg_time_dimension: ordered_at columns: # entities - name: order_id entity: type: primary name: order - name: customer_id entity: type: foreign name: customer # time dimension - name: ordered_at granularity: day dimension: type: time # categorical dimension - name: order_status dimension: type: categorical ``` ###### Legacy spec ```yml semantic_models: - name: orders model: ref('orders') entities: - name: order type: primary expr: order_id - name: customer type: foreign expr: customer_id dimensions: - name: ordered_at type: time type_params: time_granularity: day - name: status type: categorical expr: order_status ``` ##### Time dimension * `agg_time_dimension`: Set once at the model level as the default time dimension for all metrics in that semantic model. You can still override it per metric with `agg_time_dimension`. * `time granularity`: Deprecated in the new spec. Define the native grain on the time dimension column with `granularity` (for example, `hour`, `day`). ###### New spec ```yml models: - name: subscriptions semantic_model: enabled: true # default aggregation time dimension for metrics in this model agg_time_dimension: activated_at columns: - name: activated_at granularity: day # native grain on the column dimension: type: time - name: created_at granularity: hour # another time column with a different native grain dimension: type: time metrics: - name: active_subscriptions type: simple agg: count expr: 1 # inherits agg_time_dimension: activated_at - name: signups_by_created_day type: simple agg: count expr: 1 agg_time_dimension: created_at # override to use created_at as the time dimension ``` ###### Legacy spec ```yml semantic_models: - name: subscriptions model: ref('subscriptions') defaults: agg_time_dimension: activated_at dimensions: - name: activated_at type: time type_params: time_granularity: day - name: created_at type: time type_params: time_granularity: hour measures: - name: active_subscriptions agg: count metrics: - name: active_subscriptions type: simple type_params: measure: active_subscriptions ``` ##### Simple metrics Measures are deprecated in the new spec and are replaced with simple metrics. ###### New spec ```yml models: - name: customers semantic_model: enabled: true agg_time_dimension: first_ordered_at columns: - name: customer_id entity: name: customer type: primary - name: first_ordered_at dimension: type: time granularity: day metrics: - name: lifetime_spend_pretax type: simple # simple metrics agg: sum expr: amount_pretax ``` ###### Legacy spec ```yml semantic_models: - name: customers model: ref('customers') entities: - name: customer type: primary expr: customer_id dimensions: - name: first_ordered_at type: time type_params: time_granularity: day measures: - name: lifetime_spend_pretax agg: sum metrics: - name: lifetime_spend_pretax type: simple type_params: measure: lifetime_spend_pretax ``` ##### Advanced metrics Define simple metrics inside the model, and create cross‑model metrics under a top‑level `metrics` block. Top-level key is required for any metric that depends on metrics or dimensions defined in a different semantic model. ###### New spec ```yml # define simple metrics where the data lives models: - name: orders ... semantic_model: enabled: true metrics: - name: orders type: simple agg: count expr: 1 - name: website semantic_model: enabled: true metrics: - name: sessions type: simple agg: count expr: 1 # advanced metrics under top-level metrics key metrics: - name: orders_per_session type: ratio numerator: orders denominator: sessions ``` ###### Legacy spec ```yml semantic_models: - name: orders model: ref('orders') measures: - name: orders agg: count - name: website model: ref('website') measures: - name: sessions agg: count metrics: - name: orders_per_session type: ratio type_params: numerator: { measure: orders } denominator: { measure: sessions } ``` ##### `type_params` The `type_params` key is deprecated. The following are direct keys on the metric: * `expr` * `percentile` * `percentile_type` * `non_additive_dimension: { name, window_agg, group_by }` * `join_to_timespine` * `fill_nulls_with` ###### New spec ```yml models: - name: payments semantic_model: enabled: true metrics: - name: revenue_p95 type: simple agg: percentile expr: amount percentile: 95.0 percentile_type: discrete ``` ###### Legacy spec ```yml metrics: - name: revenue_p95 type: simple type_params: expr: amount percentile: 95.0 percentile_type: discrete ``` For [derived metrics](https://docs.getdbt.com/docs/build/derived.md), `type_params.metrics` is renamed `input_metrics`. ###### New spec ```yaml metrics: - name: d7_booking_change description: Difference between bookings now and 7 days ago type: derived label: d7 bookings change expr: current_bookings - bookings_7_days_ago input_metrics: - name: bookings alias: current_bookings - name: bookings offset_window: 7 days alias: bookings_7_days_ago ``` ###### Legacy spec ```yaml - name: d7_booking_change description: Difference between bookings now and 7 days ago type: derived label: d7 bookings change type_params: expr: bookings - bookings_7_days_ago metrics: - name: bookings alias: current_bookings - name: bookings offset_window: 7 days alias: bookings_7_days_ago ``` For [ratio metrics](https://docs.getdbt.com/docs/build/ratio.md), `numerator` and `denominator` are now direct keys on the metric. ###### New spec ```yaml metrics: - name: conversion_rate type: ratio numerator: conversions denominator: sessions ``` ###### Legacy spec ```yaml metrics: - name: conversion_rate type: ratio type_params: numerator: conversions denominator: sessions ``` For [cumulative metrics](https://docs.getdbt.com/docs/build/cumulative.md): * `type_params.measure` is renamed `input_metric` and must reference a metric. * `type_params.cumulative_type_params` values are direct keys on the metric: `window`, `grain_to_date`, and `period_agg`. ###### New spec ```yaml metrics: - name: revenue_mtd_cumulative type: cumulative input_metric: revenue_daily window: 30d grain_to_date: month period_agg: sum ``` ###### Legacy spec ```yaml metrics: - name: revenue_mtd_cumulative type: cumulative type_params: measure: revenue_daily cumulative_type_params: window: 30d grain_to_date: month period_agg: sum ``` For [conversion metrics](https://docs.getdbt.com/docs/build/conversion.md), the following `type_params.conversion_type_params` values are direct keys on the metric: * `entity` * `calculation` * `base_metric` (previously `base_measure`) * `conversion_metric` (previously `conversion_measure`) * `constant_properties` ###### New spec ```yaml metrics: - name: paid_signup_conversion type: conversion entity: user_id calculation: conversion_rate base_metric: signups conversion_metric: paid_signups constant_properties: - base_property: plan conversion_property: plan ``` ###### Legacy spec ```yaml metrics: - name: paid_signup_conversion type: conversion type_params: conversion_type_params: entity: user_id calculation: conversion_rate base_measure: signups conversion_measure: paid_signups constant_properties: plan: pro ``` #### Migrating to the latest spec Studio IDE YAML validation The Studio IDE validates dbt YAML using JSON Schema from the [dbt-jsonschema](https://github.com/dbt-labs/dbt-jsonschema) project. These definitions are aligned with dbt v2 and apply across all [dbt platform release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md), including when your development environment is still running dbt v1. If the Studio IDE flags your YAML as invalid but dbt commands succeed, trust your run results. Share examples with [dbt Support](mailto:support@getdbt.com) or your account team so the schema can be updated. Migrate your legacy metrics to the latest YAML spec using the dbt-autofix tool in your CLI, the [dbt VS Code extension](https://docs.getdbt.com/docs/about-dbt-extension.md), or dbt platform's Studio IDE. note dbt Copilot doesn't yet support generating semantic models with the latest YAML spec. ##### Package compatibility If your project uses dbt packages (listed in `packages.yml`) that define metrics or semantic models, the package maintainer must update those packages to use the latest YAML spec. The [dbt-autofix tool](https://github.com/dbt-labs/dbt-autofix) only updates files in your current dbt project (like models, marts, and so on) and does not update installed packages under `dbt_packages/`. If an installed package still uses the legacy metrics spec, dbt may raise parsing or validation errors after migration. To update packages, a package maintainer should: 1. Run `dbt-autofix deprecations --semantic-layer` in the package repository. 2. Validate the changes by running: * For dbt v2 and dbt users in the dbt platform CLI or locally with a valid [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md): ```bash dbt parse dbt sl validate ``` When using `dbt sl validate` locally, the command validates your local semantic manifest, and not the platform's manifest. This means your uncommitted local changes are included in the validation. * For dbt v2 CLI users not connected to dbt platform and using local MetricFlow: ```bash dbt parse mf validate-configs ``` 3. Release a new version of the package with the updated metrics definitions. After a compatible version is released, update your project to [install the new package version](https://docs.getdbt.com/docs/build/packages.md). You can then migrate your metrics to the latest spec with the following steps, depending on which tool you're using. * [Using the CLI or VS Code extension](#using-the-cli-or-vs-code-extension) * [Using the Studio IDE](#using-the-studio-ide) ##### Using the CLI or VS Code extension The [dbt-autofix tool](https://github.com/dbt-labs/dbt-autofix) rewrites legacy metrics YAML into the latest format and produces a clear, reviewable diff in version control. Make sure you have installed the latest version of the autofix tool before migrating to the new spec using the CLI or the dbt VS Code extension. 1. In your CLI or in the VS Code extension, run the following command: ```bash dbt-autofix deprecations --semantic-layer ``` 2. Review the diff and resolve all flagged items. 3. Run parsing and validations: ```bash dbt parse mf validate-configs ``` ##### Using the Studio IDE Convert your metrics in the Studio IDE in the dbt platform without having to install the `dbt-autofix` tool. 1. Navigate to the Studio IDE by clicking **Studio** in the left menu. 2. Make sure to save and commit your work before proceeding. The autofix command may overwrite any unsaved changes. 3. In the Studio IDE, run the following command: ```bash dbt-autofix deprecations --semantic-layer ``` 4. Click **Commit and sync** in the top left of the Studio IDE to commit these changes to the project repository. --- ### Migrating from state-aware orchestration to dbt State Preview ### Migrating from state-aware orchestration to dbt State [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Login required | Usage-based dbt State improves upon state-aware orchestration in a few key ways: * **Works everywhere** — dbt State works with dbt v1, dbt v2, and dbt platform, as well as external orchestrators, across both development and deployment environments. * **Smarter data freshness tracking** — dbt State tracks data freshness across the DAG and automatically propagates it through models materialized as views. Unlike state-aware orchestration's `build_after` config which compares against the model's last successful execution, dbt State's `lag_tolerance` compares against the freshness of the underlying data. * **Advanced change detection** — dbt State can detect and ignore file modifications that don't change actual transformation logic, such as adding a comment or cleaning up whitespace. If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. For details on billing after the trial ends, refer to [dbt State usage and pricing](https://docs.getdbt.com/docs/platform/billing.md#dbt-state-usage). While dbt State is in preview, there is no required migration timeline — dbt Labs will communicate a timeline when dbt State reaches general availability. #### Migrating your configuration Much of dbt State's configuration will feel familiar if you've used state-aware orchestration, but there is one significant difference: the `build_after` configs have moved out of the `freshness` block and into a new `state` block. To migrate to dbt State, move your configs from `freshness.build_after` to the new `state` block. Refer to the following table for the full mapping. | State-aware orchestration | dbt State | Notes | | -------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `freshness.build_after.updates_on` | [`state.require_fresh_data_from`](https://docs.getdbt.com/reference/resource-configs/require-fresh-data-from.md) | Same `any` and `all` options with the same behavior:<br />- `any` (default): rebuilds when *any* direct parent has fresh data<br />- `all`: rebuilds only when *all* direct parents have fresh data | | `freshness.build_after.count` + `freshness.build_after.period` | [`state.lag_tolerance`](https://docs.getdbt.com/reference/resource-configs/lag-tolerance.md) | Combined into a single field with shorthand values (for example, `1800s`, `30m`, `12h`, `1d`, `2w`) or Jinja expressions | Backward compatibility in dbt v2 In dbt v2, you can enable dbt State without updating your project configs first. * If `lag_tolerance` and `require_fresh_data_from` are not set, dbt State falls back to your existing `build_after` configs until `build_after` is deprecated. * If neither `build_after` nor the `state` configs exist, dbt State uses its [default configs](https://docs.getdbt.com/reference/resource-configs/dbt-state-configs.md): `lag_tolerance: 45m` and `require_fresh_data_from: any`. dbt Labs will communicate a migration timeline for state-aware orchestration users when dbt State reaches general availability. ##### Examples You can set these configs at the project level in `dbt_project.yml` or at the model level in a `.yml` file. **Project-level:** **Before (state-aware orchestration)** dbt\_project.yml ```yaml models: +freshness: build_after: count: 1 period: day updates_on: any ``` **After (dbt State)** dbt\_project.yml ```yaml models: +state: lag_tolerance: 1d require_fresh_data_from: any ``` **Model-level:** **Before (state-aware orchestration)** models/my\_model.yml ```yaml models: - name: my_model config: freshness: build_after: count: 1 period: day updates_on: any ``` **After (dbt State)** models/my\_model.yml ```yaml models: - name: my_model config: state: lag_tolerance: 1d require_fresh_data_from: any ``` #### Testing the migration You can run both configs side by side while validating. State-aware orchestration reads `freshness.build_after` and dbt State reads the `state` block, so they won't interfere with each other. Once you're confident everything looks right, remove the `build_after` configs. dbt\_project.yml ```yaml models: +freshness: build_after: # state-aware orchestration (remove once migrated) count: 1 period: day updates_on: any +state: # dbt State lag_tolerance: 1d require_fresh_data_from: any ``` #### Known differences from state-aware orchestration State-aware orchestration and dbt State differ in a few ways: * **More models rebuilding than expected**: If you notice more rebuilds after you migrate, the most common causes are: * **Views with `select *` on a `ref()` or `source()`**: dbt State can't determine which columns `select *` resolves to without querying the upstream schema, so it rebuilds these views rather than risk reusing a stale result. Views that use `select *` on a CTE are reused, because dbt can resolve the columns from the CTE definition. For more information, refer to [Views with `select *`](https://docs.getdbt.com/faqs/State/views-rebuilt.md#views-with-select). * **Non-determinism in Jinja-templated SQL**: Macros like `dbt_utils.get_relations_by_pattern` with `dbt_utils.union_relations` can return relations in a different order on each run, which produces different compiled SQL. dbt State detects a new hash and rebuilds the model. If that model has downstream dependencies, those models rebuild, too. * **Models with external sources on BigQuery**: Models that use external sources (such as Google Sheets) always rebuild because BigQuery doesn't expose modification timestamps for external sources, so dbt State can't determine freshness. To avoid this, configure [`loaded_at_field`](https://docs.getdbt.com/reference/resource-properties/freshness.md#loaded_at_field) or [`loaded_at_query`](https://docs.getdbt.com/reference/resource-properties/freshness.md#loaded_at_query) in your source definition to point to a timestamp field — this lets dbt State query a timestamp field directly to determine freshness, instead of relying on warehouse metadata. Refer to [Why is my model being rebuilt instead of reused?](https://docs.getdbt.com/faqs/State/views-rebuilt.md) for details on each cause and how to diagnose them. * **`build_after` vs `lag_tolerance`**: Both configs reduce how often a model runs when upstream data is frequently fresh, but they work differently: * `freshness.build_after` (for example, `{count: 4, period: hour}`) skips the model unless the configured interval has elapsed *and* upstream sources have new data since the last run. A SQL change alone does not trigger a rebuild; both conditions must be met. * `state.lag_tolerance` (for example, `4h`) skips the model unless upstream data is newer than the model's last run by at least the configured interval. Unlike `build_after`, a detected SQL change triggers a rebuild. * **No concurrent build detection**: In most cases, dbt State handles overlapping nodes the same way as state-aware orchestration: if a node is built by Job 1 before Job 2 starts, Job 2 sees the model as already built and reuses it. If Job 1 finishes building a node after Job 2 starts — but before Job 2 builds that node — Job 2 still builds it and uses compute to re-determine there's no new data. However, if Job 2 starts building the same snapshot or incremental model before Job 1 executing the same node, both jobs can detect the same changes in their separate transactions and commit them, which can lead to duplicate records or other data corruption. To prevent duplicate builds, ensure jobs that share the same nodes don't run at the same time — whether by configuring job schedules through the [dbt platform job scheduler](https://docs.getdbt.com/docs/deploy/job-scheduler.md) or an external orchestration tool (for example, Airflow or Dagster), or by avoiding manually triggered runs that overlap. * **Efficient Testing not yet available**: State-aware orchestration offers Efficient Testing (private beta); dbt State doesn't support it yet. #### Related docs * [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) * [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) * [dbt State configs](https://docs.getdbt.com/reference/resource-configs/dbt-state-configs.md) --- ### Model access "Model access" is not "User access" **Model groups and access** and **user groups and access** mean two different things. "User groups and access" is a specific term used in dbt to manage permissions. Refer to [User access](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md) for more info. The two concepts will be closely related, as we develop multi-project collaboration workflows this year: * Users with access to develop in a dbt project can view and modify **all** models in that project, including private models. * Users in the same dbt account *without* access to develop in a project cannot view that project's private models, and they can take a dependency on its public models only. #### Related documentation * [`groups`](https://docs.getdbt.com/docs/build/groups.md) * [`access`](https://docs.getdbt.com/reference/resource-configs/access.md) #### Groups Models can be grouped under a common designation with a shared owner. For example, you could group together all models owned by a particular team, or related to modeling a specific data source (`github`). Why define model `groups`? There are two reasons: * It turns implicit relationships into an explicit grouping, with a defined owner. By thinking about the interface boundaries *between* groups, you can have a cleaner (less entangled) DAG. In the future, those interface boundaries could be appropriate as the interfaces between separate projects. * It enables you to designate certain models as having "private" access—for use exclusively within that group. Other models will be restricted from referencing (taking a dependency on) those models. In the future, they won't be visible to other teams taking a dependency on your project—only "public" models will be. If you follow our [best practices for structuring a dbt project](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md), you're probably already using subdirectories to organize your dbt project. It's easy to apply a `group` label to an entire subdirectory at once: dbt\_project.yml ```yml models: my_project_name: marts: customers: +group: customer_success finance: +group: finance ``` Each model can only belong to one `group`, and groups cannot be nested. If you set a different `group` in that model's YAML or in-file config, it will override the `group` applied at the project level. ###### Considerations There are some considerations to keep in mind when using model governance features: * Model governance features like model access, contracts, and versions strengthen trust and stability in your dbt project. Because they add structure, they can make rollbacks harder (for example, removing model access) and increase maintenance if adopted too early. Before adding governance features, consider whether your dbt project is ready to benefit from them. Introducing governance while models are still changing can complicate future changes. * Governance features are model-specific. They don't apply to other resource types, including snapshots, seeds, or sources. This is because these objects can change structure over time (for example, snapshots capture evolving historical data) and aren't suited to guarantees like contracts, access, or versioning. #### Access modifiers Some models are implementation details, meant for reference only within their group of related models. Other models should be accessible through the [ref](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) function across groups and projects. Models can set an [access modifier](https://en.wikipedia.org/wiki/Access_modifiers) to indicate their intended level of accessibility. | Access | Referenceable by | | --------- | ---------------------------------------------------------------------------------------- | | private | Same group | | protected | Same project (or installed as a package) | | public | Any group, package, or project. When defined, rerun a production job to apply the change | If you try to reference a model outside of its supported access, you will see an error: ```shell dbt run -s marketing_model ... dbt.exceptions.DbtReferenceError: Parsing Error Node model.jaffle_shop.marketing_model attempted to reference node model.jaffle_shop.finance_model, which is not allowed because the referenced node is private to the finance group. ``` (Applies to dbt v1.12 and later) When a macro is invoked with `dbt run-operation`, dbt doesn't enforce model access or group controls, so you can use `ref()` to reference all models (including `private` and `protected` ones). By default, all models are `protected`. This means that other models in the same project can reference them, regardless of their group. This is largely for backward compatibility when assigning groups to an existing set of models, as there may already be existing references across group assignments. However, it is recommended to set the access modifier of a new model to `private` to prevent other project resources from taking dependencies on models not intentionally designed for sharing across groups. models/marts/customers.yml ```yaml # First, define the group and owner groups: - name: customer_success owner: name: Customer Success Team email: cx@jaffle.shop # Then, add 'group' + 'access' modifier to specific models models: # This is a public model -- it's a stable & mature interface for other teams/projects - name: dim_customers config: group: customer_success # changed to config in v1.10 access: public # changed to config in v1.10 # This is a private model -- it's an intermediate transformation intended for use in this context *only* - name: int_customer_history_rollup config: group: customer_success # changed to config in v1.10 access: private # changed to config in v1.10 # This is a protected model -- it might be useful elsewhere in *this* project, # but it shouldn't be exposed elsewhere - name: stg_customer__survey_results config: group: customer_success # changed to config in v1.10 access: protected # changed to config in v1.10 ``` Models with `materialized` set to `ephemeral` cannot have the access property set to public. For example, if you have a model config set as: models/my\_model.sql ```sql {{ config(materialized='ephemeral') }} ``` And the model access is defined: models/my\_project.yml ```yaml models: - name: my_model config: access: public # changed to config in v1.10 ``` It will lead to the following error: ```text ❯ dbt parse 02:19:30 Encountered an error: Parsing Error Node model.jaffle_shop.my_model with 'ephemeral' materialization has an invalid value (public) for the access field ``` #### FAQs ##### How does model access relate to database permissions? These are different! Specifying `access: public` on a model does not trigger dbt to automagically grant `select` on that model to every user or role in your data platform when you materialize it. You have complete control over managing database permissions on every model/schema, as makes sense to you & your organization. Of course, dbt can facilitate this by means of [the `grants` config](https://docs.getdbt.com/reference/resource-configs/grants.md), and other flexible mechanisms. For example: * Grant access to downstream queriers on public models * Restrict access to private models, by revoking default/future grants, or by landing them in a different schema As we continue to develop multi-project collaboration, `access: public` will mean that other teams are allowed to start taking a dependency on that model. This assumes that they've requested, and you've granted them access, to select from the underlying dataset. ##### How do I ref a model from another project? You can `ref` a model from another project in two ways: 1. [Project dependency](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md): In dbt Enterprise, you can use project dependencies to `ref` a model. dbt uses a behind-the-scenes metadata service to resolve the reference, enabling efficient collaboration across teams and at scale. 2. ["Package" dependency](https://docs.getdbt.com/docs/build/packages.md): Another way to `ref` a model from another project is to treat the other project as a package dependency. This requires installing the other project as a package, including its full source code, as well as its upstream dependencies. ##### How do I restrict access to models defined in a package? Source code installed from a package becomes part of your runtime environment. You can call macros and run models as if they were macros and models that you had defined in your own project. For this reason, model access restrictions are "off" by default for models defined in packages. You can reference models from that package regardless of their `access` modifier. The project is installed as a package can optionally restrict external `ref` access to just its public models. The package maintainer does this by setting a `restrict-access` config to `True` in `dbt_project.yml`. By default, the value of this config is `False`. This means that: * Models in the package with `access: protected` may be referenced by models in the root project, as if they were defined in the same project * Models in the package with `access: private` may be referenced by models in the root project, so long as they also have the same `group` config When `restrict-access: True`: * Any `ref` from outside the package to a protected or private model in that package will fail. * Only models with `access: public` can be referenced outside the package. dbt\_project.yml ```yml restrict-access: True # default is False ``` --- ### Model contracts #### Related documentation * [`contract`](https://docs.getdbt.com/reference/resource-configs/contract.md) * [`columns`](https://docs.getdbt.com/reference/resource-properties/columns.md) * [`constraints`](https://docs.getdbt.com/reference/resource-properties/constraints.md) #### Why define a contract? Defining a dbt model is as easy as writing a SQL `select` statement. Your query naturally produces a dataset with columns of names and types based on the columns you select and the transformations you apply. While this is ideal for quick and iterative development, for some models, constantly changing the shape of its returned dataset poses a risk when other people and processes are querying that model. It's better to define a set of upfront "guarantees" that define the shape of your model. We call this set of guarantees a "contract." While building your model, dbt will verify that your model's transformation will produce a dataset matching up with its contract, or it will fail to build. ###### Considerations There are some considerations to keep in mind when using model governance features: * Model governance features like model access, contracts, and versions strengthen trust and stability in your dbt project. Because they add structure, they can make rollbacks harder (for example, removing model access) and increase maintenance if adopted too early. Before adding governance features, consider whether your dbt project is ready to benefit from them. Introducing governance while models are still changing can complicate future changes. * Governance features are model-specific. They don't apply to other resource types, including snapshots, seeds, or sources. This is because these objects can change structure over time (for example, snapshots capture evolving historical data) and aren't suited to guarantees like contracts, access, or versioning. #### Prerequisites **These places support model contracts:** * `dbt_project.yml` file * `properties.yml` file * SQL models * Models materialized as one of the following: * `table` * `view` — views offer support for column names and data types, but not `constraints` * `incremental` — with `on_schema_change: append_new_columns` or `on_schema_change: fail` * Certain data platforms, but the supported and [enforced `constraints`](https://docs.getdbt.com/reference/resource-properties/constraints.md) vary by platform **These places do *NOT* support model contracts:** * Python models * `materialized view` or `ephemeral` — materialized SQL models * Custom materializations (unless added by the author) * Models with recursive CTE's in BigQuery * Other resource types, such as `sources`, `seeds`, `snapshots`, and so on #### Define a contract Let's say you have a model with a query like: models/marts/dim\_customers.sql ```sql -- lots of SQL final as ( select customer_id, customer_name, -- ... many more ... from ... ) select * from final ``` To enforce a model's contract, set `enforced: true` under the `contract` configuration. When enforced, your contract *must* include every column's `name` and `data_type` (where `data_type` matches one that your data platform understands). If your model is materialized as `table` or `incremental`, and depending on your data platform, you may optionally specify additional [constraints](https://docs.getdbt.com/reference/resource-properties/constraints.md), such as `not_null` (containing zero null values). models/marts/customers.yml ```yaml models: - name: dim_customers config: contract: enforced: true columns: - name: customer_id data_type: int constraints: - type: not_null - name: customer_name data_type: string ... ``` When building a model with a defined contract, dbt will do two things differently: 1. dbt will run a "preflight" check to ensure that the model's query will return a set of columns with names and data types matching the ones you have defined. This check is agnostic to the order of columns specified in your model (SQL) or YAML spec. 2. dbt will include the column names, data types, and constraints in the DDL statements it submits to the data platform, which will be enforced while building or updating the model's table, and order the columns per the contract instead of your dbt model. #### Platform constraint support Select the adapter-specific tab for more information on [constraint](https://docs.getdbt.com/reference/resource-properties/constraints.md) support across platforms. Constraints fall into three categories based on definability and platform enforcement: * **Definable and enforced** — The model won't build if it violates the constraint. * **Definable and not enforced** — The platform supports specifying the type of constraint, but a model can still build even if building the model violates the constraint. This constraint exists for metadata purposes only. This approach is more typical in cloud data warehouses than in transactional databases, where strict rule enforcement is more common. * **Not definable and not enforced** — You can't specify the type of constraint for the platform. ##### Redshift | Constraint type | Definable | Enforced | | --------------- | --------- | -------- | | not\_null | ✅ | ✅ | | primary\_key | ✅ | ❌ | | foreign\_key | ✅ | ❌ | | unique | ✅ | ❌ | | check | ❌ | ❌ | ##### Snowflake | Constraint type | Definable | Enforced | | --------------- | --------- | -------- | | not\_null | ✅ | ✅ | | primary\_key | ✅ | ❌ | | foreign\_key | ✅ | ❌ | | unique | ✅ | ❌ | | check | ❌ | ❌ | ##### BigQuery | Constraint type | Definable | Enforced | | --------------- | --------- | -------- | | not\_null | ✅ | ✅ | | primary\_key | ✅ | ❌ | | foreign\_key | ✅ | ❌ | | unique | ❌ | ❌ | | check | ❌ | ❌ | ##### Postgres | Constraint type | Definable | Enforced | | --------------- | --------- | -------- | | not\_null | ✅ | ✅ | | primary\_key | ✅ | ✅ | | foreign\_key | ✅ | ✅ | | unique | ✅ | ✅ | | check | ✅ | ✅ | ##### Spark Currently, `not_null` and `check` constraints are enforced only after a model is built. Because of this platform limitation, dbt considers these constraints definable but not enforced, which means they're not part of the *model contract* since they can't be enforced at build time. This table will change as the features evolve. | Constraint type | Definable | Enforced | | --------------- | --------- | -------- | | not\_null | ✅ | ❌ | | primary\_key | ✅ | ❌ | | foreign\_key | ✅ | ❌ | | unique | ✅ | ❌ | | check | ✅ | ❌ | ##### Databricks Currently, `not_null` and `check` constraints are enforced only after a model is built. Because of this platform limitation, dbt considers these constraints definable but not enforced, which means they're not part of the *model contract* since they can't be enforced at build time. This table will change as the features evolve. | Constraint type | Definable | Enforced | | --------------- | --------- | -------- | | not\_null | ✅ | ✅ | | primary\_key | ✅ | ❌ | | foreign\_key | ✅ | ❌ | | unique | ❌ | ❌ | | check | ✅ | ✅ | ##### Athena | Constraint type | Definable | Enforced | | --------------- | --------- | -------- | | not\_null | ❌ | ❌ | | primary\_key | ❌ | ❌ | | foreign\_key | ❌ | ❌ | | unique | ❌ | ❌ | | check | ❌ | ❌ | #### FAQs ##### Which models should have contracts? Any model meeting the criteria described above *can* define a contract. We recommend defining contracts for ["public" models](https://docs.getdbt.com/docs/mesh/govern/model-access.md) that are being relied on downstream. * Inside of dbt: Shared with other groups, other teams, and [other dbt projects](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md). * Outside of dbt: Reports, dashboards, or other systems & processes that expect this model to have a predictable structure. You might reflect these downstream uses with [exposures](https://docs.getdbt.com/docs/build/exposures.md). ##### How are contracts different from tests? A model's contract defines the **shape** of the returned dataset. If the model's logic or input data doesn't conform to that shape, the model does not build. [Data Tests](https://docs.getdbt.com/docs/build/data-tests.md) are a more flexible mechanism for validating the content of your model *after* it's built. So long as you can write the query, you can run the data test. Data tests are more configurable, such as with [custom severity thresholds](https://docs.getdbt.com/reference/resource-configs/severity.md). They are easier to debug after finding failures because you can query the already-built model, or [store the failing records in the data warehouse](https://docs.getdbt.com/reference/resource-configs/store_failures.md). In some cases, you can replace a data test with its equivalent constraint. This has the advantage of guaranteeing the validation at build time, and it probably requires less compute (cost) in your data platform. The prerequisites for replacing a data test with a constraint are: * Making sure that your data platform can support and enforce the constraint that you need. Most platforms only enforce `not_null`. * Materializing your model as `table` or `incremental` (**not** `view` or `ephemeral`). * Defining a full contract for this model by specifying the `name` and `data_type` of each column. **Why aren't tests part of the contract?** In a parallel for software APIs, the structure of the API response is the contract. Quality and reliability ("uptime") are also very important attributes of an API's quality, but they are not part of the contract per se. When the contract changes in a backwards-incompatible way, it is a breaking change that requires a bump in major version. ##### Do I need to define every column for a contract? Yes. dbt contracts apply to *all* columns defined in a model, and they require declaring explicit expectations about *all* of those columns. The explicit declaration of a contract is not an accident — it's very much the intent of this feature. At the same time, for models with many columns, we understand that this can mean a *lot* of YAML. See [dbt-labs/dbt#11764](https://github.com/dbt-labs/dbt/issues/11764) for discussion of potential approaches to generate and update model contract definitions. ##### How are breaking changes handled? When comparing to a previous project state, dbt will look for breaking changes that could impact downstream consumers. If breaking changes are detected, dbt will present a contract error. Breaking changes include: * Removing an existing column * Changing the data\_type of an existing column * Removing or modifying one of the `constraints` on an existing column (dbt v1.6 or higher) * Removing a contracted model by deleting, renaming, or disabling it (dbt v1.9 or higher). * versioned models will raise an error. unversioned models will raise a warning. More details are available in the [contract reference](https://docs.getdbt.com/reference/resource-configs/contract.md#incremental-models-and-on_schema_change). --- ### Model notifications dbt platform Set up dbt to notify model owners through email about issues in your deployment environments. Configure dbt to send email notifications to model owners about issues in deployment [environments](https://docs.getdbt.com/docs/dbt-platform-environments.md#types-of-environments) as soon as they happen — while the job is still running. Model owners can specify which statuses to receive notifications about: * **Success** and **Fails** for models * **Warning**, **Success**, and **Fails** for tests With model-level notifications, model owners can be the first ones to know about issues before anyone else (like the stakeholders). To be timely and keep the number of notifications to a reasonable amount when multiple models or tests trigger them, dbt observes the following guidelines when notifying the owners: * Send a notification to each unique owner/email during a job run about any models (with status of failure/success) or tests (with status of warning/failure/success). Each owner receives only one notification, the initial one. * No notifications sent about subsequent models or tests while a dbt job is still running. * Each owner/user who subscribes to notifications for one or more statuses (like failure, success, warning) will receive only *one* email notification at the end of the job run. * The email includes a consolidated list of all models or tests that match the statuses the user subscribed to, instead of sending separate emails for each status. Create configuration YAML files in your project for dbt to send notifications about the status of your models and tests in your deployment environments. #### Prerequisites * Your dbt administrator has [enabled the appropriate account setting](#enable-access-to-model-notifications) for you. * Your deployment environment(s) must be on a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) instead of a legacy dbt v1 version. #### Configure groups Define your [groups](https://docs.getdbt.com/docs/build/groups.md) in any `.yml` file in your [models directory](https://docs.getdbt.com/reference/project-configs/model-paths.md). Each group's owner can now specify one or multiple email addresses to receive model-level notifications. The `email` field supports a single email address as a string or a list of multiple email addresses. The following example shows how to define groups in a `groups.yml` file. models/groups.yml ```yml groups: - name: finance owner: # Email is required to receive model-level notifications, additional properties are also allowed. name: "Finance team" email: finance@dbtlabs.com - name: marketing owner: name: "Marketing team" email: marketing@dbtlabs.com config: meta: slack: '#marketing-team' # Example of multiple emails supported - name: documentation team owner: name: "Docs team" email: - docs@dbtlabs.com - community@dbtlabs.com - product@dbtlabs.com config: meta: slack: '#docs-fox' ``` tip The `owner` field supports `name` and `email`, which are required values. Additional arbitrary fields (such as `favorite_food`) are deprecated and will no longer be allowed in a future release. To store additional metadata (like Slack channels, team info, or custom attributes), use `config.meta` instead. #### Attach groups to models Attach groups to models as you would any other config, in either the `dbt_project.yml` or `whatever.yml` files. For example: models/marts.yml ```yml models: - name: sales description: "Sales data model" config: group: finance - name: campaigns description: "Campaigns data model" config: group: marketing ``` By assigning groups in the `dbt_project.yml` file, you can capture all models in a subdirectory at once. In this example, model notifications related to staging models go to the data engineering group, `marts/sales` models to the finance team, and `marts/campaigns` models to the marketing team. dbt\_project.yml ```yml config-version: 2 name: "jaffle_shop" [...] models: jaffle_shop: staging: +group: data_engineering marts: sales: +group: finance campaigns: +group: marketing ``` Attaching a group to a model also encompasses its tests, so you will also receive notifications for a model's test failures. #### Enable access to model notifications Provide dbt account members the ability to configure and receive alerts about issues with models or tests that are encountered during job runs. To use model-level notifications, your dbt account must have access to the feature. Ask your dbt administrator to enable this feature for account members by following these steps: 1. Navigate to **Notification settings** from your profile name in the sidebar (lower left-hand side). 2. From **Email notifications**, enable the setting **Enable group/owner notifications on models** under the **Model notifications** section. Then, specify which statuses to receive notifications about (Success, Warning, and/or Fails). 3. Click **Save**. [![Example of the setting Enable group/owner notifications on models](/img/docs/dbt-platform/example-enable-model-notifications.png?v=2 "Example of the setting Enable group/owner notifications on models")](#)Example of the setting Enable group/owner notifications on models --- ### Model performance dbt platform | Enterprise, Enterprise+ Catalog provides metadata on dbt runs for in-depth model performance and quality analysis. This feature assists in reducing infrastructure costs and saving time for data teams by highlighting where to fine-tune projects and deployments — such as model refactoring or job configuration adjustments. To view model performance in Catalog: 1. Navigate to **Catalog** in the left-hand menu. 2. Select your project in the left sidebar. 3. Click the **Performance** tab. [![Overview of Performance page navigation.](/img/docs/collaborate/dbt-explorer/explorer-model-performance.gif?v=2 "Overview of Performance page navigation.")](#)Overview of Performance page navigation. On-demand learning If you enjoy video courses, check out our [dbt Catalog on-demand course](https://learn.getdbt.com/courses/dbt-catalog) and learn how to best explore your dbt project(s)! #### The Performance overview page You can pinpoint areas for performance enhancement by using the Performance overview page. This page presents a comprehensive analysis across all project models and displays the longest-running models, those most frequently executed, and the ones with the highest failure rates during runs/tests. Data can be segmented by environment and job type which can offer insights into: * Most executed models (total count). * Models with the longest execution time (average duration). * Models with the most failures, detailing run failures (percentage and count) and test failures (percentage and count). Each data point links to individual models in Catalog. [![Example of Performance overview page](/img/docs/collaborate/dbt-explorer/example-performance-overview-page.png?v=2 "Example of Performance overview page")](#)Example of Performance overview page You can view historical metadata for up to the past three months. Select the time horizon using the filter, which defaults to a two-week lookback. [![Example of dropdown](/img/docs/collaborate/dbt-explorer/ex-2-week-default.png?v=2 "Example of dropdown")](#)Example of dropdown #### The Model performance tab The **Model performance** section in Catalog displays historical trends to help you identify optimization opportunities and understand model resource consumption. [![Cost Insights in Catalog](/img/docs/dbt-platform/cost-insights/cost-insights-model.png?v=2 "Cost Insights in Catalog")](#)Cost Insights in Catalog ##### Key metrics The **Model performance** section displays the following metrics that summarize the overall cost and optimization impact for your project: * **Total cost reduction** * **Total % reduction** * **Total query run time deduction** * **Reused assets** (when dbt State or state-aware orchestration is enabled) ##### Filters Use the time period filter to customize the data you want to view: from the last 3 months up to the last 1 week. For **Cost insights**, **Usage**, and **Query run time** tabs, you can set the view granularity by **Daily**, **Weekly**, or **Monthly**. ##### Visualization tabs * **Cost insights**: Shows the estimated warehouse costs incurred by this model and cost reduction from dbt State or state-aware orchestration. * **Usage**: Shows the estimated warehouse usage consumed by this model over time. The **Usage** tab represents generic usage for your warehouse. The specific unit depends on your data warehouse: * Snowflake: Credits * BigQuery: Slot hours or bytes scanned (currently combined into one generic usage number) * Databricks: Databricks Units (DBUs) * Amazon Redshift Serverless: Redshift Processing Unit hours (RPU-hours) * Amazon Redshift Provisioned: Node-hours * **Query run time**: Shows the estimated query execution time and the reduction in run duration from dbt State or state-aware orchestration. * **Build time**: Shows average execution time for the model and how it trends over the selected period. * **Build count**: Tracks how many times the model was built or reused, including any failures or errors. * **Test results**: Displays test execution outcomes and pass/fail rates for tests on this model. * **Consumption queries**: Shows queries running against this model, helping you understand downstream usage patterns. ##### Table view For **Cost insights**, **Usage**, and **Query run time** tabs, you can access the table view by clicking **Show table**, which provides detailed optimization data such as models reused, usage reduction, and cost reduction. Use the **All** and **Jobs** buttons to switch between views: * **All**: Shows aggregated cost data across all jobs in the project. * **Jobs**: Shows cost data broken down by individual jobs. When viewing the table, you can export the data as a CSV file using the **Download** button. When **Jobs** is selected, the CSV export includes job-level data. ##### Chart interactions For **Build time** and **Build count** tabs: * Click on any data point in the charts to see a detailed table listing all job runs for that day. * Each row in the table provides a direct link to the run details if you want to investigate further. --- ### Model query history dbt platform | Enterprise, Enterprise+ Model query history helps data teams track model usage by analyzing query logs. Model query history allows you to: * View the count of consumption queries for a model based on the data warehouse's query logs. * Provides data teams insight, so they can focus their time and infrastructure spend on the worthwhile used data products. * Enable analysts to find the most popular models used by other people. Model query history is powered by a single consumption query of the query log table in your data warehouse aggregated on a daily basis.  What is a consumption query? Consumption query is a metric of queries in your dbt project that has used the model in a given time. It filters down to `select` statements only to gauge model consumption and excludes dbt model build and test executions. So for example, if `model_super_santi` was queried 10 times in the past week, it would count as having 10 consumption queries for that particular time period. Model query history is supported in the following data warehouses: * Snowflake (Enterprise-tier or higher) * BigQuery * Redshift * Databricks #### Prerequisites To access the features, you should meet the following requirements: * You have a dbt account on an [Enterprise-tier plan](https://www.getdbt.com/pricing/). Single-tenant accounts should contact their account representative for setup. * You have set up a [production](https://docs.getdbt.com/docs/deploy/deploy-environments.md#set-as-production-environment) deployment environment for each project you want to explore, with at least one successful job run. * You have [admin permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) in dbt to edit project settings or production environment settings. * You use Snowflake, BigQuery, Redshift, or Databricks as your data warehouse and can enable [query history permissions](#credential-permissions) or work with an admin to do so. * For Snowflake users: You must have a Snowflake Enterprise-tier or higher subscription. #### Enable query history in dbt New and existing production environments * **New production environments:** dbt automatically enables query history when you create a new [production deployment environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md#set-as-production-environment) in **Orchestration** (marked **PROD**). This applies to Snowflake, BigQuery, Redshift, and Databricks on [Enterprise-tier plans](https://www.getdbt.com/pricing/). Development environments are not included. * **Existing production environments:** You must enable query history manually in your **PROD** environment settings. Configure the [credential permissions](#credential-permissions) for your warehouse prior to starting the steps in this section. To enable model query history in dbt, follow these steps: 1. Navigate to **Orchestration** and then **Environments**. 2. Select the environment marked **PROD** and click **Settings**. 3. Click **Edit** and scroll to the **Query History** section. 4. Click the **Test Permissions** button to validate the deployment credentials permissions are sufficient to support query history. 5. Click the **Enable query history** box to enable. 6. **Save** your settings. If query history fails to retrieve data, dbt automatically disables it to prevent unintended warehouse costs. * If the failure is temporary (like a network timeout), dbt may retry. * If the problem keeps happening (for example, missing permissions), dbt turns off query history so customers don’t waste warehouse compute. To turn it back on, click **Test Permissions** in **Environment settings**. If the test succeeds, dbt re-enables the environment. [![Enable query history in your environment settings.](/img/docs/collaborate/dbt-explorer/enable-query-history.png?v=2 "Enable query history in your environment settings.")](#)Enable query history in your environment settings. #### Credential permissions This section explains the permissions and steps you need to enable and view model query history in Catalog. The model query history feature uses the credentials in your production environment to gather metadata from your data warehouse’s query logs. This means you may need elevated permissions with the warehouse. Before making any changes to your data platform permissions, confirm the configured permissions in dbt: 1. Navigate to **Orchestration** and then **Environments**. 2. Select the Environment marked **PROD** and click **Settings**. 3. Click **Edit**. 4. Click the sync icon in the **Connection profiles** section. 5. Look at the information under **Deployment credentials**. * Note: Querying query history entails warehouse costs / uses credits. [![Click the sync icon in the Connection profiles section.](/img/docs/dbt-platform/sync-icon.png?v=2 "Click the sync icon in the Connection profiles section.")](#)Click the sync icon in the Connection profiles section. [![Confirm your deployment credentials in your environment settings page.](/img/docs/collaborate/dbt-explorer/model-query-credentials.png?v=2 "Confirm your deployment credentials in your environment settings page.")](#)Confirm your deployment credentials in your environment settings page. 6. Copy or cross reference those credential permissions with the warehouse permissions and grant your user the right permissions. ##### Snowflake model query history Model query history uses metadata tables available to [Snowflake Enterprise-tier](https://docs.snowflake.com/en/user-guide/intro-editions#enterprise-edition) accounts or higher: `QUERY_HISTORY` and `ACCESS_HISTORY`. The Snowflake user in the production environment must have the `GOVERNANCE_VIEWER` permission to view the data. Before enabling model query history, your `ACCOUNTADMIN` must run the following grant statement in Snowflake to ensure access: ```sql GRANT DATABASE ROLE SNOWFLAKE.GOVERNANCE_VIEWER TO ROLE <YOUR_DBT_CLOUD_DEPLOYMENT_ROLE>; ``` Without this grant, model query history won't display any data. For more information, refer to the [Snowflake documentation](https://docs.snowflake.com/en/sql-reference/account-usage#enabling-other-roles-to-use-schemas-in-the-snowflake-database). ##### BigQuery model query history The model query history uses metadata from the [`INFORMATION_SCHEMA.JOBS` view](https://docs.cloud.google.com/bigquery/docs/information-schema-jobs) in BigQuery. To access the metadata, the production environment user must have the correct [IAM role](https://docs.cloud.google.com/bigquery/docs/access-control#bigquery.resourceViewer) or permission to access this data: * If you use a BigQuery provided role, we recommend `roles/bigquery.resourceViewer`. * If you use a custom role, ensure it includes the `bigquery.jobs.listAll permission`. ##### Redshift model query history Model query history uses the `SYS_QUERY_HISTORY` and `SYS_QUERY_DETAIL` system views in Redshift. By default, users can only see their own queries in these views. To surface query history across all warehouse users, your database admin must grant the production environment credentials one of the following: * The `sys:monitor` role: ```sql GRANT ROLE sys:monitor TO <YOUR_DBT_USER>; ``` * Unrestricted syslog access on the user: ```sql ALTER USER <YOUR_DBT_USER> SYSLOG ACCESS UNRESTRICTED; ``` Without one of these, model query history won't display data from other users. For more information, refer to the [Redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/cm_chap_system-tables.html#c_visibility-of-data). ###### Redshift considerations Redshift model query history is derived from physical table scans (`SYS_QUERY_DETAIL` where `step_name = 'scan'`). This means usage data reflects a high-quality signal rather than an exact query count. Because Redshift expands regular views at execution time, scans are attributed to underlying base tables and view names do not appear in scan metadata. Materialized views do not have this limitation. Keep the following in mind when reviewing usage data: * Models materialized as `view` will likely show zero usage. * Models materialized as `table` or `incremental` are attributed correctly. * `ephemeral` models cannot be attributed. If your project relies heavily on views, usage may appear lower than expected. This is a known limitation of scan-based attribution rather than missing data. ##### Databricks model query history Model query history uses two Unity Catalog system tables: `system.query.history` and `system.access.table_lineage`. Before granting access, confirm the following prerequisites are met: * Your account has at least one Unity Catalog-enabled workspace. Users who are still on the legacy Hive Metastore cannot use model query history. * The metastore is on Privilege Model Version 1.0. For more information, refer to [Unity Catalog privileges reference](https://docs.databricks.com/aws/en/data-governance/unity-catalog/access-control/privileges-reference). * The `access` and `query` schemas are enabled in the `system` catalog. A Databricks account admin can enable them under **Account Console** > **Settings** > **System tables**. Grant the service principal used in your production environment the following privileges: ```sql GRANT USE CATALOG ON CATALOG system TO `<YOUR_SERVICE_PRINCIPAL>`; GRANT USE SCHEMA ON SCHEMA system.access TO `<YOUR_SERVICE_PRINCIPAL>`; GRANT USE SCHEMA ON SCHEMA system.query TO `<YOUR_SERVICE_PRINCIPAL>`; GRANT SELECT ON TABLE system.access.table_lineage TO `<YOUR_SERVICE_PRINCIPAL>`; GRANT SELECT ON TABLE system.query.history TO `<YOUR_SERVICE_PRINCIPAL>`; ``` note `SELECT` on the tables alone is not sufficient. Databricks also requires `USE CATALOG` on the parent catalog and `USE SCHEMA` on the parent schemas. For more information, refer to the [Databricks Unity Catalog privileges documentation](https://docs.databricks.com/aws/en/data-governance/unity-catalog/manage-privileges/privileges). ###### Databricks considerations Keep the following in mind when using model query history with Databricks: * **Usage data may be incomplete**. Databricks does not capture lineage for all queries, so some queries may not appear in usage data. This is a known Databricks limitation. For more information, refer to [Databricks lineage system tables](https://docs.databricks.com/aws/en/admin/system-tables/lineage). * **Lineage data is not real-time**. `system.access.table_lineage` is updated throughout the day with no guaranteed timeframe. Delays of up to 24 hours are possible. * **Views may produce inconsistent lineage records**. Materialized views and streaming tables are included in lineage events. Standard views may or may not produce lineage records, depending on how Databricks Unity Catalog resolves them. * **Statement text encryption affects query filtering**. If statement text encryption is enabled in Databricks, `statement_text` in `system.query.history` is `NULL`. In this case, dbt test queries cannot be filtered and may be counted as user queries. * **Serverless compute lineage may be incomplete**. `system.query.history` includes queries from SQL warehouses and serverless compute, but lineage attribution for serverless executions may not always be captured. #### View query history in Catalog To enhance your discovery, you can view your model query history in various locations within Catalog: * [View from Performance charts](#view-from-performance-charts) * [View from Project lineage](#view-from-project-lineage) * [View from Model list](#view-from-model-list) ##### View from Performance charts 1. Navigate to Catalog by clicking **Catalog** in the navigation. 2. In the main **Overview** page, click on **Performance** under the **Project details** section. Scroll down to view the **Most consumed models**. 3. Use the dropdown menu on the right to select the desired time period, with options available for up to the past 3 months. [![View most consumed models on the 'Performance' page in dbt Catalog.](/img/docs/collaborate/dbt-explorer/most-consumed-models.png?v=2 "View most consumed models on the 'Performance' page in dbt Catalog.")](#)View most consumed models on the 'Performance' page in dbt Catalog. 4. Click on a model for more details and go to the **Performance** tab. 5. On the **Performance** tab, scroll down to the **Model performance** section. 6. Select the **Consumption queries** tab to view the consumption queries over a given time for that model. [![View consumption queries over time for a given model.](/img/docs/collaborate/dbt-explorer/model-consumption-queries.png?v=2 "View consumption queries over time for a given model.")](#)View consumption queries over time for a given model. ##### View from Project lineage 1. To view your model in your project lineage, go to the main **Overview page** and click on **Project lineage.** 2. In the lower left of your lineage, click on **Lenses** and select **Consumption queries**. [![View model consumption queries in your lineage using the 'Lenses' feature.](/img/docs/collaborate/dbt-explorer/model-consumption-lenses.png?v=2 "View model consumption queries in your lineage using the 'Lenses' feature.")](#)View model consumption queries in your lineage using the 'Lenses' feature. 3. Your lineage should display a small purple box above each model, indicating the consumption query number. The number for each model represents the model consumption over the last 30 days. ##### View from Model list 1. To view a list of models, go to the main **Overview page**. 2. In the left navigation, go to the **Resources** tab and click on **Models** to view the models list. 3. You can view the consumption query count for the models and sort by most or least consumed. The consumption query number for each model represents the consumption over the last 30 days. [![View models consumption in the 'Models' list page under the 'Consumption' column.](/img/docs/collaborate/dbt-explorer/model-consumption-list.jpg?v=2 "View models consumption in the 'Models' list page under the 'Consumption' column.")](#)View models consumption in the 'Models' list page under the 'Consumption' column. --- ### Model versions Model versions, dbt\_project.yml versions, and .yml versions The word "version" appears in multiple places in docs site and with different meanings: * [Model versions](https://docs.getdbt.com/docs/mesh/govern/model-versions.md) — A dbt Mesh feature that enables better governance and data model management by allowing you to track changes and updates to models over time. * [dbt\_project.yml version](https://docs.getdbt.com/reference/project-configs/version.md#dbt_projectyml-versions)(optional) — `dbt_project.yml` version is unrelated to Mesh and refers to the compatibility of the dbt project with a specific version of dbt. * [.yml property file version](https://docs.getdbt.com/reference/project-configs/version.md#yml-property-file-versions)(optional) — Version numbers within .yml property files inform how dbt parses those YAML files. Unrelated to Mesh. Versioning APIs is a hard problem in software engineering. The root of the challenge is that the producers and consumers of an API have competing incentives: * Producers of an API need the ability to modify its logic and structure. There is a real cost to maintaining legacy endpoints forever, but losing the trust of downstream users is far costlier. * Consumers of an API need to trust in its stability: their queries will keep working, and won't break without warning. Although migrating to a newer API version incurs an expense, an unplanned migration is far costlier. When sharing a final dbt model with other teams or systems, that model is operating like an API. When the producer of that model needs to make significant changes, how can they avoid breaking the queries of its users downstream? Model versioning is a tool to tackle this problem, thoughtfully and head-on. The goal is not to make the problem go away entirely, nor to pretend it's easier or simpler than it is. ###### Considerations There are some considerations to keep in mind when using model governance features: * Model governance features like model access, contracts, and versions strengthen trust and stability in your dbt project. Because they add structure, they can make rollbacks harder (for example, removing model access) and increase maintenance if adopted too early. Before adding governance features, consider whether your dbt project is ready to benefit from them. Introducing governance while models are still changing can complicate future changes. * Governance features are model-specific. They don't apply to other resource types, including snapshots, seeds, or sources. This is because these objects can change structure over time (for example, snapshots capture evolving historical data) and aren't suited to guarantees like contracts, access, or versioning. #### Related documentation * [`versions`](https://docs.getdbt.com/reference/resource-properties/versions.md) * [`latest_version`](https://docs.getdbt.com/reference/resource-properties/latest_version.md) * [`include` and `exclude`](https://docs.getdbt.com/reference/resource-properties/versions.md#include) * [`ref` with `version` argument](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#versioned-ref) #### Why version a model? If a model defines a ["contract"](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md) (a set of guarantees for its structure), it's also possible to change that model's structure in a way that breaks the previous set of guarantees. This could be as obvious as removing or renaming a column, or more subtle, like changing its data type or nullability. One approach is to force every model consumer to immediately handle the breaking change as soon as it's deployed to production. This is actually the appropriate answer at many smaller organizations, or while rapidly iterating on a not-yet-mature set of data models. But it doesn’t scale well beyond that. Instead, for mature models at larger organizations, powering queries inside & outside dbt, the model owner can use **model versions** to: * Test "prerelease" changes (in production, in downstream systems) * Bump the latest version, to be used as the canonical source of truth * Offer a migration window off the "old" version During that migration window, anywhere that model is being used downstream, it can continue to be referenced at a specific version. dbt 1.6 introduced first-class support for **deprecating models** by specifying a [`deprecation_date`](https://docs.getdbt.com/reference/resource-properties/deprecation_date.md). Taken together, model versions and deprecation offer a pathway for model producers to *sunset* old models, and consumers the time to *migrate* across breaking changes. It's a way of managing change across an organization: develop a new version, bump the latest, slate the old version for deprecation, update downstream references, and then remove the old version. There is a real trade-off that exists here—the cost to frequently migrate downstream code, and the cost (and clutter) of materializing multiple versions of a model in the data warehouse. Model versions do not make that problem go away, but by setting a deprecation date, and communicating a clear window for consumers to gracefully migrate off old versions, they put a known boundary on the cost of that migration. #### When should you version a model? By enforcing a model's contract, dbt can help you catch unintended changes to column names and data types that could cause a big headache for downstream queriers. If you're making these changes intentionally, you should create a new model version. If you're making a non-breaking change, you don't need a new version—such as adding a new column, or fixing a bug in an existing column's calculation. Of course, it's possible to change a model's definition in other ways—recalculating a column in a way that doesn't change its name, data type, or enforceable characteristics—but would substantially change the results seen by downstream queriers. This is always a judgment call. As the maintainer of a widely-used model, you know best what's a bug fix and what's an unexpected behavior change. The process of sunsetting and migrating model versions requires real work, and likely significant coordination across teams. You should opt for non-breaking changes whenever possible. Inevitably, however, these non-breaking additions will leave your most important models with lots of unused or deprecated columns. Rather than constantly adding a new version for each small change, you should opt for a predictable cadence (once or twice a year, communicated well in advance) where you bump the "latest" version of your model, removing columns that are no longer being used. #### How is this different from "version control"? [Version control](https://docs.getdbt.com/docs/platform/git/git-version-control.md) allows your team to collaborate simultaneously on a single code repository, manage conflicts between changes, and review changes before deploying into production. In that sense, version control is an essential tool for versioning the deployment of an entire dbt project—always the latest state of the `main` branch. In general, only one version of your project code is deployed into an environment at a time. If something goes wrong, you have the ability to roll back changes by reverting a commit or pull request, or by leveraging data platform capabilities around "time travel." When you make updates to a model's source code — its logical definition, in SQL or Python, or related configuration — dbt can [compare your project to the previous state](https://docs.getdbt.com/reference/node-selection/syntax.md#about-node-selection), enabling you to rebuild only models that have changed, and models downstream of a change. In this way, it's possible to develop changes to a model, quickly test in CI, and efficiently deploy into production — all coordinated via your version control system. **Versioned models are different.** Defining model `versions` is appropriate when people, systems, and processes beyond your team's control, inside or outside of dbt, depend on your models. You can neither simply go migrate them all, nor break their queries on a whim. You need to offer a migration path, with clear diffs and deprecation dates. Multiple versions of a model will live in the same code repository at the same time, and be deployed into the same data environment simultaneously. This is similar to how web APIs are versioned: Multiple versions live simultaneously, two or three, and not more). Over time, newer versions come online, and older versions are sunsetted . #### How is this different from just creating a new model? Honestly, it's only a little bit different! There isn't much magic here, and that's by design. You've always been able to copy-paste, create a new model file, and name it `dim_customers_v2.sql`. Why should you opt for a "real" versioned model instead? As the **producer** of a versioned model: * You keep track of all live versions in one place, rather than scattering them throughout the codebase * You can reuse the model's configuration, and highlight just the diffs between versions * You can select models to build (or not) based on whether they're a `latest`, `prerelease`, or `old` version * dbt will notify consumers of your versioned model when new versions become available, or when they are slated for deprecation As the **consumer** of a versioned model: * You use a consistent `ref`, with the option of pinning to a specific live version * You will be notified throughout the life cycle of a versioned model All versions of a model preserve the model's original name. They are `ref`'d by that name, rather than the name of the file that they're defined in. By default, the `ref` resolves to the latest version (as declared by that model's maintainer), but you can also `ref` a specific version of the model, with a `version` keyword. Let's say that `dim_customers` has three versions defined: `v2` is the "latest", `v3` is "prerelease," and `v1` is an old version that's still within its deprecation window. Because `v2` is the latest version, it gets some special treatment: it can be defined in a file without a suffix, and `ref('dim_customers')` will resolve to `v2` if a version pin is not specified. The table below breaks down the standard conventions: | v | version | `ref` syntax | File name | Database relation | | - | ------------ | ---------------------------------------------------------- | ------------------------------------------------- | ---------------------------------------------------------------------------- | | 3 | "prerelease" | `ref('dim_customers', v=3)` | `dim_customers_v3.sql` | `analytics.dim_customers_v3` | | 2 | "latest" | `ref('dim_customers', v=2)` **and** `ref('dim_customers')` | `dim_customers_v2.sql` **or** `dim_customers.sql` | `analytics.dim_customers_v2` **and** `analytics.dim_customers` (recommended) | | 1 | "old" | `ref('dim_customers', v=1)` | `dim_customers_v1.sql` | `analytics.dim_customers_v1` | As you'll see in the implementation section below, a versioned model can reuse the majority of its YAML properties and configuration. Each version needs to only say how it *differs* from the shared set of attributes. This gives you, as the producer of a versioned model, the opportunity to highlight the differences across versions—which is otherwise difficult to detect in models with dozens or hundreds of columns—and to clearly track, in one place, all versions of the model which are currently live. dbt also supports [`version`-based selection](https://docs.getdbt.com/reference/node-selection/methods.md#version). For example, you could define a [default YAML selector](https://docs.getdbt.com/reference/node-selection/yaml-selectors.md#default) that avoids running any old model versions in development, even while you continue to run them in production through a sunset and migration period. (You could accomplish something similar by applying `tags` to these models, and cycling through those tags over time.) selectors.yml ```yml selectors: - name: exclude_old_versions default: "{{ target.name == 'dev' }}" definition: method: fqn value: "*" exclude: - method: version value: old ``` Because dbt knows that these models are *actually the same model*, it can notify downstream consumers as new versions become available, and as older versions are slated for deprecation. ```bash Found an unpinned reference to versioned model 'dim_customers'. Resolving to latest version: my_model.v2 A prerelease version 3 is available. It has not yet been marked 'latest' by its maintainer. When that happens, this reference will resolve to my_model.v3 instead. Try out v3: {{ ref('my_dbt_project', 'my_model', v='3') }} Pin to v2: {{ ref('my_dbt_project', 'my_model', v='2') }} ``` #### How to create a new version of a model Most often, you'll start with a model that is not yet versioned. Let's go back in time to when `dim_customers` was a simple standalone model, with an enforced contract. For simplicity, let's pretend it has only two columns, `customer_id` and `country_name`, though most mature models will have many more. models/dim\_customers.sql ```sql -- lots of sql final as ( select customer_id, country_name from ... ) select * from final ``` models/schema.yml ```yaml models: - name: dim_customers config: materialized: table contract: enforced: true columns: - name: customer_id description: This is the primary key data_type: int - name: country_name description: Where this customer lives data_type: varchar ``` Let's say you need to make a breaking change to the model: Removing the `country_name` column, which is no longer reliable. First, create a new model file (SQL or Python) encompassing those breaking changes. The default convention is naming the new file with a `_v<version>` suffix. Let's make a new file, named `dim_customers_v2.sql`. (We don't need to rename the existing model file just yet, while it's still the "latest" version.) models/dim\_customers\_v2.sql ```sql -- lots of sql final as ( select customer_id -- country_name has been removed! from ... ) select * from final ``` Now, you could define properties and configuration for `dim_customers_v2` as a new standalone model, with no actual relation to `dim_customers` save a striking resemblance. Instead, we're going to declare that these are versions of the same model, both named `dim_customers`. We can define their properties in common, and then **just** highlight the diffs between them. (Or, you can choose to define each model version with full specifications, and repeat the values they have in common.) ##### Diffs only (recommended) models/schema.yml ```yaml models: - name: dim_customers latest_version: 1 config: materialized: table contract: {enforced: true} columns: - name: customer_id description: This is the primary key data_type: int - name: country_name description: Where this customer lives data_type: varchar # Declare the versions, and highlight the diffs versions: - v: 1 # Matches what's above -- nothing more needed - v: 2 # Removed a column -- this is the breaking change! columns: # This means: use the 'columns' list from above, but exclude country_name - include: all exclude: [country_name] ``` ##### Fully specified models/schema.yml ```yaml models: - name: dim_customers latest_version: 1 # declare the versions, and fully specify them versions: - v: 2 config: materialized: table contract: {enforced: true} columns: - name: customer_id description: This is the primary key data_type: int # no country_name column - v: 1 config: materialized: table contract: {enforced: true} columns: - name: customer_id description: This is the primary key data_type: int - name: country_name description: Where this customer lives data_type: varchar ``` Note: If none of your model versions specify columns, you don't need to define columns at all and can omit the `columns/include`/`exclude` keys from the versioned model. In this case, dbt will automatically use all top-level columns for all versions. The configuration above says: Instead of two unrelated models, I have two versioned definitions of the same model: `dim_customers_v1` and `dim_customers_v2`. **Where are they defined?** dbt expects each model version to be defined in a file named `<model_name>_v<v>`. In this case: `dim_customers_v1.sql` and `dim_customers_v2.sql`. It's also possible to define the "latest" version in `dim_customers.sql` (no suffix), without additional configuration. Finally, you can override this convention by setting [`defined_in: any_file_name_you_want`](https://docs.getdbt.com/reference/resource-properties/versions.md#defined_in)—but we strongly encourage you to follow the convention, unless you have a very good reason. **Where will they be materialized?** Each model version will create a database relation with alias `<model_name>_v<v>`. In this case: `dim_customers_v1` and `dim_customers_v2`. See [the section below](#configuring-database-location-with-alias) for more details on configuring aliases. **Which version is "latest"?** If not specified explicitly, the `latest_version` would be `2`, because it's numerically greatest. In this case, we've explicitly specified that `latest_version: 1`. That means `v2` is a "prerelease," in early development and testing. When we're ready to roll out `v2` to everyone by default, we would bump `latest_version: 2`, or remove `latest_version` from the specification. ##### Configuring versioned models You can reconfigure each version independently. For example, you could materialize `v2` as a table and `v1` as a view: models/schema.yml ```yml versions: - v: 2 config: materialized: table - v: 1 config: materialized: view ``` Like with all config inheritance, any configs set *within* the versioned model's definition (`.sql` or `.py` file) will take precedence over the configs set in YAML. ##### Configuring database location with `alias` Following the example, let's say you wanted `dim_customers_v1` to continue populating the database table named `dim_customers`. That's what the table was named previously, and you may have several other dashboards or tools expecting to read its data from `<dbname>.<schemaname>.dim_customers`. You could use the `alias` configuration: models/schema.yml ```yml - v: 1 config: alias: dim_customers # keep v1 in its original database location ``` (Applies to dbt v1.12 and later) If you want a view that always tracks the latest model version instead of pinning to a specific one, see [Pointing to the latest version](#pointing-to-the-latest-version). ##### Pointing to the latest version The [`latest_version_pointer`](https://docs.getdbt.com/reference/resource-configs/latest_version_pointer.md) config automatically creates a view named after the model's base name (for example, `dim_customers`) that always points to the latest versioned relation (for example, `dim_customers_v2`). When you enable it, querying outside of dbt always returns the current version. This config only applies to versioned models. Enable this feature in your project by setting the [`latest_version_pointer_enabled_by_default`](https://docs.getdbt.com/reference/global-configs/behavior-flags/latest_version_pointer_enabled_by_default.md) flag to `true` in `dbt_project.yml`, or enable it per model with the `latest_version_pointer.enabled` config: ##### Enable globally dbt\_project.yml ```yaml flags: latest_version_pointer_enabled_by_default: true ``` ##### Enable per model models/schema.yml ```yaml models: - name: dim_customers versions: - v: 1 - v: 2 config: latest_version_pointer: enabled: true # overrides the project flag when set alias: dim_customers_current # optional custom name ``` The pointer view uses the model's base name by default (for example, `dim_customers`). You can override the alias per model with `latest_version_pointer.alias`, or globally by overriding the [`generate_latest_version_pointer_alias`](https://docs.getdbt.com/docs/build/custom-aliases.md#generate_latest_version_pointer_alias) macro in your project. ###### Naming collisions To prevent naming collisions, dbt raises a `dbt1005` error if the latest version's alias is the same as the pointer view name. In dbt v2, where `latest_version_pointer` is enabled by default, this error can surface on models that have an explicit `alias` matching the model's base name, even if you never configured the pointer yourself. For example, the following configuration would raise `dbt1005` because both `dim_customers_v2` and the pointer view would resolve to `dim_customers`: ```text dbt1005 (Cannot create latest version pointer: the latest version of 'dim_customers' is already aliased to 'dim_customers') ``` ```yaml models: - name: dim_customers versions: - v: 1 - v: 2 config: alias: dim_customers # collides with the pointer view name config: latest_version_pointer: enabled: true ``` To fix this, select one of the following options: * [Remove the `alias` (recommended)](#remove-the-alias-recommended) * [Disable the latest version pointer for that model](#disable-the-latest-version-pointer-for-that-model) * [Set a unique `alias`](#set-a-unique-alias) * [Override the `generate_latest_version_pointer_alias` macro](#override-the-generate_latest_version_pointer_alias-macro) ###### Remove the `alias` (recommended) Remove the `alias` from the latest version and let the automatic pointer handle it: ```yaml config: alias: dim_customers ``` ###### Disable the latest version pointer for that model This approach is immediately backward-compatible for pre-existing `alias` configs: ```yaml config: alias: dim_customers config: latest_version_pointer: enabled: false ``` ###### Set a unique `alias` ```yaml config: alias: dim_customers_latest ``` ###### Override the `generate_latest_version_pointer_alias` macro Override the [`generate_latest_version_pointer_alias`](https://docs.getdbt.com/docs/build/custom-aliases.md#generate_latest_version_pointer_alias) macro to use a different naming convention globally: macros/generate\_latest\_version\_pointer\_alias.sql ```sql {% macro generate_latest_version_pointer_alias(custom_alias_name=none, node=none) -%} {{ node.name ~ "_latest" }} {%- endmacro %} ``` ##### Run a model with multiple versions To run a model with multiple versions, you can use the [`--select` flag](https://docs.getdbt.com/reference/node-selection/syntax.md). For example: * Run all versions of `dim_customers`: ```bash dbt run --select dim_customers # Run all versions of the model ``` * Run only version 2 of `dim_customers`: You can use either of the following commands (both achieve the same result): ```bash dbt run --select dim_customers.v2 # Run a specific version of the model dbt run --select dim_customers_v2 # Alternative syntax for the specific version ``` * Run the latest version of `dim_customers` using the `--select` flag shorthand: ```bash dbt run -s dim_customers,version:latest # Run the latest version of the model ``` These commands provide flexibility in managing and executing different versions of a dbt model. ##### Optimizing model versions How you define each model version is completely up to you. While it's easy to start by copy-pasting from one model's SQL definition into another, you should think about *what actually is changing* from one version to another. For example, if your new model version is only renaming or removing certain columns, you could define one version as a view on top of the other one: models/dim\_customers\_v2.sql ```sql {{ config(materialized = 'view') }} {% set dim_customers_v1 = ref('dim_customers', v=1) %} select {{ dbt_utils.star(from=dim_customers_v1, except=["country_name"]) }} from {{ dim_customers_v1 }} ``` Of course, if one model version makes meaningful and substantive changes to logic in another, it may not be possible to optimize it in this way. At that point, the cost of human intuition and legibility is more important than the cost of recomputing similar transformations. We expect to develop more opinionated recommendations as teams start adopting model versions in practice. One recommended pattern we can envision: Prioritize the definition of the `latest_version`, and define other versions (old and prerelease) based on their diffs from the latest. How? * Define the properties and configuration for the latest version in the top-level model YAML, and the diffs for other versions below (via `include`/`exclude`) * Where possible, define other versions as `select` transformations, which take the latest version as their starting point * When bumping the `latest_version`, migrate the SQL and YAML accordingly. In the example above, the third point might be tricky. It's easier to *exclude* `country_name`, than it is to add it back in. Instead, we might need to keep around the full original logic for `dim_customers_v1`—but materialize it as a `view`, to minimize the data warehouse cost of building it. If downstream queriers see slightly degraded performance, it's still significantly better than broken queries, and all the more reason to migrate to the new "latest" version. #### Coordinate model versioning Safely releasing a new model version requires coordination between model producers (who build the models) and model consumers (who depend on them). For practical guidance on how producers and consumers should communicate, test, and roll out versioned models across projects, refer to [Coordinating model versions best practices](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-6-coordinate-versions.md). --- ### Models and pricing Learn how dbt Wizard usage is metered and how model access works across the dbt platform and local CLI. This page covers which AI models dbt Wizard can use and how usage is priced. To start a trial, add a credit card, or set a spend limit, refer to [Trial and billing](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/trial-and-billing.md). Get started with free dbt-managed usage credits. Every credit balance is *per account, not per user*, which means everyone on your account shares the same pool. Enterprise and Enterprise+ accounts get monthly usage credits, and all other plans get a 30-day free trial with $100 in credits. There are two ways to pay for AI usage: let dbt Labs handle the models and the billing, or bring your own provider key. Here's how they compare: | | [dbt managed](#dbt-managed-providers) | [Bring your own key (BYOK)](#bring-your-own-key-byok) | | ------------------ | -------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | | **Choose it when** | You want to start now, with no AI provider account of your own | You already have provider credits, or your company requires a specific vendor | | **Who bills you** | dbt Labs, through your dbt account | Your AI provider, directly | | **Spend controls** | Consumption pool and spend limits in **Billing & Usage** | Managed with your provider | | **Setup** | Nothing to configure | [Configure a provider key](https://docs.getdbt.com/docs/platform/wizard-byok-platform.md) | Both options are metered per token. For dbt managed rates, refer to the [Model Provider Rate Table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model). With BYOK, you pay whatever your provider charges. Most users start with the dbt managed model and only reach for BYOK or a specific managed frontier provider when they need it. #### Key terms These four terms come up on every dbt Wizard billing page: | Term | What it means | | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Token | The unit of AI model usage. How many tokens you use, and the rate you pay for them, depends on the model, the context window, the length and complexity of your prompt, and the size of the response | | Usage credits | The free balance included with your plan, shared by everyone on the account, whether thats trial credits on Developer and Starter, or monthly credits on Enterprise and Enterprise+. Scoped to dbt Wizard only | | Consumption pool | Your overall dbt managed usage balance, including any pool you purchase once your credits run out. A purchased pool covers both dbt Wizard and dbt State. Shown as **Consumption pool** in **Billing & Usage** | | Spend limit | The maximum dbt managed usage your account can consume in a billing period | Usage draws down from your credits first, then your consumption pool, and stops at your spend limit. Usage costs are passed through directly from the AI provider. #### dbt managed providers In the dbt platform, dbt Wizard offers a managed OpenAI model by default. dbt Labs also offers managed access to Anthropic and to open weight models. You'll see two kinds of models across these pages: * **Frontier models** are the flagship models hosted by providers like OpenAI and Anthropic. They're the most capable and the most expensive per token. * **Open weight models**, such as DeepSeek and Kimi, are openly published models available through dbt Labs. They offer a strong balance of capability and cost compared to frontier models. Refer to the [Model Provider Rate Table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) and [Supported AI providers](#supported-ai-providers) for the full list of what's available where. #### Bring your own key (BYOK) With BYOK, you supply your own credentials for a provider — OpenAI, Anthropic, Azure AI Foundry, AWS Bedrock, Google Gemini, Snowflake Cortex, or Databricks. To set up BYOK, refer to [Configure BYOK for dbt platform](https://docs.getdbt.com/docs/platform/wizard-byok-platform.md) or [Configure BYOK for the CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md). #### Supported AI providers dbt Wizard supports [managed models](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/overview.md#dbt-managed-providers) (billed by dbt Labs, no key to manage) and [bring-your-own-key (BYOK)](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) models (billed directly by your provider). Here are the following AI providers supported depending on where you work. Refer to [Model Provider Rate table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) for the full list of available models. ##### dbt platform | Provider | Access | | ---------------------------------------------------------------------------- | ------------------- | | [OpenAI](https://openai.com/policies/row-terms-of-use/) (default) | dbt managed or BYOK | | [Anthropic](https://www.anthropic.com/legal/consumer-terms)† | dbt managed or BYOK | | Open weight models (like DeepSeek, Kimi, and so on). | dbt managed | | [Azure AI Foundry](https://www.microsoft.com/licensing/terms) / Azure OpenAI | BYOK | ##### Locally (CLI) | Provider | Access | | ---------------------------------------------------------------------------- | ------------------- | | [OpenAI](https://openai.com/policies/row-terms-of-use/) | dbt managed or BYOK | | [Anthropic](https://www.anthropic.com/legal/consumer-terms)† | dbt managed or BYOK | | Open weight models (like DeepSeek, Kimi, and so on). | dbt managed | | [Azure AI Foundry](https://www.microsoft.com/licensing/terms) / Azure OpenAI | BYOK | | [AWS Bedrock](https://aws.amazon.com/service-terms/) | BYOK | | [Google Gemini](https://ai.google.dev/gemini-api/terms) | BYOK | | [Snowflake Cortex](https://www.snowflake.com/en/legal/terms-of-service/) | BYOK | | [Databricks Unity AI Gateway](https://www.databricks.com/legal/mcsa) | BYOK | You can also connect a personal OpenAI ChatGPT subscription instead of a key. †Anthropic enterprise and subscription licenses (such as Claude Enterprise) aren't supported per Anthropic's [terms of service](https://www.anthropic.com/legal/consumer-terms). #### Choose a model With dbt managed inference, you can switch between the supported managed models at any using the model picker dropdown next to the **Agent mode** control (where you choose **Ask for approval** or **Edit files automatically**). * In [Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) and the [dbt Wizard home tab](https://docs.getdbt.com/docs/platform/wizard-home.md), open the model picker dropdown next to the **Agent mode** control in the dbt Wizard panel, then select a model. * The picker lists the managed models available to you. Refer to the [Model Provider Rate table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) for the available models and their token rates. * If you [bring your own key (BYOK)](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md), dbt Wizard uses the provider and model you configured with your key rather than the managed model picker. [![The model picker dropdown next to the Agent mode control in the Wizard panel.](/img/docs/dbt-platform/wizard-model-picker.png?v=2 "The model picker dropdown next to the Agent mode control in the Wizard panel.")](#)The model picker dropdown next to the Agent mode control in the Wizard panel. #### Related docs * [Trial and billing](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/trial-and-billing.md) for trials, spend limits, and paid access * [Billing](https://docs.getdbt.com/docs/platform/billing.md) for general dbt platform billing * [Model Provider Rate Table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) for per-model token rates * [Service Consumption Table](https://www.getdbt.com/legal/service-consumption-table) --- ### Monitor dbt State activity Preview ### Monitor dbt State activity [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Login required | Usage-based Learn how to monitor dbt State activity in dbt platform for better visibility into model builds and cost savings. dbt State monitoring helps you: * **Track effectiveness of dbt State** — See how dbt State reduces unnecessary model rebuilds by only building models when there are changes to the data or code. dbt State provides transparency into how the optimization works across your projects. * **Analyze build patterns** — Gain insights into your project's build frequency and identify opportunities for further optimization. #### dbt State metrics When you go to **Account settings** > **Billing & Usage** > **Usage-based features**, the **State** tab shows how many days remain in your trial period. Once dbt State is enabled, it displays the following for the current month: * **Models reused this month**: How many model builds dbt State skipped or cloned instead of rebuilding from scratch. * **Total % build reduction**: The overall reduction in model builds across your account. * **Total query run time reduction**: The total time dbt State saved by not executing unnecessary model builds. The **State** tab also displays the following charts: * **DATT** — Shows the target tables processed by dbt State, split into **Billable** and **Free**. Daily active target tables (DATTs) are the [billable units](https://docs.getdbt.com/docs/platform/billing/dbt-state-usage.md#daily-active-target-tables) for dbt State. During a trial, all DATTs are counted as free. * **Asset builds** — Shows all model builds for the month, including models reused and cloned. #### Lag tolerance recommendations The **dbt State** page, which you can access from the left-side menu of the dbt platform, includes a **Lag tolerance recommendations** section that identifies models that could safely tolerate more lag, letting dbt State skip more runs and save additional compute. The recommendations table displays the following columns: | Column | Description | | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Model name** | The name of the model that could benefit from a higher `lag_tolerance` value. | | **Project** | The dbt project the model belongs to, as set by `name:` in `dbt_project.yml`. This may differ from the project name in the dbt platform. | | **Current lag** | The model's current `lag_tolerance` setting. | | **Recommended lag** | The `lag_tolerance` value dbt State recommends based on your model's 30-day build history. | | **% time saved** | The estimated percentage of build time you'd save by applying the recommended `lag_tolerance`. | | **Projected 30d time savings** | The estimated build time you could save over the next 30 days by applying the recommended `lag_tolerance`, based on redundant builds in the previous 30 days. This estimate includes only this model, so actual savings may be higher if downstream models also do not rebuild. | You can search for a specific model using the search bar, or filter recommendations by project using the **Project** dropdown menu. To apply a recommendation, update the model's `lag_tolerance` config. For configuration syntax and examples, refer to the [`lag_tolerance` config page](https://docs.getdbt.com/reference/resource-configs/lag-tolerance.md). ##### How dbt State calculates recommendations dbt State analyzes each model’s build history from the previous 30 days. Models with fewer than 10 recorded builds are excluded because there isn't enough history to make a reliable recommendation. For each eligible model, dbt State: 1. Identifies builds where the model’s definition and inputs had not changed since the previous build. 2. Estimates the build time that different `lag_tolerance` values would have saved. 3. Recommends the smallest value that would have saved more than 30 minutes, helping reduce redundant builds while keeping data as fresh as possible. A model doesn’t appear in the table if it’s a view or its current `lag_tolerance` value is already equal to or greater than the recommended value. Each account displays 20 models with the highest projected savings. [![Lag tolerance recommendations](/img/docs/dbt-platform/using-dbt-platform/lag-tolerance-recommendations.png?v=2 "Lag tolerance recommendations")](#)Lag tolerance recommendations #### Models built and reused chart When you go to your **Account home**, you'll see a chart showing the number of models built and reused, giving you visibility into how dbt State is optimizing your data builds. You can also view the number of reused models per project on **Account home**. #### Logs view of built models When you run a job, or when you run `dbt run` or `dbt build` locally, a structured logs view shows which models were built, skipped, or reused. [![Logs view of built models](/img/docs/dbt-platform/using-dbt-platform/sao-logs-view.png?v=2 "Logs view of built models")](#)Logs view of built models 1. Each model has an icon indicating its status. 2. The **Reused** tab indicates the total number of reused models. 3. You can use the search bar or filter the logs to show **All**, **Success**, **Warning**, **Failed**, **Running**, **Skipped**, **Reused**, or **Debugged** messages. 4. Detailed log messages provide context on why models were built, reused, or skipped. These messages are highlighted in the logs. #### Reused tag in the Latest status lens Lineage lenses are interactive visual filters in [dbt Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md#lenses) that show additional context on your lineage graph to understand how resources are defined or performing. When you apply a lens, tags become visible on the nodes in the lineage graph, indicating the layer value along with coloration based on that value. If you're significantly zoomed out, only the tags and their colors are visible in the graph. The **Latest status** lens shows the status from the latest execution of the resource in the current environment. When you use this lens to view your lineage, dbt State tags reused models with **Reused**. [![Latest status lens showing reused models](/img/docs/dbt-platform/using-dbt-platform/sao-latest-status-lens.png?v=2 "Latest status lens showing reused models")](#)Latest status lens showing reused models To view your lineage with the **Latest status** lens: 1. From the main menu, go to **Orchestration** > **Runs**. 2. Select your run. 3. Go to the **Lineage** tab. You'll see your project's lineage. 4. In the **Lenses** field, select **Latest status**. #### Explain tab To see why dbt State rebuilt, reused, or cloned a specific resource, go to **Orchestration** > **Runs**. Select a run and go to the **Explain** tab. The **Explain** tab appears on job runs that used [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md). It shows why dbt State rebuilt, reused, or cloned each resource, so you can investigate unexpected behavior or verify that State is working as expected. The tab is available while a run is in progress and updates as resources finish. The tab displays an **Explain results** table with one row per resource. You can search by resource name and download the full results as a text file. Expand a row to see the full decision details. Not all analyses apply to every resource type: | Field | Description | | --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | **Resource name** | The name of the resource. | | **Resource type** | The resource type: model, seed, snapshot, test, and so on. | | **Decision** | The reason dbt State rebuilt, reused, or cloned this resource. | | **Run step** | The job command that ran this resource (for example, `dbt build --exclude tag:ml_pipeline`). | | **Table analysis** | Whether the target table already exists in the schema. | | **Query analysis** | Whether the resource query or its upstream queries have changed. | | **Data freshness analysis** | Whether upstream data is fresh or within the configured [`lag_tolerance`](https://docs.getdbt.com/reference/resource-configs/lag-tolerance.md). | [![Explain tab showing the decision breakdown](/img/docs/dbt-platform/deployment/explain-tab.png?v=2 "Explain tab showing the decision breakdown")](#)Explain tab showing the decision breakdown #### Related docs * [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) * [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) * [dbt State trial and billing](https://docs.getdbt.com/docs/deploy/dbt-state-trial.md) * [dbt State configs](https://docs.getdbt.com/reference/resource-configs/dbt-state-configs.md) * [`lag_tolerance` config reference](https://docs.getdbt.com/reference/resource-configs/lag-tolerance.md) * [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md) --- ### Monitor jobs and alerts dbt platform Monitor your dbt jobs to help identify improvement and set up alerts to proactively alert the right people or team. This portion of our documentation will go over dbt's various capabilities that help you monitor your jobs and set up alerts to ensure seamless orchestration, including: * [Visualize and orchestrate downstream exposures](https://docs.getdbt.com/docs/deploy/orchestrate-exposures.md) [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") — Automatically visualize and orchestrate exposures from dashboards and proactively refresh the underlying data sources during scheduled dbt jobs. * [Leverage artifacts](https://docs.getdbt.com/docs/deploy/artifacts.md) — dbt generates and saves artifacts for your project, which it uses to power features like creating docs for your project and reporting freshness of your sources. * [Job notifications](https://docs.getdbt.com/docs/deploy/job-notifications.md) — Receive email, Slack, or Microsoft Teams notifications when a job run succeeds, encounters warnings, fails, or is canceled. * [Model notifications](https://docs.getdbt.com/docs/deploy/model-notifications.md) — Receive email notifications about any issues encountered by your models and tests as soon as they occur while running a job. * [Retry jobs](https://docs.getdbt.com/docs/deploy/retry-jobs.md) — Rerun your errored jobs from start or the failure point. * [Run visibility](https://docs.getdbt.com/docs/deploy/run-visibility.md) — View your run history to help identify where improvements can be made to scheduled jobs. * [Source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) — Monitor data governance by enabling snapshots to capture the freshness of your data sources. * [Webhooks](https://docs.getdbt.com/docs/deploy/webhooks.md) — Use webhooks to send events about your dbt jobs' statuses to other systems. To set up and add data health tiles to view data freshness and quality checks in your dashboard, refer to [data health tiles](https://docs.getdbt.com/docs/explore/data-tile.md). [![An overview of a dbt job run which contains run summary, job trigger, run duration, and more.](/img/docs/dbt-platform/deployment/deploy-scheduler.png?v=2 "An overview of a dbt job run which contains run summary, job trigger, run duration, and more.")](#)An overview of a dbt job run which contains run summary, job trigger, run duration, and more. [![Run history dashboard allows you to monitor the health of your dbt project and displays jobs, job status, environment, timing, and more.](/img/docs/dbt-platform/deployment/run-history.png?v=2 "Run history dashboard allows you to monitor the health of your dbt project and displays jobs, job status, environment, timing, and more.")](#)Run history dashboard allows you to monitor the health of your dbt project and displays jobs, job status, environment, timing, and more. [![Access logs for run steps](/img/docs/dbt-platform/deployment/access-logs.gif?v=2 "Access logs for run steps")](#)Access logs for run steps --- ### Navigate the dbt Insights interface dbt platform | Enterprise, Enterprise+ Learn how to navigate Insights interface and use the main components. Insights provides an interactive interface for writing, running, and analyzing SQL queries. This section highlights the main components of Insights. #### Query console The query console is the main component of Insights. It allows you to write, run, and analyze SQL queries. The Query console supports: 1. Query console editor, which allows you to write, run, and analyze SQL queries: * It supports syntax highlighting and autocomplete suggestions * Hyperlink from SQL code `ref` to the corresponding Explorer page 2. [Query console menu](#query-console-menu), which contains **Bookmark (icon)**, **Develop**, and **Run** buttons. 3. [Query output panel](#query-output-panel), below the query editor and displays the results of a query: * Has three tabs: **Data**, **Chart**, and **Details**, which allow you to analyze query execution and visualize results. 4. [Query console sidebar menu](#query-console-sidebar-menu), which contains the **Catalog**, **Bookmark**, **Query history**, and **dbt Copilot** icons. [![dbt Insights main interface with blank query editor](/img/docs/dbt-insights/insights-main.png?v=2 "dbt Insights main interface with blank query editor")](#)dbt Insights main interface with blank query editor ##### Query console menu The Query console menu is located at the top right of the Query editor. It contains the **Bookmark**, **Develop**, and **Run** buttons: * **Bookmark** button — Save your frequently used SQL queries as favorites for easier access. * When you click **Bookmark**, a **Bookmark Query Details** modal (pop up box) will appear where you can add a **Title** and **Description**. * Let dbt Copilot do the writing for you — use the AI assistant to automatically generate a helpful description for your bookmark. * Access the newly created bookmark from the **Bookmark** icon in the [Query console sidebar menu](#query-console-sidebar-menu). * **Develop**: Open the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) or [Canvas](https://docs.getdbt.com/docs/platform/canvas.md) to continue editing your SQL query. * **Run** button — Run your SQL query and view the results in the **Data** tab. #### Semantic Layer querying The Semantic Layer querying in dbt Insights lets you build queries against the Semantic Layer without writing SQL code. It guides you in creating queries based on available metrics, dimensions, and entities. With this feature, you can: * Build analyses from your predefined semantic layer metrics. * Have filters, time ranges, and aggregates tailored to the semantic model. * View the underlying SQL code for each metric query. To build a query in dbt Insights: 1. From the main menu, go to **Insights**. 2. Click **Build a query**. 3. From the Environment dropdown, select **PROD** or **STG**. Even thought you see **DEV** as an option, queries against development environments are deferred based on deferral rules, so selecting it will not query your development environment directly. 4. Select what you want to include in your query. * Click **Add Metric** to select the metrics for your query. * Click **Add Group by** to choose the dimensions that break down your metric, such as time grain (day, week, month), region, product, or customer. * Click **Add Filter** to create a filter to narrow your results. * Click **Add Order by** to select how you want to sort the results of your query. * Click **Add Limit**, select the amount of results you want to see when you run your query. If left blank, you will get all results. 5. Click **Run** to run your query. Results are available in the **Data** tab. You can see the SQL code generated in the **Details** tab. [![Semantic Layer querying within dbt Insights](/img/docs/dbt-insights/insights-query-builder-interface.png?v=2 "Semantic Layer querying within dbt Insights")](#)Semantic Layer querying within dbt Insights [![Results are displayed in the Data tab](/img/docs/dbt-insights/insights-query-builder.png?v=2 "Results are displayed in the Data tab")](#)Results are displayed in the Data tab [![The generated SQL code in the Details tab](/img/docs/dbt-insights/insights-query-builder-sql.png?v=2 "The generated SQL code in the Details tab")](#)The generated SQL code in the Details tab #### Query output panel The Query output panel is below the query editor and displays the results of a query. It displays the following tabs to analyze query execution and visualize results: * **Data** tab — Preview your SQL results, with results paginated. * **Details** tab — Generates succinct details of executed SQL query: * Query metadata — dbt Copilot's AI-generated title and description. Along with the supplied SQL and compiled SQL. * Connection details — Relevant data platform connection information. * Query details — Query duration, status, column count, row count. * **Chart** tab — Visualizes query results with built-in charts. * Use the chart icon to select the type of chart you want to visualize your results. Available chart types are **line chart, bar chart, or scatterplot**. * Use the **Chart settings** to customize the chart type and the columns you want to visualize. * Available chart types are **line chart, bar chart, or scatterplot**. * **Download** button — Allows you to export the results to CSV [![dbt Insights Data tab](/img/docs/dbt-insights/insights-chart-tab.png?v=2 "dbt Insights Data tab")](#)dbt Insights Data tab [![dbt Insights Chart tab](/img/docs/dbt-insights/insights-chart.png?v=2 "dbt Insights Chart tab")](#)dbt Insights Chart tab [![dbt Insights Details tab](/img/docs/dbt-insights/insights-details.png?v=2 "dbt Insights Details tab")](#)dbt Insights Details tab #### Query console sidebar menu The Query console sidebar menu and icons contains the following options: ##### dbt Catalog **Catalog icon** — View your project's models, columns, metrics, and more using the integrated Catalog view. [![dbt Insights dbt Catalog icon](/img/docs/dbt-insights/insights-explorer.png?v=2 "dbt Insights dbt Catalog icon")](#)dbt Insights dbt Catalog icon ##### Bookmark Save and access your frequently used queries from the query console sidebar. To save and manage bookmarks: 1. Click **Bookmark** in the [Query console menu](#query-console-menu) to save the current query. In the **Bookmark Query Details** modal, add a **Title** and **Description**. 2. Select the **Bookmark** icon in the sidebar to open your list of saved queries. 3. Select a saved Insight to view its details or open it in the query editor. For sharing saved Insights and version history, check out [Save your Insights](https://docs.getdbt.com/docs/explore/access-dbt-insights.md#save-your-insights). [![Bookmark Query Details modal](/img/docs/dbt-insights/save-insights-modal.png?v=2 "Bookmark Query Details modal")](#)Bookmark Query Details modal [![Manage your query bookmarks](/img/docs/dbt-insights/manage-bookmarks.png?v=2 "Manage your query bookmarks")](#)Manage your query bookmarks [![List view of saved Insights](/img/docs/dbt-insights/view-saved-insights.png?v=2 "List view of saved Insights")](#)List view of saved Insights ##### Query history View past queries, their statuses (All, Success, Error, or Pending), start time, and duration. Search for past queries and filter by status. You can also re-run a query from the Query history. [![dbt Insights Query history icon](/img/docs/dbt-insights/insights-query-history.png?v=2 "dbt Insights Query history icon")](#)dbt Insights Query history icon ##### dbt Copilot dbt Wizard is the recommended agent for dbt work dbt Wizard is the recommended AI agent for governed data development in dbt. It handles the full development lifecycle — investigation, building, validation, and shipping — grounded in your dbt project's lineage, tests, contracts, and metric definitions. Refer to [dbt AI FAQs](https://docs.getdbt.com/docs/dbt-ai/dbt-ai-faqs.md#is-dbt-wizard-the-same-as-dbt-copilot), [Billing](https://docs.getdbt.com/docs/dbt-ai/wizard-billing-faqs.md), and [dbt's Terms of Use](https://www.getdbt.com/terms-of-use) for more information. Use dbt Copilot to modify or generate queries using natural language prompts or to chat with the Analyst agent to gather insights about your data. Use dbt Copilot in Insights in two ways: * **Generate SQL**: Build queries in Insights with natural language prompts to explore and query data with an intuitive, context-rich interface. For more information, see [Build queries in dbt Insights](https://docs.getdbt.com/docs/explore/navigate-dbt-insights.md). * **dbt Copilot Analyst agent** [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles"): Ask questions to get intelligent data analysis with automated workflows, governed insights, and actionable recommendations. This is a conversational AI feature where you can ask natural language prompts and receive analysis in real-time. Enable beta features under **Account settings** > **Personal profile** > **Experimental features** to use the Analyst agent. For steps on how to enable, see [Preview new dbt platform features](https://docs.getdbt.com/docs/dbt-versions/experimental-features.md). Some sample questions you can ask the agent: * *What region are my sales growing the fastest?* * *What was the revenue last month?* * *How should I optimize my marketing spend next quarter?* * *How many customers do I have, broken down by customer type?* dbt Copilot Analyst agent creates an analysis plan based on your question. dbt Copilot Analyst agent: 1. Gets context using your semantic models and metrics. 2. Generates SQL queries using your project's definitions. 3. Executes the SQL query and returns results with context. 4. Reviews and summarizes the generated insights and provides a comprehensive answer. The agent can loop through these steps multiple times if it hasn't reached a complete answer, allowing for complex, multi-step analysis.⁠ For more information, see [dbt Copilot in Insights](https://docs.getdbt.com/docs/dbt-ai/analyst-agent.md). [![dbt Copilot in Insights](/img/docs/dbt-insights/insights-copilot-tabs.png?v=2 "dbt Copilot in Insights")](#)dbt Copilot in Insights #### LSP features The following Language Server Protocol (LSP) features are available for projects upgraded to dbt v2: * **Live CTE previews:** Preview a CTE’s output for faster validation and debugging. [![Preview CTE in Insights](/img/docs/dbt-insights/preview-cte.png?v=2 "Preview CTE in Insights")](#)Preview CTE in Insights * **Real-time error detection:** Automatically validate your SQL code to detect errors and surface warnings, without hitting the warehouse. This includes both dbt errors (like invalid `ref`) and SQL errors (like invalid column name or SQL syntax). [![Live error detection](/img/docs/dbt-insights/sql-validation.png?v=2 "Live error detection")](#)Live error detection * **`ref` suggestions:** Autocomplete model names when using the `ref()` function to reference other models in your project. [![ref suggestions in Insights](/img/docs/dbt-insights/ref-autocomplete.png?v=2 "ref suggestions in Insights")](#)ref suggestions in Insights * **Hover insights:** View context on tables, columns, and functions without leaving your code. Hover over any SQL element to see details like column names and data types. [![Sample column details](/img/docs/dbt-insights/column-info.png?v=2 "Sample column details")](#)Sample column details [![Sample column details](/img/docs/dbt-insights/column-hover.png?v=2 "Sample column details")](#)Sample column details --- ### Navigating the state-aware interface dbt platform Learn how to navigate the state-aware orchestration interface for better visibility into model builds and cost tracking. State-aware orchestration is now dbt State [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) works with all engines and environments: dbt v1, dbt platform, and dbt v2 If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). #### Models built and reused chart When you go to your **Account home**, you'll see a chart showing the number of models built and reused, giving you visibility into how state-aware orchestration is optimizing your data builds. This chart helps you to: * **Track effectiveness of state-aware orchestration** — See how state-aware orchestration reduces unnecessary model rebuilds by only building models when there are changes to the data or code⁠. This chart provides transparency into how the optimization is working across your dbt implementation. * **Analyze build patterns** — Gain insights into your project's build frequency and identify opportunities for further optimization. You can also view the number of reused models per project in the **Accounts home** page. [![Models built and reused chart in Account home](/img/docs/dbt-platform/using-dbt-platform/account-home-chart.png?v=2 "Models built and reused chart in Account home")](#)Models built and reused chart in Account home [![View reused models count per project in the Accounts home page](/img/docs/deploy/sao-model-reuse.png?v=2 "View reused models count per project in the Accounts home page")](#)View reused models count per project in the Accounts home page #### Model consumption view in jobs State-aware jobs provide charts that show information about your job runs, and how many models were built and reused by your job in the past week, in the last 14 days, or in the last 30 days. In the **Overview** section of your job, the following charts are available: Under the **Runs** tab: * **Recent runs** * **Total run duration time** [![Charts for Recent runs and Total run duration time](/img/docs/dbt-platform/using-dbt-platform/sao-runs-chart.png?v=2 "Charts for Recent runs and Total run duration time")](#)Charts for Recent runs and Total run duration time Under the **Models** tab: * **Models built** * **Models reused** [![Charts for Models built and Models reused](/img/docs/dbt-platform/using-dbt-platform/sao-models-chart.png?v=2 "Charts for Models built and Models reused")](#)Charts for Models built and Models reused #### Logs view of built models When running a job, a structured logs view shows which models were built, skipped, or reused. [![Logs view of built models](/img/docs/dbt-platform/using-dbt-platform/sao-logs-view.png?v=2 "Logs view of built models")](#)Logs view of built models 1. Each model has an icon indicating its status. 2. The **Reused** tab indicates the total number of reused models. 3. You can use the search bar or filter the logs to show **All**, **Success**, **Warning**, **Failed**, **Running**, **Skipped**, **Reused**, or **Debugged** messages. 4. Detailed log messages are provided to get more context on why models were built, reused, or skipped. These messages are highlighted in the logs. #### Reused tag in the Latest status lens Lineage lenses are interactive visual filters in [dbt Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md#lenses) that show additional context on your lineage graph to understand how resources are defined or performing. When you apply a lens, tags become visible on the nodes in the lineage graph, indicating the layer value along with coloration based on that value. If you're significantly zoomed out, only the tags and their colors are visible in the graph. The **Latest status** lens shows the status from the latest execution of the resource in the current environment. When you use this lens to view your lineage, models that were reused from state-aware orchestration are tagged with **Reused**. [![Latest status lens showing reused models](/img/docs/dbt-platform/using-dbt-platform/sao-latest-status-lens.png?v=2 "Latest status lens showing reused models")](#)Latest status lens showing reused models To view your lineage with the **Latest status** lens: 1. From the main menu, go to **Orchestration** > **Runs**. 2. Select your run. 3. Go to the **Lineage** tab. The lineage of your project appears. 4. In the **Lenses** field, select **Latest status**. #### Clear cache button State-aware orchestration uses a cached hash of both code and data state for each model in an environment stored in Redis. When running a job, dbt checks if there are changes in the hash for the model being built between the saved state in Redis and the current state that would be built by the job. If there is a change, dbt builds the model. If there are no changes, dbt reuses the model from the last time it was built. * To wipe this state clean and start again, clear the cache by going to **Orchestration** > **Environments**. Select your environment and click the **Clear cache** button. * The **Clear cache** button is only available if you have enabled state-aware orchestration. * After clearing the cache, the next run rebuilds every model from scratch. Subsequent runs rely on the regenerated cache. [![Clear cache button](/img/docs/dbt-platform/using-dbt-platform/sao-clear-cache.png?v=2 "Clear cache button")](#)Clear cache button #### Related docs * [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) * [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) * [dbt State configs](https://docs.getdbt.com/reference/resource-configs/dbt-state-configs.md) * [Migrate to dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md) --- ### Optimize your dbt builds By default, dbt rebuilds every selected node on every run — even if nothing has changed. This makes it easy to get started, but adds up for larger projects: longer job times, higher warehouse costs, and slower feedback during development. dbt has features designed to help you skip unnecessary work and reuse existing results instead. #### dbt State [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) is a service that makes dbt smarter about what to build. It integrates into any dbt deployment — including self-hosted deployments using dbt v2 or dbt v1 — without requiring a recurring dbt platform subscription. Instead of rebuilding every node on every run, it compares each node's logic and upstream data against the previous run and picks the most efficient path: * **Skip** — If the node's logic and upstream data haven't changed, dbt reuses the existing object in your target schema as-is. * **Clone** — If the data is fresh but exists in a different schema (for example, production), dbt clones it rather than rebuilding from scratch. * **Build** — If reuse isn't possible, dbt rebuilds normally and automatically defers unselected upstream nodes to production — no `--defer` or `--state` flags required. To enable dbt State: * **dbt v1.7–1.12**: ```bash cd path/to/your/project pip install dbt-state ``` * **dbt v2**: ```bash cd path/to/your/project dbt login ``` Authentication requires a dbt platform account with a [30-day free trial](https://docs.getdbt.com/docs/deploy/dbt-state-trial.md). dbt State pricing is usage-based — you're billed per target table that dbt State reuses each day, not per dbt platform seat. For full setup instructions, refer to [Setting up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md). #### Deferral [Deferral](https://docs.getdbt.com/reference/node-selection/defer.md) lets you build a subset of your project without building all upstream dependencies first. Instead of running everything upstream, dbt points unbuilt references at existing objects in another environment — typically production. This is useful in development and CI environments, where you want to test one or two models without waiting for a full pipeline run. ```bash dbt build --select my_model --defer --state path/to/prod/artifacts ``` #### dbt clone [`dbt clone`](https://docs.getdbt.com/reference/commands/clone.md) creates copies of selected nodes in a target schema. On warehouses that support zero-copy cloning (for example, Snowflake), it creates lightweight database clones without duplicating the underlying data. On other warehouses, it creates views pointing at the upstream relations. This is useful in development when you want to quickly populate a dev environment with production-like objects without running the full pipeline. ```bash dbt clone --select my_model ``` #### dbt's selection syntax dbt has a [variety of selectors](https://docs.getdbt.com/reference/node-selection/syntax.md) you can use to target specific parts of your project instead of building everything every time. For example, `+my_model` selects `my_model` and all of its upstream dependencies, while `my_model+2` selects `my_model` and two levels of downstream dependents. This lets you test your changes in isolation without running your entire project. ```bash dbt build --select +my_model ``` --- ### Orchestrate downstream exposures Beta ### Orchestrate downstream exposures [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt platform | Enterprise, Enterprise+ Use [dbt platform job scheduler](https://docs.getdbt.com/docs/deploy/job-scheduler.md) to proactively refresh downstream exposures and the underlying data sources (extracts) that power your Tableau Workbooks. (Applies to dbt v2.0 and later) dbt v2 not supported for this beta Orchestrating downstream exposures *isn’t* available when your deployments use [**v2 Stable**](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). Supported jobs must use [**v1 Latest**](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) with the dbt v1 engine. On the [**v2 Stable**](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) release track, orchestrating downstream exposures isn’t supported yet. Setting `DBT_ACTIVE_EXPOSURES` and `DBT_ACTIVE_EXPOSURES_BUILD_AFTER` won’t enable orchestration behavior (like Tableau extract refreshes) or the related job log entries. The private beta is for dbt Enterprise accounts. Contact your account representative for access. Orchestrating exposures integrates with [downstream exposures](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md) and uses your `dbt build` job to ensure that Tableau extracts are updated regularly. Control the frequency of these refreshes by configuring environment variables in your dbt environment.  Differences between visualizing and orchestrating downstream exposures The following table summarizes the differences between visualizing and orchestrating downstream exposures: | Info | Set up and visualize downstream exposures | Orchestrate downstream exposures [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") | | ----------------- | ------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Purpose | Automatically brings downstream assets into your dbt lineage. | Proactively refreshes the underlying data sources during scheduled dbt jobs. | | Benefits | Provides visibility into data flow and dependencies. | Ensures BI tools always have up-to-date data without manual intervention. | | Location | Exposed in [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) | Exposed in [dbt scheduler](https://docs.getdbt.com/docs/deploy/deployments.md) | | Supported BI tool | Tableau | Tableau | | Use case | Helps users understand how models are used and reduces incidents. | Optimizes timeliness and reduces costs by running models when needed. | #### Prerequisites To orchestrate downstream exposures, you should meet the following: * [Configured downstream exposures](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md) and ensured desired exposures are included in your lineage. * Verified your environment and jobs are on a supported dbt [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). * Have a dbt account on the [Enterprise or Enterprise+ plan](https://www.getdbt.com/pricing/). * Created a [production](https://docs.getdbt.com/docs/deploy/deploy-environments.md#set-as-production-environment) deployment environment for each project you want to explore, with at least one successful job run. * Have [admin permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) in dbt to edit project settings or production environment settings. * Configured a [Tableau personal access token (PAT)](https://help.tableau.com/current/server/en-us/security_personal_access_tokens.htm) whose creator has privileges to view and refresh the data sources used by your exposures. The PAT inherits the permissions of its creator. Use a PAT created by: * A Tableau server or site administrator * A data source owner or a project leader #### Orchestrate downstream exposures To orchestrate downstream exposures and see refreshes happen automatically during scheduled jobs on [Latest](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) with the dbt v1 engine: 1. In the dbt platform, click **Deploy**, then **Environments**, and select the **Environment variables** tab. 2. Click **Add variable** and set the [environment level variable](https://docs.getdbt.com/docs/build/environment-variables.md#setting-and-overriding-environment-variables) `DBT_ACTIVE_EXPOSURES` to `1` within the environment you want the refresh to happen. 3. Then set the `DBT_ACTIVE_EXPOSURES_BUILD_AFTER` to control the maximum refresh frequency (in minutes) you want between each exposure refresh. 4. Set the variable to **1440** minutes (24 hours) by default. This means that downstream exposures won’t refresh Tableau extracts more often than this set interval, even if the related models run more frequently. [![Set the environment variable \`DBT\_ACTIVE\_EXPOSURES\` to \`1\`.](/img/docs/platform-integrations/auto-exposures/active-exposures-env-var.jpg?v=2 "Set the environment variable `DBT_ACTIVE_EXPOSURES` to `1`.")](#)Set the environment variable \`DBT\_ACTIVE\_EXPOSURES\` to \`1\`. 5. Run a production job on [Latest](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) with dbt v1. Each run can trigger a downstream exposure refresh; if a job runs before the configured interval has passed, dbt skips the downstream exposure refresh and marks it as `skipped` in the job logs. 6. View downstream exposure entries in your run job logs. [![View the downstream exposure logs in the dbt run job logs.](/img/docs/platform-integrations/auto-exposures/active-exposure-log.jpg?v=2 "View the downstream exposure logs in the dbt run job logs.")](#)View the downstream exposure logs in the dbt run job logs. * View more details in the debug logs for any troubleshooting. --- ### Organize your outputs [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/build/custom-schemas.md) ###### [Custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md) [Learn how to use the `schema` configuration key to specify a custom schema.](https://docs.getdbt.com/docs/build/custom-schemas.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/build/custom-databases.md) ###### [Custom databases](https://docs.getdbt.com/docs/build/custom-databases.md) [Learn how to use the `database` configuration key to specify a custom database.](https://docs.getdbt.com/docs/build/custom-databases.md) <br /> [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/build/custom-aliases.md) ###### [Custom aliases](https://docs.getdbt.com/docs/build/custom-aliases.md) [Learn how to use the `alias` model configuration to change the name of a model's identifier in the database.](https://docs.getdbt.com/docs/build/custom-aliases.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/build/custom-target-names.md) ###### [Custom target names](https://docs.getdbt.com/docs/build/custom-target-names.md) [Learn how to define a custom target name for a dbt job.](https://docs.getdbt.com/docs/build/custom-target-names.md) --- ### Packages Software engineers frequently modularize code into libraries. These libraries help programmers operate with leverage: they can spend more time focusing on their unique business logic, and less time implementing code that someone else has already spent the time perfecting. In dbt, libraries like these are called *packages*. dbt's packages are so powerful because so many of the analytic problems we encountered are shared across organizations, for example: * transforming data from a consistently structured SaaS dataset, for example: * turning [Snowplow](https://hub.getdbt.com/dbt-labs/snowplow/latest/) or [Segment](https://hub.getdbt.com/dbt-labs/segment/latest/) pageviews into sessions * transforming [AdWords](https://hub.getdbt.com/dbt-labs/adwords/latest/) or [Facebook Ads](https://hub.getdbt.com/dbt-labs/facebook_ads/latest/) spend data into a consistent format. * writing dbt macros that perform similar functions, for example: * [generating SQL](https://github.com/dbt-labs/dbt-utils#sql-helpers) to union together two relations, pivot columns, or construct a surrogate key * creating [custom schema tests](https://github.com/dbt-labs/dbt-utils#schema-tests) * writing [audit queries](https://hub.getdbt.com/dbt-labs/audit_helper/latest/) * building models and macros for a particular tool used in your data stack, for example: * Models to understand [Redshift](https://hub.getdbt.com/dbt-labs/redshift/latest/) privileges. * Macros to work with data loaded by [Stitch](https://hub.getdbt.com/dbt-labs/stitch_utils/latest/). dbt *packages* are in fact standalone dbt projects, with models, macros, and other resources that tackle a specific problem area. As a dbt user, by adding a package to your project, all of the package's resources will become part of your own project. This means: * Models in the package will be materialized when you `dbt run`. * You can use `ref` in your own models to refer to models from the package. * You can use `source` to refer to sources in the package. * You can use macros in the package in your own project. * It's important to note that defining and installing dbt packages is different from [defining and installing Python packages](https://docs.getdbt.com/docs/build/python-models.md#using-pypi-packages) #### Use cases The following setup will work for every dbt project: * Add [any package dependencies](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#when-to-use-project-dependencies) to `packages.yml` * Add [any project dependencies](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#when-to-use-package-dependencies) to `dependencies.yml` However, you may be able to consolidate both into a single `dependencies.yml` file. Read the following section to learn more. ###### About packages.yml and dependencies.yml The `dependencies.yml`. file can contain both types of dependencies: "package" and "project" dependencies. * [Package dependencies](https://docs.getdbt.com/docs/build/packages.md#how-do-i-add-a-package-to-my-project) lets you add source code from someone else's dbt project into your own, like a library. * Project dependencies provide a different way to build on top of someone else's work in dbt. * Private packages are not supported in `dependencies.yml` because they intentionally don't support Jinja rendering or conditional configuration. This is to maintain static and predictable configuration and ensures compatibility with other services, like dbt. If your dbt project doesn't require the use of Jinja within the package specifications, you can simply rename your existing `packages.yml` to `dependencies.yml`. However, something to note is if your project's package specifications use Jinja, particularly for scenarios like adding an environment variable or a [Git token method](https://docs.getdbt.com/docs/build/packages.md#git-token-method) in a private Git package specification, you should continue using the `packages.yml` file name. Use the following toggles to understand the differences and determine when to use `dependencies.yml` or `packages.yml` (or both). Refer to the [FAQs](#faqs) for more info.  When to use Project dependencies Project dependencies are designed for the [dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md) and [cross-project reference](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#how-to-write-cross-project-ref) workflow: * Use `dependencies.yml` when you need to set up cross-project references between different dbt projects, especially in a dbt Mesh setup. * Use `dependencies.yml` when you want to include both projects and non-private dbt packages in your project's dependencies. * Use `dependencies.yml` for organization and maintainability if you're using both [cross-project refs](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#how-to-write-cross-project-ref) and [dbt Hub packages](https://hub.getdbt.com/). This reduces the need for multiple YAML files to manage dependencies.  When to use Package dependencies Package dependencies allow you to add source code from someone else's dbt project into your own, like a library: * If you only use packages like those from the [dbt Hub](https://hub.getdbt.com/), remain with `packages.yml`. * Use `packages.yml` when you want to download dbt packages, such as dbt projects, into your root or parent dbt project. Something to note is that it doesn't contribute to the dbt Mesh workflow. * Use `packages.yml` to include packages in your project's dependencies. This includes both public packages, such as those from the [dbt Hub](https://hub.getdbt.com/), and private packages. dbt now supports [native private packages](https://docs.getdbt.com/docs/build/packages.md#native-private-packages). * [`packages.yml` supports Jinja rendering](https://docs.getdbt.com/docs/build/dbt-tips.md#yaml-tips) for historical reasons, allowing dynamic configurations. This can be useful if you need to insert values, like a [Git token method](https://docs.getdbt.com/docs/build/packages.md#git-token-method) from an environment variable, into your package specifications. Previously, to use private Git repositories in dbt, you needed to use a workaround that involved embedding a Git token with Jinja. This is not ideal as it requires extra steps like creating a user and sharing a Git token. We’ve introduced support for [native private packages](https://docs.getdbt.com/docs/build/packages.md#native-private-packages-) to address this. #### How do I create a package? Creating packages is an advanced use of dbt, but it can be a relatively simple task. The only strict requirement is the presence of a [`dbt_project.yml` file](https://docs.getdbt.com/reference/dbt_project.yml.md). The most common use-cases for packages are: * Sharing [models](https://docs.getdbt.com/docs/build/models.md) to share across multiple projects. * Sharing [macros](https://docs.getdbt.com/docs/build/jinja-macros.md) to share across multiple projects. Note that packages can be [private](#private-packages) — they don't need to be shared publicly. Private packages can be hosted on your own Git provider (for example, GitHub or GitLab). For instructions on creating dbt packages and additional information, refer to our guide [Building dbt packages](https://docs.getdbt.com/guides/building-packages.md?step=1). #### How do I add a package to my project? 1. Add a file named `dependencies.yml` or `packages.yml` to your dbt project. This should be at the same level as your `dbt_project.yml` file. 2. Specify the package(s) you wish to add using one of the supported syntaxes, for example: ```yaml packages: - package: dbt-labs/snowplow version: 0.7.0 - git: "https://github.com/dbt-labs/dbt-utils.git" revision: 0.9.2 - local: /opt/dbt/redshift ``` The default [`packages-install-path`](https://docs.getdbt.com/reference/project-configs/packages-install-path.md) is `dbt_packages`. 3. Run `dbt deps` to install the package(s). Packages get installed in the `dbt_packages` directory – by default this directory is ignored by git, to avoid duplicating the source code for the package. #### How do I specify a package? You can specify a package using one of the following methods, depending on where your package is stored. ##### Hub packages (recommended) dbt Labs hosts the [Package hub](https://hub.getdbt.com), registry for dbt packages, as a courtesy to the dbt Community, but does not certify or confirm the integrity, operability, effectiveness, or security of any Packages. Please read the [dbt Labs Package Disclaimer](https://hub.getdbt.com/disclaimer/) before installing Hub packages. You can install available hub packages in the following way: packages.yml ```yaml packages: - package: dbt-labs/snowplow version: 0.7.3 # version number ``` Hub packages require a version to be specified – you can find the latest release number on dbt Hub. Since Hub packages use [semantic versioning](https://semver.org/), we recommend pinning your package to the latest patch version from a specific minor release, like so: ```yaml packages: - package: dbt-labs/snowplow version: [">=0.7.0", "<0.8.0"] ``` `dbt deps` "pins" each package by default. See ["Pinning packages"](#pinning-packages) for details. Where possible, we recommend installing packages via dbt Hub, since this allows dbt to handle duplicate dependencies. This is helpful in situations such as: * Your project uses both the dbt-utils and Snowplow packages, and the Snowplow package *also* uses the dbt-utils package. * Your project uses both the Snowplow and Stripe packages, both of which use the dbt-utils package. In comparison, other package installation methods are unable to handle the duplicate dbt-utils package. Advanced users can choose to host an internal version of the package hub based on [this repository](https://github.com/dbt-labs/hub.getdbt.com) and setting the (Applies to dbt v1.11 and later) `DBT_ENGINE_PACKAGE_HUB_URL` environment variable. ###### Prerelease versions Some package maintainers may wish to push prerelease versions of packages to the dbt Hub, in order to test out new functionality or compatibility with a new version of dbt. A prerelease version is demarcated by a suffix, such as `a1` (first alpha), `b2` (second beta), or `rc3` (third release candidate). By default, `dbt deps` will not include prerelease versions when resolving package dependencies. You can enable the installation of prereleases in one of two ways: * Explicitly specifying a prerelease version in your `version` criteria * Setting `install_prerelease` to `true`, and providing a compatible version range For example, both of the following configurations would successfully install `0.4.5-a2` for the [`dbt_artifacts` package](https://hub.getdbt.com/brooklyn-data/dbt_artifacts/latest/): ```yaml packages: - package: brooklyn-data/dbt_artifacts version: 0.4.5-a2 ``` ```yaml packages: - package: brooklyn-data/dbt_artifacts version: [">=0.4.4", "<0.4.6"] install_prerelease: true ``` ##### Git packages Packages stored on a Git server can be installed using the `git` syntax, like so: packages.yml ```yaml packages: - git: "https://github.com/dbt-labs/dbt-utils.git" # git URL revision: 0.9.2 # tag or branch name ``` Add the Git URL for the package, and optionally specify a revision. The revision can be: * a branch name * a tagged release * a specific commit (full 40-character hash) Example of a revision specifying a 40-character hash: ```yaml packages: - git: "https://github.com/dbt-labs/dbt-utils.git" revision: 4e28d6da126e2940d17f697de783a717f2503188 ``` By default, `dbt deps` "pins" each package. See ["Pinning packages"](#pinning-packages) for details. ##### Internally hosted tarball URL Some organizations have security requirements to pull resources only from internal services. To address the need to install packages from hosted environments such as Artifactory or cloud storage buckets, dbt enables you to install packages from internally-hosted tarball URLs. ```yaml packages: - tarball: https://codeload.github.com/dbt-labs/dbt-utils/tar.gz/0.9.6 name: 'dbt_utils' ``` Where `name: 'dbt_utils'` specifies the subfolder of `dbt_packages` that's created for the package source code to be installed within. #### Private packages ##### Native private packages Native private packages let you install packages from [supported](#prerequisites) private Git repos using the `private` key, without having to configure a [token](#git-token-method) or write out a full Git URL. This simplifies setup and reduces credential management. * dbt platform: Uses your existing Git [integration](https://docs.getdbt.com/docs/platform/git/configure-git.md) for authentication. * Locally using dbt v2 or dbt v1.12+: Uses your system's SSH configuration. Requires the [`provider` key](#using-the-provider-key). ###### Prerequisites * **dbt platform**: You must have one of the following Git providers configured in the **Integrations** section of your **Account settings**: * **[GitHub](https://docs.getdbt.com/docs/platform/git/connect-github.md)** * **[Azure DevOps](https://docs.getdbt.com/docs/platform/git/connect-azure-devops.md)** * Use the `org/project/repo` path with the `ado` provider. * **[GitLab](https://docs.getdbt.com/docs/platform/git/connect-gitlab.md)** * Every GitLab repo with private packages must also be a dbt platform project. * **Locally using dbt v2 or dbt v1.12+**: You must have an SSH key configured on your machine for the relevant Git provider and include the [`provider` key](#using-the-provider-key) in your package configuration. ###### Configuration Use the `private` key in your `packages.yml` or `dependencies.yml` to clone package repos using your existing dbt Git integration without having to provision an access token or create a dbt environment variable. packages.yml ```yaml packages: - private: dbt-labs/awesome_repo # your-org/your-repo path provider: "github" # Supported values: "github", "gitlab", "ado" - package: normal packages [...] ``` dependencies.yml ```yaml packages: - private: dbt-labs/awesome_repo # your-org/your-repo path provider: "github" # Supported values: "github", "gitlab", "ado" ``` Azure DevOps considerations and limitations There are some considerations and limitations when using native private packages from Azure DevOps. Open the expandable section to learn more.  Native private packages and Azure DevOps limitations 1. Use the `ado` provider and specify the `org/project/repo` path in the `private` key. packages.yml ```yaml packages: - private: my-org/my-project/my-repo provider: "ado" ``` 2. On dbt platform, native private packages from Azure DevOps can fail when the package is in a different Azure DevOps project than the job that installs it, especially if your account is connected to many Azure DevOps projects: * This happens because dbt platform has a 32 KB limit for the Azure DevOps authentication details it can use at job runtime. If your connected Azure DevOps projects exceed that limit, dbt platform can only use the current project's connection. As a result, packages in other Azure DevOps projects can't be accessed and `dbt deps` fails. * As a workaround, reduce the number of Azure DevOps projects connected to your account, then rerun `dbt deps`. The number of projects you can connect depends on your Azure DevOps organization structure and repo count. This limitation doesn't affect local development with dbt v1 or dbt v2 when cloning private packages over SSH. We're currently working to address this, and if you're running into issues, please contact your dbt Labs account team. You can pin private packages similar to regular dbt packages: ```yaml packages: - private: dbt-labs/awesome_repo revision: "0.9.5" # Pin to a tag, branch, or complete 40-character commit hash ``` ###### Using the `provider` key Add the `provider` key when: * You are using multiple Git integrations or using dbt v2. * You are using dbt v2 locally (with the [dbt v2 CLI](https://docs.getdbt.com/docs/local/install-dbt.md?version=2) or the [VS Code extension](https://docs.getdbt.com/docs/local/install-dbt.md?version=2)) (required). * You are using dbt v1.12 or later for SSH-based cloning (required). ```yaml packages: - private: dbt-labs/awesome_repo provider: "github" # Supported values: "github", "gitlab", "ado" ``` dbt v1 and dbt v2 use the `provider` value to construct the correct SSH URL for cloning, based on the provider: | Provider | SSH URL format | | -------- | ------------------------------------------- | | `github` | `git@github.com:org/repo.git` | | `gitlab` | `git@gitlab.com:org/repo.git` | | `ado` | `git@ssh.dev.azure.com:v3/org/project/repo` | dbt v1 and dbt v2 rely on your system's SSH configuration to authenticate and clone the private repository. If `git clone` works on your system for the private package repo, the private package install should work too. ##### SSH key method (CLI only) note This method uses the `git:` key with a full SSH URL, which is different from [native private packages](#native-private-packages) that use the `private:` key. For most use cases, native private packages is the recommended approach as it simplifies setup. If you're using the Command Line, private packages can be cloned via SSH and an SSH key. When you use SSH keys to authenticate to your git remote server, you don’t need to supply your username and password each time. Read more about SSH keys, how to generate them, and how to add them to your git provider here: [Github](https://docs.github.com/en/github/authenticating-to-github/connecting-to-github-with-ssh) and [GitLab](https://docs.gitlab.com/ee/user/ssh.html). packages.yml ```yaml packages: - git: "git@github.com:dbt-labs/dbt-utils.git" # git SSH URL ``` If you're using the dbt platform, the SSH key method will not work, but you can use [native private packages](#native-private-packages) or the [HTTPS Git Token Method](https://docs.getdbt.com/docs/build/packages.md#git-token-method). ##### Git token method note [Native private packages](#native-private-packages) is the recommended approach for GitHub, GitLab, and Azure DevOps. The git token method is still functional in dbt v1, dbt v2, and the dbt platform, but requires provisioning a personal access token. It remains the supported path for dbt v1 users who need HTTPS-based cloning. This method allows the user to clone via HTTPS by passing in a git token via an environment variable. Be careful of the expiration date of any token you use, as an expired token could cause a scheduled run to fail. Additionally, user tokens can create a challenge if the user ever loses access to a specific repo. dbt usage If you are using dbt, you must adhere to the naming conventions for environment variables. Environment variables in dbt must be prefixed with either `DBT_` or `DBT_ENV_SECRET`. Environment variables keys are uppercased and case sensitive. When referencing `{{env_var('DBT_KEY')}}` in your project's code, the key must match exactly the variable defined in dbt's UI. In GitHub: packages.yml ```yaml packages: # use this format when accessing your repository via a github application token - git: "https://{{env_var('DBT_ENV_SECRET_GIT_CREDENTIAL')}}@github.com/dbt-labs/awesome_repo.git" # git HTTPS URL # use this format when accessing your repository via a classical personal access token - git: "https://{{env_var('DBT_ENV_SECRET_GIT_CREDENTIAL')}}@github.com/dbt-labs/awesome_repo.git" # git HTTPS URL # use this format when accessing your repository via a fine-grained personal access token (username sometimes required) - git: "https://GITHUB_USERNAME:{{env_var('DBT_ENV_SECRET_GIT_CREDENTIAL')}}@github.com/dbt-labs/awesome_repo.git" # git HTTPS URL ``` Read more about creating a GitHub Personal Access token [here](https://docs.github.com/en/enterprise-server@3.1/github/authenticating-to-github/keeping-your-account-and-data-secure/creating-a-personal-access-token). You can also use a GitHub App installation [token](https://docs.github.com/en/rest/reference/apps#create-an-installation-access-token-for-an-app). In GitLab: packages.yml ```yaml packages: - git: "https://{{env_var('DBT_USER_NAME')}}:{{env_var('DBT_ENV_SECRET_DEPLOY_TOKEN')}}@gitlab.example.com/dbt-labs/awesome_project.git" # git HTTPS URL ``` Read more about creating a GitLab Deploy Token [here](https://docs.gitlab.com/ee/user/project/deploy_tokens/#creating-a-deploy-token) and how to properly construct your HTTPS URL [here](https://docs.gitlab.com/ee/user/project/deploy_tokens/#git-clone-a-repository). Deploy tokens can be managed by Maintainers only. In Azure DevOps: packages.yml ```yaml packages: - git: "https://{{env_var('DBT_ENV_SECRET_PERSONAL_ACCESS_TOKEN')}}@dev.azure.com/dbt-labs/awesome_project/_git/awesome_repo" # git HTTPS URL ``` Read more about creating a Personal Access Token [here](https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops\&tabs=preview-page#create-a-pat). In Bitbucket: packages.yml ```yaml packages: - git: "https://{{env_var('DBT_USER_NAME')}}:{{env_var('DBT_ENV_SECRET_PERSONAL_ACCESS_TOKEN')}}@bitbucketserver.com/scm/awesome_project/awesome_repo.git" # for Bitbucket Server ``` Read more about creating a Personal Access Token [here](https://confluence.atlassian.com/bitbucketserver/personal-access-tokens-939515499.html). #### Configure subdirectory for packaged projects In general, dbt expects `dbt_project.yml` to be located as a top-level file in a package. If the packaged project is instead nested in a subdirectory—perhaps within a much larger mono repo—you can optionally specify the folder path as `subdirectory`. dbt will attempt a [sparse checkout](https://git-scm.com/docs/git-sparse-checkout) of just the files located within that subdirectory. Note that you must be using a recent version of `git` (`>=2.26.0`). packages.yml ```yaml packages: - git: "https://github.com/dbt-labs/dbt-labs-experimental-features" # git URL subdirectory: "materialized-views" # name of subdirectory containing `dbt_project.yml` ``` ##### Local packages A "local" package is a dbt project accessible from your local file system. They're best suited for when there is a common collection of models and macros that you want to share across multiple downstream dbt projects (but each downstream project still has its own unique models, macros, etc). You can install local packages by specifying the project's path. It works best when you nest the project within a subdirectory relative to your current project's directory. packages.yml ```yaml packages: - local: relative/path/to/subdirectory ``` Other patterns may work in some cases, but not always. For example, if you install this project as a package elsewhere, or try running it on a different system, the relative and absolute paths will yield the same results. packages.yml ```yaml packages: # not recommended - support for these patterns vary - local: /../../redshift # relative path to a parent directory - local: /opt/dbt/redshift # absolute path on the system ``` There are a few specific use cases where we recommend using a "local" package: 1. **Monorepo** — When you have multiple projects, each nested in a subdirectory, within a monorepo. "Local" packages allow you to combine projects for coordinated development and deployment. 2. **Testing changes** — To test changes in one project or package within the context of a downstream project or package that uses it. By temporarily switching the installation to a "local" package, you can make changes to the former and immediately test them in the latter for quicker iteration. This is similar to [editable installs](https://pip.pypa.io/en/stable/topics/local-project-installs/) in Python. 3. **Nested project** — When you have a nested project that defines fixtures and tests for a project of utility macros, like [the integration tests within the `dbt-utils` package](https://github.com/dbt-labs/dbt-utils/tree/main/integration_tests). #### What packages are available? To see the library of published dbt packages, check out the [dbt package hub](https://hub.getdbt.com)! #### dbt v2 package compatibility To determine if a package is compatible with dbt v2, visit the [dbt package hub](https://hub.getdbt.com/) and look for the dbt v2-compatible badge, or review the package's [`require-dbt-version` configuration](https://docs.getdbt.com/reference/project-configs/require-dbt-version.md#pin-to-a-range). * Packages with a `require-dbt-version` that equals or contains `2.0.0` are compatible with dbt v2. For example, `require-dbt-version: ">=1.10.0,<3.0.0"`. Even if a package doesn't reflect compatibility in the package hub, it may still work with v2. Work with package maintainers to track updates, and [thoroughly test packages](https://docs.getdbt.com/guides/dbt-package-compat?step=5) that aren't clearly compatible before deploying. * Package maintainers who would like to make their package compatible with v2 can refer to the [dbt v2 package upgrade guide](https://docs.getdbt.com/guides/dbt-package-compat.md) for instructions. Fivetran package considerations: * The Fivetran `source` and `transformation` packages have been combined into a single package. * If you manually installed source packages like `fivetran/github_source`, you need to ensure `fivetran/github` is installed and deactivate the transformation models. ###### Package compatibility messages Inconsistent v2 warnings and `dbt-autofix` logs dbt v2 warnings and `dbt-autofix` logs may show different messages about package compatibility. If you use [`dbt-autofix`](https://github.com/dbt-labs/dbt-autofix) while upgrading to v2 in the Studio IDE or dbt VS Code extension, you may see different messages about package compatibility between `dbt-autofix` and v2 warnings. Here's why: * dbt v2 warnings are emitted based on a package's `require-dbt-version` and whether `require-dbt-version` contains `2.0.0`. * Some packages are already v2-compatible even though package maintainers haven't yet updated `require-dbt-version`. * `dbt-autofix` knows about these compatible packages and will not try to upgrade a package that it knows is already compatible. This means that even if you see a v2 warning for a package that `dbt-autofix` identifies as compatible, you don't need to change the package. The message discrepancy is temporary while we implement and roll out `dbt-autofix`'s enhanced compatibility detection to v2 warnings. Here's an example of a v2 warning in the Studio IDE that says a package isn't compatible with v2 but `dbt-autofix` indicates it is compatible: ```text dbt1065: Package 'dbt_utils' requires dbt version [>=1.30,<2.0.0], but current version is 2.0.0-preview.72. This package may not be compatible with your dbt version. dbt(1065) [Ln 1, Col 1] ``` #### Advanced package configuration ##### Updating a package When you update a version or revision in your `packages.yml` file, it isn't automatically updated in your dbt project. You should run `dbt deps` to update the package. You may also need to run a [full refresh](https://docs.getdbt.com/reference/commands/run.md) of the models in this package. ##### Uninstalling a package When you remove a package from your `packages.yml` file, it isn't automatically deleted from your dbt project, as it still exists in your `dbt_packages/` directory. If you want to completely uninstall a package, you should either: * delete the package directory in `dbt_packages/`; or * run `dbt clean` to delete *all* packages (and any compiled models), followed by `dbt deps`. ##### Pinning packages Running [`dbt deps`](https://docs.getdbt.com/reference/commands/deps.md) "pins" each package by creating or updating the `package-lock.yml` file in the *project\_root* where `packages.yml` is recorded. * The `package-lock.yml` file contains a record of all packages installed. * If subsequent `dbt deps` runs contain no changes to `dependencies.yml` or `packages.yml`, dbt installs from `package-lock.yml`. For example, if you use a branch name, the `package-lock.yml` file pins to the head commit. If you use a version range, it pins to the latest release. In either case, subsequent commits or versions will **not** be installed. To get new commits or versions, run `dbt deps --upgrade` or add `package-lock.yml` to your .gitignore file. dbt will warn you if you install a package using the `git` syntax without specifying a revision (see below). ##### Configuring packages You can configure the models and seeds in a package from the `dbt_project.yml` file, like so: dbt\_project.yml ```yml vars: snowplow: 'snowplow:timezone': 'America/New_York' 'snowplow:page_ping_frequency': 10 'snowplow:events': "{{ ref('sp_base_events') }}" 'snowplow:context:web_page': "{{ ref('sp_base_web_page_context') }}" 'snowplow:context:performance_timing': false 'snowplow:context:useragent': false 'snowplow:pass_through_columns': [] models: snowplow: +schema: snowplow seeds: snowplow: +schema: snowplow_seeds ``` For example, when using a dataset specific package, you may need to configure variables for the names of the tables that contain your raw data. Configurations made in your project YAML file (`dbt_project.yml`) will override any configurations in a package (either in the project YAML file of the package, or in config blocks). ##### Specifying unpinned Git packages If your project specifies an "unpinned" Git package, you may see a warning like: ```text The git package "https://github.com/dbt-labs/dbt-utils.git" is not pinned. This can introduce breaking changes into your project without warning! ``` This warning can be silenced by setting `warn-unpinned: false` in the package specification. **Note:** This is not recommended. packages.yml ```yaml packages: - git: https://github.com/dbt-labs/dbt-utils.git warn-unpinned: false ``` #### Troubleshooting If you encounter errors while working with dbt packages, see the following FAQs: Why am I receiving a Runtime Error in my packages? If you're receiving the runtime error below in your packages.yml folder, it may be due to an old version of your dbt\_utils package that isn't compatible with your current dbt version. ```shell Running with dbt=xxx Runtime Error Failed to read package: Runtime Error Invalid config version: 1, expected 2 Error encountered in dbt_utils/dbt_project.yml ``` Try updating the old version of the dbt\_utils package in your packages.yml to the latest version found in the [dbt hub](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/): ```shell packages: - package: dbt-labs/dbt_utils version: xxx ``` If you've tried the workaround above and are still experiencing this behavior - reach out to the Support team at <support@getdbt.com> and we'll be happy to help! \[Error] Could not find my\_project package If a package name is included in the `search_order` of a project-level `dispatch` config, dbt expects that package to contain macros which are viable candidates for dispatching. If an included package does not contain *any* macros, dbt will raise an error like: ```shell Compilation Error In dispatch: Could not find package 'my_project' ``` This does not mean the package or root project is missing—it means that any macros from it are missing, and so it is missing from the search spaces available to `dispatch`. If you've tried the step above and are still experiencing this behavior - reach out to the Support team at <support@getdbt.com> and we'll be happy to help! --- ### Parallel microbatch execution Use parallel batch execution to process your microbatch models faster. The microbatch strategy offers the benefit of updating a model in smaller, more manageable batches. Depending on your use case, configuring your microbatch models to run in parallel offers faster processing, in comparison to running batches sequentially. Parallel batch execution means that multiple batches are processed at the same time, instead of one after the other (sequentially) for faster processing of your microbatch models. dbt automatically detects whether a batch can be run in parallel in most cases, which means you don’t need to configure this setting. However, the [`concurrent_batches` config](https://docs.getdbt.com/reference/resource-properties/concurrent_batches.md) is available as an override (not a gate), allowing you to specify whether batches should or shouldn’t be run in parallel in specific cases. For example, if you have a microbatch model with 12 batches, you can execute those batches to run in parallel. Specifically they'll run in parallel limited by the number of [available threads](https://docs.getdbt.com/docs/running-a-dbt-project/using-threads.md). #### Prerequisites To use parallel execution, you must meet the following prerequisites: * Use a supported adapter: Snowflake or BigQuery. * We'll continue to test and add concurrency support for more adapters in the future. * A batch can only be run in parallel if: * The batch is *not* the first batch. * The batch is *not* the last batch. #### How parallel batch execution works After checking for the conditions in the [prerequisites](#prerequisites), and if `concurrent_batches` value isn't set, dbt will intelligently auto-detect if the model invokes the [`{{ this }}`](https://docs.getdbt.com/reference/dbt-jinja-functions/this.md) Jinja function. If it references `{{ this }}`, the batches will run sequentially since `{{ this }}` represents the database of the current model and referencing the same relation causes conflict. Otherwise, if `{{ this }}` isn't detected (and other conditions are met), the batches will run in parallel, which can be overriden when you [set a value for `concurrent_batches`](https://docs.getdbt.com/reference/resource-properties/concurrent_batches.md). #### Parallel or sequential execution Choosing between parallel batch execution and sequential processing depends on the specific requirements of your use case. * Parallel batch execution is faster but requires logic independent of batch execution order. For example, if you're developing a data pipeline for a system that processes user transactions in batches, each batch is executed in parallel for better performance. However, the logic used to process each transaction shouldn't depend on the order of how batches are executed or completed. * Sequential processing is slower but essential for calculations like [cumulative metrics](https://docs.getdbt.com/docs/build/cumulative.md) in microbatch models. It processes data in the correct order, allowing each step to build on the previous one. #### Configure `concurrent_batches` By default, dbt auto-detects whether batches can run in parallel for microbatch models, and this works correctly in most cases. However, you can override dbt's detection by setting the [`concurrent_batches` config](https://docs.getdbt.com/reference/resource-properties/concurrent_batches.md) in your `dbt_project.yml` or model `.sql` file to specify parallel or sequential execution, given you meet all the [conditions](#prerequisites): ##### dbt\_project.yml dbt\_project.yml ```yaml models: +concurrent_batches: true # value set to true to run batches in parallel ``` ##### my\_model.sql models/my\_model.sql ```sql {{ config( materialized='incremental', incremental_strategy='microbatch', event_time='session_start', begin='2020-01-01', batch_size='day', concurrent_batches=true, # value set to true to run batches in parallel ... ) }} select ... ``` --- ### Power BI Preview ### Power BI [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt platform | Starter, Enterprise, Enterprise+ The Power BI integration enables you to query the Semantic Layer directly, allowing you to build dashboards with trusted, live data in Power BI. It provides a live connection to the Semantic Layer through Power BI Desktop or Power BI Service. info This integration is currently in preview. Production readiness depends on Microsoft's participation in the Apache Ossie standard. Using Power BI? You can help shape the future of this integration. Ask your Microsoft representative about Ossie support! #### Prerequisites * You have [configured the Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md). * You are on a supported [dbt release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) or on dbt v1.6 or higher. * You installed [Power BI Desktop or Power BI On-premises Data Gateway](https://learn.microsoft.com/en-us/power-bi/connect-data/service-gateway-custom-connectors). * Power BI Service doesn't natively support custom connectors. To use the connector in Power BI Service, you must install and configure it on an On-premises Data Gateway. * You need your [dbt host](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md#3-view-connection-detail), [Environment ID](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md#set-up-dbt-semantic-layer), and a [service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) or a [personal access token](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) to log in. This account should be set up with the Semantic Layer. * You must have a dbt Starter or Enterprise-tier [account](https://www.getdbt.com/pricing). Suitable for both Multi-tenant and Single-tenant deployment. 📹 Learn about the dbt Semantic Layer with on-demand video courses! Explore our [dbt Semantic Layer on-demand course](https://learn.getdbt.com/courses/semantic-layer) to learn how to define and query metrics in your dbt project. Additionally, dive into mini-courses for querying the dbt Semantic Layer in your favorite tools: [Tableau](https://courses.getdbt.com/courses/tableau-querying-the-semantic-layer), [Excel](https://learn.getdbt.com/courses/querying-the-semantic-layer-with-excel), [Hex](https://courses.getdbt.com/courses/hex-querying-the-semantic-layer), and [Mode](https://courses.getdbt.com/courses/mode-querying-the-semantic-layer). #### Install the connector power bi versions The Power BI connector may be incompatible with older versions of Power BI desktop. For the best results, we recommend installing the most recent version directly from the [Microsoft Store](https://apps.microsoft.com/detail/9ntxr16hnw1t?hl=en-US\&gl=US) or [Download Center](https://www.microsoft.com/en-us/download/details.aspx?id=58494). The Semantic Layer Power BI connector consists of a custom `.pqx` Power BI connector and an ODBC driver. Install both using our Windows installer by following these steps: 1. Download and install the [`.msi` installer](https://github.com/dbt-labs/semantic-layer-powerbi-connector/releases/download/v1.0.0/dbt.Semantic.Layer.for.Power.BI.zip) 2. Run the installer and follow the on-screen instructions to install the ODBC driver and connector onto your Power BI Desktop. ##### Verify installation Note that users on older versions of Power BI may have to [configure the connector](#configure-the-connector) before they can verify the installation. To verify the installation: 1. Open **ODBC Data Sources (64-bit)** file on your computer. 2. Navigate to **System DSN** and verify that the `dbt Labs ODBC DSN` is registered. 3. Navigate to **Drivers** and verify that the `dbt Labs ODBC Driver` is installed. 4. Open Power BI Desktop, navigate to **Settings**, then **Data Source Settings**. Verify that the `dbt Semantic Layer` connector is properly loaded. To allow published reports in Power BI Service to use the connector. An IT admin in your organization needs to install and configure the connector on an On-premises Data Gateway. #### For IT admins This section is for IT admins trying to install the ODBC driver and connector into an On-premises Data Gateway. To allow published reports to use the connector in Power BI Service, an IT Admin must install and configure the connector: 1. Install the ODBC driver and connector into an On-premises Data Gateway. Run the same `.msi` installer used for Power BI Desktop and install it on the machine where your gateway is hosted. 2. Copy connector file to Gateway directory: 1. Locate that `.pqx` file: `C:\Users\<YourUser>\Documents\Power BI Desktop\Custom Connectors\dbtSemanticLayer.pqx`. 2. Copy it to the Power BI On-premises Data Gateway custom connectors directory: `C:\Windows\ServiceProfiles\PBIEgwService\Documents\Power BI Desktop\Custom Connectors`. 3. Verify installation by following the steps from the [install the connector](#verify-installation) section. 4. Enable connector in Power BI Enterprise Gateway: 1. Open the `EnterpriseGatewayConfigurator.exe`. 2. Navigate to **Connectors**. 3. Verify that the `dbt Semantic Layer` connector is installed and active. For more information on how to set up custom connectors in the Power BI On-premises Data Gateway, refer to Power BI’s [official documentation](https://learn.microsoft.com/en-us/power-bi/connect-data/service-gateway-custom-connectors). #### Configure the connector After installing the connector, you’ll have to configure your project credentials to connect to the Semantic Layer from a report. To configure project credentials in Power BI Desktop: 1. Create a blank report. 2. On the top-left, click on **Get data**. 3. Search for Semantic Layer, then click **Connect**. 4. Fill in your connection details. You can find your Host and Environment ID under the Semantic Layer configuration for your dbt project. tip Make sure you select **DirectQuery** under **Data Connectivity mode** since the Semantic Layer connector does not support **Import** mode. See [Considerations](#considerations) for more details. 5. Click **OK** to proceed. [![Select DirectQuery mode](/img/docs/platform-integrations/sl-pbi/pbi-directquery.jpg?v=2 "Select DirectQuery mode")](#)Select DirectQuery mode 6. On the next screen, paste your service or personal token and then click **Connect**. 7. You should see a side pane with a few "virtual" tables. `ALL` represents all of your defined semantic layer objects. The other tables represent each of your saved queries. Select the one you want to load into your dashboard. Then click **Load**. [![Select tables in the side panel](/img/docs/platform-integrations/sl-pbi/pbi-sidepanel.jpg?v=2 "Select tables in the side panel")](#)Select tables in the side panel Now that you've configured the connector, you can configure published reports in the next section to use the connector. #### Configure published reports After publishing a report and the first time you hit **Publish** on a given report, configure Power BI Service to use your organization’s On-premises Data Gateway to access data from the Semantic Layer: 1. On the top right, click on **Settings > Power BI settings**. [![Navigate to Settings > Power BI Settings](/img/docs/platform-integrations/sl-pbi/pbi-settings.jpg?v=2 "Navigate to Settings > Power BI Settings")](#)Navigate to Settings > Power BI Settings 2. Navigate to the **Semantic models** tab and select your report on the sidebar on the left. 3. Under **Gateway and cloud connections**, select the **On-premises Data Gateway** where your IT admin has installed the Semantic Layer connector. * If the Status is **Not configured correctly**, you’ll have to configure it. [![Configure the gateway connection](/img/docs/platform-integrations/sl-pbi/pbi-gateway-cloud-connections.jpg?v=2 "Configure the gateway connection")](#)Configure the gateway connection 4. Click on the arrow under **Actions** and then, click on **Manually add to gateway**. [![Manually add to gateway](/img/docs/platform-integrations/sl-pbi/pbi-manual-gateway.jpg?v=2 "Manually add to gateway")](#)Manually add to gateway 5. Provide a name for your connection and enter your connection details. * Set the connection as **Encrypted** (Required). Failing to do so will result in the Semantic Layer servers rejecting the connection. [![Set the connection as Encrypted](/img/docs/platform-integrations/sl-pbi/pbi-encrypted.jpg?v=2 "Set the connection as Encrypted")](#)Set the connection as Encrypted 6. Click **Create**. This will run a connection test (unless you choose to skip it). If the connection succeeds, the connection will be saved. You can now go back to your published report on Power BI Service to assert data loads as expected. #### Use the connector This section describes how to use the Semantic Layer connector in Power BI. The Semantic Layer connector creates: * A virtual table for each saved query. * A `METRICS.ALL` table containing all metrics, and dimensions and entities appear as regular dimension columns. These tables do not actually map to an underlying table in your data warehouse. Instead, Power BI sends queries to these tables and (before actually executing on the warehouse) the Semantic Layer servers: * Parse the SQL. * Extract all the queried columns, group bys and filters. * Generates SQL to query your existing tables. * Returns data back to Power BI, which doesn’t know any of this happened. [![Power BI integration diagram](/img/docs/platform-integrations/sl-pbi/sl-pbi.jpg?v=2 "Power BI integration diagram")](#)Power BI integration diagram This allows for very flexible analytics workflows, like drag and drop metrics and slice by dimensions and entities — the Semantic Layer will generate the appropriate SQL to actually query your data source for you. ###### Modifying time granularity When you select time dimensions in the **Group By** menu, you'll see a list of available time granularities. The lowest granularity is selected by default. Metric time is the default time dimension for grouping your metrics. info Note: [Custom time granularities](https://docs.getdbt.com/docs/build/metricflow-time-spine.md#add-custom-granularities) (like fiscal year) aren't currently supported or accessible in this integration. Only [standard granularities](https://docs.getdbt.com/docs/build/dimensions.md?dimension=time_gran#time) (like day, week, month, and so on) are available. If you'd like to access custom granularities, consider using the [Semantic Layer APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md). #### Considerations  Not every “column” of METRICS.ALL are compatible with every other column * `METRICS.ALL` combines all your existing metrics, entities and dimensions. Queries must be valid Semantic Layer queries, otherwise they'll fail with MetricFlow query compilation errors. * For saved query tables, all “columns” will be compatible with every other “column” since, by definition, saved queries are valid queries that can be sliced by any of the dimensions present in the query.  The dbt Semantic Layer connector does not support Import mode natively * Use `DirectQuery` mode to ensure compatibility. * `Import` mode tries to select an entire table to import into Power BI, which means it'll likely generate SQL that translates to an invalid Semantic Layer query which will try to query all metrics, dimensions and entities at the same time. * To import data into a PowerBI report, select a valid combination of columns to import, (something that will generate a valid Semantic Layer query). * You can use `Table.SelectColumns` for this: `= Table.SelectColumns(Source{[Item="ALL",Schema="METRICS",Catalog=null]}[Data], {"Total Profit", "Metric Time (Day)"})` * Be aware that all calculations will happen inside of Power BI and won’t pass through Semantic Layer servers. This could lead to incorrect or diverging results. * For example, the Semantic Layer is usually responsible for rolling up cumulative metrics to coarser time granularities. Doing a sum over all the weeks in a year to get a yearly granularity out of a weekly Semantic Layer query will most likely generate incorrect results. Instead, you should query the Semantic Layer directly to get accurate results.  The dbt Semantic Layer connector ignores aggregations defined in Power BI * If you change the aggregation type of a metric from `SUM()` to `COUNT()` or anything else, nothing will change. This is because aggregation functions are defined in the Semantic Layer and we ignore them when translating Power BI generated SQL into Semantic Layer queries. * Aggregations like `Count (Distinct)`, `Standard Deviation`, `Variance`, and `Median` in Power BI may return an error and not work at all.  What actions aren't supported? The following are not supported: * Custom modeling * Joining tables * Creating custom columns within a table * Custom Data Analysis Expressions (DAX) or Power Query (PQ) --- ### Preview new and experimental features in the dbt platform dbt platform dbt Labs often tests experimental features before deciding to continue on the [Product lifecycle](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles.md). You can access experimental features to preview beta features that haven’t yet been released to dbt. You can toggle on or off all experimental features by clicking your account name in the bottom left, selecting **Your profile**, and finding the toggle under **User profile**. Experimental features: * May not be feature-complete or fully stable as we’re actively developing them. * Could be discontinued at any time. * May require feedback from you to understand their limitations or impact. Each experimental feature collects feedback directly in dbt, which may impact dbt Labs' decisions to implement. * May have limited technical support and be excluded from our Support SLAs. * May not have public documentation available. To enable or disable experimental features: 1. Click your account name in the bottom left and select **Your profile**. 2. Under **User profile**, find **Experimental features**. 3. Click **Beta** to toggle the features on or off as shown in the following image. ![Experimental features](/assets/images/experimental-feats-a099dce8fc8f8ac6081f85df4b0aa379.png) #### Beta terms and conditions By using or enabling features that are not yet in general release ("Beta Features"), you agree to the [Beta Features Terms and Conditions](https://docs.getdbt.com/assets/files/beta-tc-740ff696113c89c38a96bb70b968775e.pdf). --- ### Product lifecycles dbt Labs manages the lifecycle of features across the cloud-hosted dbt platform and self-hosted v1 and v2. Each feature is assigned a lifecycle status that describes its stability, support level, and availability. Use the tabs below to find the lifecycle stages for the product you're using. Service level objective (SLO) support varies between products and lifecycles. ##### dbt platform dbt platform features adhere to the following lifecycle path: **Beta** In active development. May not be fully stable and breaking changes can occur. Documentation may be incomplete, technical support is limited, and SLOs may not apply. Download the [Beta Terms and Conditions](https://docs.getdbt.com/assets/files/beta-tc-740ff696113c89c38a96bb70b968775e.pdf) for details. If marked `Private`, access must be enabled by dbt Labs. **Preview** Stable and functionally ready for production. Planned additions or non-backward-compatible changes may still occur before GA. Includes documentation, technical support, and SLOs. If marked `Private`, access must be enabled by dbt Labs. **Generally available (GA)** Stable features available to all qualified dbt accounts. SLOs, documentation, and technical support apply. Pricing changes may change or apply. Feature availability may depend on your environment's dbt version. Use a supported [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) to receive the latest GA features. **Deprecated** No longer being actively developed or enhanced. Features continue to function as-is and documentation remains available until the removal date. Technical support no longer applies. **Removed** No longer available on the platform in any capacity. ##### dbt v1 and v2 Self-hosted dbt releases follow semantic versioning. Read more in [About dbt versions](https://docs.getdbt.com/docs/dbt-versions.md). Both v1 and v2 releases adhere to the following lifecycle path: **Undocumented** Open source dbt codebases may have visibility into internal, non-contracted, or intentionally undocumented functionality. Not considered part of the release's product surface area. **Unreleased** Planned for the next minor version prerelease. No commitments on behavior or implementation. Maintainers reserve the right to change or remove it entirely. **Alpha** No commitments on behavior or implementation and not intended for any production work. Use at your own discretion. Maintainers reserve the right to change or remove it entirely. **Beta** First glimpse of net-new features in an upcoming release. Code should work without regressions, but new features may be incomplete or have known edge cases. Changes are not locked and maintainers may still alter or remove them. **Release Candidate** A 2-week window for production-level testing before final release. Features are expected to ship as-is, though maintainers may still address significant bugs before the final release. **Generally Available** Ready for use in production. **Deprecated** No longer actively developed or enhanced. Continues to function as-is until its removal date. --- ### Project dependencies dbt platform | Enterprise, Enterprise+ Available on dbt [Enterprise or Enterprise+](https://www.getdbt.com/pricing) plans. For a long time, dbt has supported code reuse and extension by installing other projects as [packages](https://docs.getdbt.com/docs/build/packages.md). When you install another project as a package, you are pulling in its full source code, and adding it to your own. This enables you to call macros and run models defined in that other project. While this is a great way to reuse code, share utility macros, and establish a starting point for common transformations, it's not a great way to enable collaboration across teams and at scale, especially in larger organizations. dbt Labs supports an expanded notion of `dependencies` across multiple dbt projects: * **Packages** — Familiar and pre-existing type of dependency. You take this dependency by installing the package's full source code (like a software library). * **Projects** — The dbt method to take a dependency on another project. Using a metadata service that runs behind the scenes, dbt resolves references on-the-fly to public models defined in other projects. You don't need to parse or run those upstream models yourself. Instead, you treat your dependency on those models as an API that returns a dataset. The maintainer of the public model is responsible for guaranteeing its quality and stability. #### Prerequisites * Available in [dbt Enterprise or Enterprise+](https://www.getdbt.com/pricing). To use it, designate a [public model](https://docs.getdbt.com/docs/mesh/govern/model-access.md) and add a [cross-project ref](#how-to-write-cross-project-ref). * For the upstream ("producer") project setup: * Configure models in upstream project with [`access: public`](https://docs.getdbt.com/reference/resource-configs/access.md) and have at least one successful job run after defining `access`. * Define a [Production deployment environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md#set-as-production-environment) in the upstream project and make sure at least *one deployment job* has run successfully there. This job should generate a [`manifest.json` file](https://docs.getdbt.com/reference/artifacts/manifest-json.md) — it includes the metadata needed for downstream projects. * If the upstream project has a Staging environment, run at least one successful deployment job there to ensure downstream cross-project references resolve correctly. * Each project `name` must be unique in your dbt account. For example, if you have a dbt project (codebase) for the `jaffle_marketing` team, avoid creating projects for `Jaffle Marketing - Dev` and `Jaffle Marketing - Prod`; use [environment-level isolation](https://docs.getdbt.com/docs/dbt-platform-environments.md#types-of-environments) instead. * dbt supports [Connections](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md#connection-management), available to all dbt users. Connections allows different data platform connections per environment, eliminating the need to duplicate projects. Projects can use multiple connections of the same warehouse type. Connections are reusable across projects and environments. * The `dbt_project.yml` file is case-sensitive, which means the project name must exactly match the name in your `dependencies.yml`. For example, `jaffle_marketing`, not `JAFFLE_MARKETING`. #### Use cases The following setup will work for every dbt project: * Add [any package dependencies](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#when-to-use-project-dependencies) to `packages.yml` * Add [any project dependencies](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#when-to-use-package-dependencies) to `dependencies.yml` However, you may be able to consolidate both into a single `dependencies.yml` file. Read the following section to learn more. ###### About packages.yml and dependencies.yml The `dependencies.yml`. file can contain both types of dependencies: "package" and "project" dependencies. * [Package dependencies](https://docs.getdbt.com/docs/build/packages.md#how-do-i-add-a-package-to-my-project) lets you add source code from someone else's dbt project into your own, like a library. * Project dependencies provide a different way to build on top of someone else's work in dbt. * Private packages are not supported in `dependencies.yml` because they intentionally don't support Jinja rendering or conditional configuration. This is to maintain static and predictable configuration and ensures compatibility with other services, like dbt. If your dbt project doesn't require the use of Jinja within the package specifications, you can simply rename your existing `packages.yml` to `dependencies.yml`. However, something to note is if your project's package specifications use Jinja, particularly for scenarios like adding an environment variable or a [Git token method](https://docs.getdbt.com/docs/build/packages.md#git-token-method) in a private Git package specification, you should continue using the `packages.yml` file name. Use the following toggles to understand the differences and determine when to use `dependencies.yml` or `packages.yml` (or both). Refer to the [FAQs](#faqs) for more info.  When to use Project dependencies Project dependencies are designed for the [dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md) and [cross-project reference](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#how-to-write-cross-project-ref) workflow: * Use `dependencies.yml` when you need to set up cross-project references between different dbt projects, especially in a dbt Mesh setup. * Use `dependencies.yml` when you want to include both projects and non-private dbt packages in your project's dependencies. * Use `dependencies.yml` for organization and maintainability if you're using both [cross-project refs](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#how-to-write-cross-project-ref) and [dbt Hub packages](https://hub.getdbt.com/). This reduces the need for multiple YAML files to manage dependencies.  When to use Package dependencies Package dependencies allow you to add source code from someone else's dbt project into your own, like a library: * If you only use packages like those from the [dbt Hub](https://hub.getdbt.com/), remain with `packages.yml`. * Use `packages.yml` when you want to download dbt packages, such as dbt projects, into your root or parent dbt project. Something to note is that it doesn't contribute to the dbt Mesh workflow. * Use `packages.yml` to include packages in your project's dependencies. This includes both public packages, such as those from the [dbt Hub](https://hub.getdbt.com/), and private packages. dbt now supports [native private packages](https://docs.getdbt.com/docs/build/packages.md#native-private-packages). * [`packages.yml` supports Jinja rendering](https://docs.getdbt.com/docs/build/dbt-tips.md#yaml-tips) for historical reasons, allowing dynamic configurations. This can be useful if you need to insert values, like a [Git token method](https://docs.getdbt.com/docs/build/packages.md#git-token-method) from an environment variable, into your package specifications. Previously, to use private Git repositories in dbt, you needed to use a workaround that involved embedding a Git token with Jinja. This is not ideal as it requires extra steps like creating a user and sharing a Git token. We’ve introduced support for [native private packages](https://docs.getdbt.com/docs/build/packages.md#native-private-packages-) to address this. #### Define project dependencies If your dbt project relies on models from another project, you can define that relationship using project dependencies. The following steps walk you through specifying project dependencies in dbt: 1. Create a file called `dependencies.yml` at the root of your dbt project. 2. In the `dependencies.yml`, list the upstream dbt project your project depends on as they appear in the `dbt_projects.yml` file. 3. (Optional) Define the specific models you expect from that upstream project to make the dependency explicit. 4. Use [`ref()`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) with the project name to reference upstream models in your SQL. 5. Commit the changes and ensure the dependency is configured in dbt. 6. dbt will resolve the dependency, ensure upstream projects are built first, and surface cross-project lineage in the lineage and DAG (Directed Acyclic Graph) views. ##### Example As an example, let's say you work on the Marketing team at the Jaffle Shop. The name of your team's project is `jaffle_marketing`: dbt\_project.yml ```yml name: jaffle_marketing ``` As part of your modeling of marketing data, you need to take a dependency on two other projects: * `dbt_utils` as a package: A collection of utility macros you can use while writing the SQL for your own models. This package is open-source public and maintained by dbt Labs. * `jaffle_finance` as a project use case: Data models about the Jaffle Shop's revenue. This project is private and maintained by your colleagues on the Finance team. You want to select from some of this project's final models, as a starting point for your own work. Refer to [Use cases](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#use-cases) for information on package and project dependencies. dependencies.yml ```yml packages: - package: dbt-labs/dbt_utils version: 1.1.1 projects: - name: jaffle_finance # case sensitive and matches the 'name' in the 'dbt_project.yml' ``` What's happening here? The `dbt_utils` package — When you run `dbt deps`, dbt will pull down this package's full contents (100+ macros) as source code and add them to your environment. You can then call any macro from the package, just as you can call macros defined in your own project. The `jaffle_finance` projects — This is a new scenario. Unlike installing a package, the models in the `jaffle_finance` project will *not* be pulled down as source code and parsed into your project. Instead, dbt provides a metadata service that resolves references to [**public models**](https://docs.getdbt.com/docs/mesh/govern/model-access.md) defined in the `jaffle_finance` project. ##### Advantages When you're building on top of another team's work, resolving the references in this way has several advantages: * You're using an intentional interface designated by the model's maintainer with `access: public`. * You're keeping the scope of your project narrow, and avoiding unnecessary resources and complexity. This is faster for you and faster for dbt. * You don't need to mirror any conditional configuration of the upstream project such as `vars`, environment variables, or `target.name`. You can reference them directly wherever the Finance team is building their models in production. Even if the Finance team makes changes like renaming the model, changing the name of its schema, or [bumping its version](https://docs.getdbt.com/docs/mesh/govern/model-versions.md), your `ref` would still resolve successfully. * You eliminate the risk of accidentally building those models with `dbt run` or `dbt build`. While you can select those models, you can't actually build them. This prevents unexpected warehouse costs and permissions issues. This also ensures proper ownership and cost allocation for each team's models. ##### How to write cross-project ref **Writing `ref`:** Models referenced from a `project`-type dependency must use [two-argument `ref`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models), including the project name: models/marts/roi\_by\_channel.sql ```sql with monthly_revenue as ( select * from {{ ref('jaffle_finance', 'monthly_revenue') }} ), ... ``` (Applies to dbt v1.12 and later) Cross-project refs unsupported in latest SL YAML spec When using [dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md) with the [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md), [referencing models from different projects](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models) is only supported in the legacy YAML spec, where semantic models are defined as top-level resources and can reference models across projects. In the [latest YAML spec](https://docs.getdbt.com/docs/build/latest-metrics-spec.md), semantic models are defined within model YAML files, and cross-project references are not yet supported. Support for this capability in the latest spec is planned for a future release. ###### Cycle detection You can enable bidirectional dependencies across projects so these relationships can go in either direction, meaning that the `jaffle_finance` project can add a new model that depends on any public models produced by the `jaffle_marketing` project, so long as the new dependency doesn't introduce any node-level cycles. dbt checks for cycles across projects and raises errors if any are detected. When setting up projects that depend on each other, it's important to do so in a stepwise fashion. Each project must run and produce public models before the original producer project can take a dependency on the original consumer project. For example, the order of operations would be as follows for a simple two-project setup: 1. The `project_a` project runs in a deployment environment and produces public models. 2. The `project_b` project adds `project_a` as a dependency. 3. The `project_b` project runs in a deployment environment and produces public models. 4. The `project_a` project adds `project_b` as a dependency. For more guidance on how to use Mesh, refer to the dedicated [Mesh guide](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md) and also our freely available [Mesh learning course](https://learn.getdbt.com/courses/dbt-mesh). ##### Safeguarding production data with staging environments When working in a Development environment, cross-project `ref`s normally resolve to the Production environment of the project. However, to protect production data, set up a [Staging deployment environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md#staging-environment) within your projects. With a staging environment integrated into the project, Mesh automatically fetches public model information from the producer’s staging environment if the consumer is also in staging. Similarly, Mesh fetches from the producer’s production environment if the consumer is in production. This ensures consistency between environments and adds a layer of security by preventing access to production data during development workflows. Read [Why use a staging environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md#why-use-a-staging-environment) for more information about the benefits. ###### Staging with downstream dependencies dbt begins using the Staging environment to resolve cross-project references from downstream projects as soon as it exists in a project without "fail-over" to Production. This means that dbt will consistently use metadata from the Staging environment to resolve references in downstream projects, even if there haven't been any successful runs in the configured Staging environment. To avoid causing downtime for downstream developers, you should define and trigger a job before marking the environment as Staging: 1. Create a new environment, but do NOT mark it as **Staging**. 2. Define a job in that environment. 3. Trigger the job to run, and ensure it completes successfully. 4. Update the environment to mark it as **Staging**. ##### Comparison If you were to instead install the `jaffle_finance` project as a `package` dependency, you would instead be pulling down its full source code and adding it to your runtime environment. This means: * dbt needs to parse and resolve more inputs (which is slower) * dbt expects you to configure these models as if they were your own (with `vars`, env vars, etc) * dbt will run these models as your own unless you explicitly `--exclude` them * You could be using the project's models in a way that their maintainer (the Finance team) hasn't intended There are a few cases where installing another internal project as a package can be a useful pattern: * Unified deployments — In a production environment, if the central data platform team of Jaffle Shop wanted to schedule the deployment of models across both `jaffle_finance` and `jaffle_marketing`, they could use dbt's [selection syntax](https://docs.getdbt.com/reference/node-selection/syntax.md) to create a new "passthrough" project that installed both projects as packages. * Coordinated changes — In development, if you wanted to test the effects of a change to a public model in an upstream project (`jaffle_finance.monthly_revenue`) on a downstream model (`jaffle_marketing.roi_by_channel`) *before* introducing changes to a staging or production environment, you can install the `jaffle_finance` package as a package within `jaffle_marketing`. The installation can point to a specific git branch, however, if you find yourself frequently needing to perform end-to-end testing across both projects, we recommend you re-examine if this represents a stable interface boundary. These are the exceptions, rather than the rule. Installing another team's project as a package adds complexity, latency, and risk of unnecessary costs. By defining clear interface boundaries across teams, by serving one team's public models as "APIs" to another, and by enabling practitioners to develop with a more narrowly defined scope, we can enable more people to contribute, with more confidence, while requiring less context upfront. #### FAQs Can I define private packages in the dependencies.yml file? It depends on how you're accessing your private packages: * If you're using [native private packages](https://docs.getdbt.com/docs/build/packages.md#native-private-packages), you can define them in the `dependencies.yml` file. * If you're using the [git token method](https://docs.getdbt.com/docs/build/packages.md#git-token-method), you must define them in the `packages.yml` file instead of the `dependencies.yml` file. This is because conditional rendering (like Jinja-in-yaml) is not supported in `dependencies.yml`. Why doesn’t an indirectly referenced upstream public model appear in Explorer? For [project dependencies](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md) in Mesh, [Catalog](https://docs.getdbt.com/docs/explore/explore-multiple-projects.md) only displays directly referenced [public models](https://docs.getdbt.com/docs/mesh/govern/model-access.md) from upstream projects, even if an upstream model indirectly depends on another public model. So for example, if: * `project_b` adds `project_a` as a dependency * `project_b`'s model `downstream_c` references `project_a.upstream_b` * `project_a.upstream_b` references another public model, `project_a.upstream_a` Then: * In Explorer, only directly referenced public models (`upstream_b` in this case) appear. * In the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) lineage view, however, `upstream_a` (the indirect dependency) *will* appear because dbt dynamically resolves the full dependency graph. This behavior makes sure that Catalog only shows the immediate dependencies available to that specific project. #### Related docs * Refer to the [Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md) guide for more guidance on how to use Mesh. * [Quickstart with Mesh](https://docs.getdbt.com/guides/mesh-qs.md) * [Hybrid development with dbt platform and dbt v2](https://docs.getdbt.com/guides/dbt-platform-local-workflow.md) — manage credentials, env vars, deferral, and dbt v2 versions when developing locally against a Mesh project --- ### Project recommendations dbt platform | Enterprise, Enterprise+ Catalog provides recommendations about your project from the `dbt_project_evaluator` [package](https://hub.getdbt.com/dbt-labs/dbt_project_evaluator/latest/) using metadata from the [Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md). * Catalog also offers a global view, showing all the recommendations across the project for easy sorting and summarizing. * These recommendations provide insight into how you can create a better-documented, better-tested, and better-built dbt project, creating more trust and less confusion. * For a seamless and consistent experience, recommendations use `dbt_project_evaluator`'s pre-defined settings and don't import customizations applied to your package or project. On-demand learning If you enjoy video courses, check out our [dbt Catalog on-demand course](https://learn.getdbt.com/courses/dbt-catalog) and learn how to best explore your dbt project(s)! #### Recommendations page The Recommendations overview page includes two top-level metrics measuring the test and documentation coverage of the models in your project. * **Model test coverage** — The percent of models in your project (models not from a package or imported via Mesh) with at least one dbt test configured on them. * **Model documentation coverage** — The percent of models in your project (models not from a package or imported via Mesh) with a description. [![Example of the Recommendations overview page with project metrics and the recommendations for all resources in the project](/img/docs/collaborate/dbt-explorer/example-recommendations-overview.png?v=2 "Example of the Recommendations overview page with project metrics and the recommendations for all resources in the project")](#)Example of the Recommendations overview page with project metrics and the recommendations for all resources in the project #### List of rules The following table lists the rules currently defined in the `dbt_project_evaluator` [package](https://hub.getdbt.com/dbt-labs/dbt_project_evaluator/latest/). | Category | Name | Description | Package Docs Link | | ------------- | ----------------------------- | -------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | | Modeling | Direct Join to Source | Model that joins both a model and source, indicating a missing staging model | [GitHub](https://dbt-labs.github.io/dbt-project-evaluator/0.8/rules/modeling/#direct-join-to-source) | | Modeling | Duplicate Sources | More than one source node corresponds to the same data warehouse relation | [GitHub](https://dbt-labs.github.io/dbt-project-evaluator/0.8/rules/modeling/#duplicate-sources) | | Modeling | Multiple Sources Joined | Models with more than one source parent, indicating lack of staging models | [GitHub](https://dbt-labs.github.io/dbt-project-evaluator/0.8/rules/modeling/#multiple-sources-joined) | | Modeling | Root Model | Models with no parents, indicating potential hardcoded references and need for sources | [GitHub](https://dbt-labs.github.io/dbt-project-evaluator/0.8/rules/modeling/#root-models) | | Modeling | Source Fanout | Sources with more than one model child, indicating a need for staging models | [GitHub](https://dbt-labs.github.io/dbt-project-evaluator/0.8/rules/modeling/#source-fanout) | | Modeling | Unused Source | Sources that are not referenced by any resource | [GitHub](https://dbt-labs.github.io/dbt-project-evaluator/0.8/rules/modeling/#unused-sources) | | Performance | Exposure Dependent on View | Exposures with at least one model parent materialized as a view, indicating potential query performance issues | [GitHub](https://dbt-labs.github.io/dbt-project-evaluator/0.8/rules/performance/#exposure-parents-materializations) | | Testing | Missing Primary Key Test | Models with insufficient testing on the grain of the model. | [GitHub](https://dbt-labs.github.io/dbt-project-evaluator/0.8/rules/testing/#missing-primary-key-tests) | | Documentation | Undocumented Models | Models without a model-level description | [GitHub](https://dbt-labs.github.io/dbt-project-evaluator/0.8/rules/documentation/#undocumented-models) | | Documentation | Undocumented Source | Sources (collections of source tables) without descriptions | [GitHub](https://dbt-labs.github.io/dbt-project-evaluator/0.8/rules/documentation/#undocumented-sources) | | Documentation | Undocumented Source Tables | Source tables without descriptions | [GitHub](https://dbt-labs.github.io/dbt-project-evaluator/0.8/rules/documentation/#undocumented-source-tables) | | Governance | Public Model Missing Contract | Models with public access that do not have a model contract to ensure the data types | [GitHub](https://dbt-labs.github.io/dbt-project-evaluator/0.8/rules/governance/#public-models-without-contracts) | #### The Recommendations tab Models, sources, and exposures each also have a **Recommendations** tab on their resource details page, with the specific recommendations that correspond to that resource: [![Example of the Recommendations tab ](/img/docs/collaborate/dbt-explorer/example-recommendations-tab.png?v=2 "Example of the Recommendations tab ")](#)Example of the Recommendations tab --- ### Project variables dbt provides a mechanism called [variables](https://docs.getdbt.com/reference/dbt-jinja-functions/var.md) to provide data to models for compilation. Variables allow you to define configurable values for your project instead of hardcoding them in SQL. (Applies to dbt v1.12 and later) You can define variables in three ways: * In the `vars.yml` file (recommended for projects with several variables, or when you need to reference variables from within [`dbt_project.yml`](https://docs.getdbt.com/reference/dbt_project.yml.md)) * In the `dbt_project.yml` file * On the command line Variables defined in the `vars.yml` or `dbt_project.yml` act as project-wide defaults. These defaults apply anywhere the variable is referenced. You can override them at runtime using the `--vars` command-line argument, which is useful when you want to change a value for a specific run. For example, when testing with a different date range or running models with environment-specific settings. You might use variables to [configure timezones](https://github.com/dbt-labs/snowplow/blob/0.3.9/dbt_project.yml#L22), set reporting date ranges, [avoid hardcoding table names](https://github.com/dbt-labs/quickbooks/blob/v0.1.0/dbt_project.yml#L23), or otherwise control how models are compiled. To use a variable in a model, hook, or macro, use the `{{ var('...') }}` function. The `var()` function retrieves the value defined in your project or passed using `--vars`. For more information, see [About var function](https://docs.getdbt.com/reference/dbt-jinja-functions/var.md). Note, refer to [YAML tips](https://docs.getdbt.com/docs/build/dbt-tips.md#yaml-tips) for more YAML information. (Applies to dbt v1.12 and later) ##### Defining variables in `vars.yml` `vars.yml` is a dedicated file for project variables located at the project root. Because dbt parses it *before* `dbt_project.yml`, variables from `vars.yml` are available when `dbt_project.yml` is evaluated, so you can reference them in `dbt_project.yml` using `{{ var('...') }}`. For more information, refer to [Using vars.yml variables in dbt\_project.yml](#using-varsyml-variables-in-dbt_projectyml). To define variables in a `vars.yml` file, add a `vars` config. You can scope these variables globally, or to a specific package imported in your project. The following example shows variables in a `vars.yml` file: vars.yml ```yaml vars: # The `start_date` variable is accessible to all resources start_date: '2016-06-01' # The `platforms` variable is only accessible to resources in the my_dbt_project project my_dbt_project: platforms: ['web', 'mobile'] # The `app_ids` variable is only accessible to resources in the snowplow package snowplow: app_ids: ['marketing', 'app', 'landing-page'] ``` You cannot define variables in both `vars.yml` and `dbt_project.yml`. If both files contain a `vars` block with definitions, dbt raises an error. If `vars.yml` is empty or doesn't exist, variables defined in `dbt_project.yml` are used instead. ###### Using `vars.yml` variables in `dbt_project.yml` Because `vars.yml` is parsed before `dbt_project.yml`, you can reference variables from `vars.yml` within `dbt_project.yml` using `{{ var('...') }}`. Variables referenced in `dbt_project.yml` are resolved only from `vars.yml` or the `--vars` command-line argument, not from the `vars` block in `dbt_project.yml` itself. If `dbt_project.yml` references a variable that is defined only in its own `vars` block, dbt raises an error. For example, you have the following variables defined in `vars.yml`: vars.yml ```yaml vars: schema_name: analytics materialization: table ``` You can reference them in `dbt_project.yml`: dbt\_project.yml ```yaml models: my_dbt_project: +schema: "{{ var('schema_name') }}" +materialized: "{{ var('materialization') }}" ``` ##### Defining variables in `dbt_project.yml` info Jinja is not supported within the `vars` config, and all values will be interpreted literally. To define variables in a dbt project, add a `vars` config to your `dbt_project.yml` file. These `vars` can be scoped globally, or to a specific package imported in your project. dbt\_project.yml ```yaml name: my_dbt_project version: 1.0.0 config-version: 2 vars: # The `start_date` variable will be accessible in all resources start_date: '2016-06-01' # The `platforms` variable is only accessible to resources in the my_dbt_project project my_dbt_project: platforms: ['web', 'mobile'] # The `app_ids` variable is only accessible to resources in the snowplow package snowplow: app_ids: ['marketing', 'app', 'landing-page'] models: ... ``` (Applies to dbt v1.12 and later) You cannot define variables in both `vars.yml` and `dbt_project.yml`. If both files contain a `vars` block with definitions, dbt raises an error. ##### Defining variables on the command line The `dbt_project.yml` file is a great place to define variables that rarely change. When you need to override a variable for a specific run, use the `--vars` command line option. For example, when you want to test with a different date range, run models with environment-specific settings, or adjust behavior dynamically. Use `--vars` to pass one or more variables to a dbt command. Provide the argument as a YAML dictionary string. For example: ```shell $ dbt run --vars '{"event_type": "signup"}' ``` You can use the same `--vars` syntax with other dbt commands, such as `dbt snapshot`: ```shell $ dbt snapshot --select my_snapshot --vars '{"cutoff_date": "2026-01-01"}' ``` Inside a model, snapshot, or macro, access the value using the `var()` function: ```shell select '{{ var("event_type") }}' as event_type ``` When you pass variables using `--vars`, you can access them anywhere you use the `var()` function in your project. You can pass multiple variables at once: ```shell $ dbt run --vars '{event_type: signup, region: us}' ``` If only one variable is being set, the brackets are optional: ```shell $ dbt run --vars 'event_type: signup' ``` The `--vars` argument accepts a YAML dictionary as a string on the command line. YAML is convenient because it does not require strict quoting as with JSON. Both of the following are valid and equivalent: ```shell $ dbt run --vars '{"key": "value", "date": 20180101}' $ dbt run --vars '{key: value, date: 20180101}' ``` Variables defined using `--var`, override values defined in `dbt_project.yml`. This makes `--vars` useful for temporarily overriding configuration without changing your committed project files. For the complete order of precedence (including package-scoped variables and default values defined in `var()`), see [Variable precedence](https://docs.getdbt.com/docs/build/project-variables.md#variable-precedence). You can find more information on defining dictionaries with YAML [here](https://github.com/Animosity/CraftIRC/wiki/Complete-idiot%27s-introduction-to-yaml). ##### Variable precedence (Applies to dbt v1.12 and later) Variables defined with the `--vars` command line argument override variables defined in `vars.yml` or `dbt_project.yml`. They are globally scoped and accessible to the root project and all installed packages. The order of precedence for variable declaration is as follows (highest priority first): 1. The variables defined on the command line with `--vars` 2. The package-scoped variable declaration in the root `vars.yml` file (if present) 3. The global variable declaration in the root `vars.yml` file (if present) 4. The package-scoped variable declaration in the root `dbt_project.yml` file 5. The global variable declaration in the root `dbt_project.yml` file 6. If this node is defined in a package: variable declarations in that package's `dbt_project.yml` file 7. The variable's default argument (if one is provided) `vars.yml` is parsed *before* `dbt_project.yml`, so variables defined in `vars.yml` are available when `dbt_project.yml` is parsed. If `vars.yml` is empty or doesn't exist, variables from `dbt_project.yml` are used instead. When any variable (in `vars.yml`, `dbt_project.yml`, or from the command line) changes, partial parsing is disabled for the next run. If dbt is unable to find a definition for a variable after checking all possible variable declaration places, then a compilation error will be raised. **Note:** Variable scope is based on the node ultimately using that variable. Imagine the case where a model defined in the root project is calling a macro defined in an installed package. That macro, in turn, uses the value of a variable. The variable will be resolved based on the *root project's* scope, rather than the package's scope. --- ### Python models Note that only specific data platforms support `dbt-py` models. Check the [platform configuration pages](https://docs.getdbt.com/reference/resource-configs.md) to confirm if Python models are supported. Python models for Snowflake, BigQuery, and Databricks are supported in [dbt v2](https://docs.getdbt.com/docs/introduction.md). Please refer to the [supported features](https://docs.getdbt.com/docs/dbt/supported-features.md) page to learn more about dbt v2. We encourage you to: * Read [the original discussion](https://github.com/dbt-labs/dbt/discussions/5261) that proposed this feature. * Share your thoughts and ideas on [next steps for Python models](https://github.com/dbt-labs/dbt/discussions/5742). * Join the **#dbt-python-models** channel in the [dbt Community Slack](https://www.getdbt.com/community/join-the-community/). #### Overview dbt Python (`dbt-py`) models can help you solve use cases that can't be solved with SQL. You can perform analyses using tools available in the open-source Python ecosystem, including state-of-the-art packages for data science and statistics. Before, you would have needed separate infrastructure and orchestration to run Python transformations in production. Python transformations defined in dbt are models in your project with all the same capabilities around testing, documentation, and lineage. models/my\_python\_model.py ```python import ... def model(dbt, session): my_sql_model_df = dbt.ref("my_sql_model") final_df = ... # stuff you can't write in SQL! return final_df ``` models/config.yml ```yml models: - name: my_python_model # Document within the same codebase description: My transformation written in Python # Configure in ways that feel intuitive and familiar config: materialized: table tags: ['python'] # Test the results of my Python transformation columns: - name: id # Standard validation for 'grain' of Python results data_tests: - unique - not_null data_tests: # Write your own validation logic (in SQL) for Python results - custom_generic_test ``` [![SQL + Python, together at last](/img/docs/building-a-dbt-project/building-models/python-models/python-model-dag.png?v=2 "SQL + Python, together at last")](#)SQL + Python, together at last The prerequisites for dbt Python models include using an adapter for a data platform that supports a fully featured Python runtime when using dbt v1 or dbt v2 engine. In a dbt Python model, all Python code is executed remotely on the platform. None of it is run by dbt locally. We believe in clearly separating *model definition* from *model execution*. In this and many other ways, you'll find that dbt's approach to Python models mirrors its longstanding approach to modeling data in SQL. We've written this guide assuming that you have some familiarity with dbt. If you've never before written a dbt model, we encourage you to start by first reading [dbt Models](https://docs.getdbt.com/docs/build/models.md). Throughout, we'll be drawing connections between Python models and SQL models, as well as making clear their differences. ##### What is a Python model? A dbt Python model is a function that reads in dbt sources or other models, applies a series of transformations, and returns a transformed dataset. DataFrame operations define the starting points, the end state, and each step along the way. This is similar to the role of CTEs in dbt SQL models. We use CTEs to pull in upstream datasets, define (and name) a series of meaningful transformations, and end with a final `select` statement. You can run the compiled version of a dbt SQL model to see the data included in the resulting view or table. When you `dbt run`, dbt wraps that query in `create view`, `create table`, or more complex DDL to save its results in the database. Instead of a final `select` statement, each Python model returns a final DataFrame. Each DataFrame operation is "lazily evaluated." In development, you can preview its data, using methods like `.show()` or `.head()`. When you run a Python model, the full result of the final DataFrame will be saved as a table in your data warehouse. dbt Python models have access to almost all of the same configuration options as SQL models. You can test and document them, add `tags` and `meta` properties, and grant access to their results to other users. You can select them by their name, file path, configurations, whether they are upstream or downstream of another model, or if they have been modified compared to a previous project state. ##### Defining a Python model Each Python model lives in a `.py` file in your `models/` folder. It defines a function named **`model()`**, which takes two parameters: * **`dbt`**: A class compiled by dbt unique to each model, enables you to run your Python code in the context of your dbt project and DAG. * **`session`**: A class representing your data platform’s connection to the Python backend. The session is needed to read in tables as DataFrames, and to write DataFrames back to tables. In PySpark, by convention, the `SparkSession` is named `spark`, and available globally. For consistency across platforms, we always pass it into the `model` function as an explicit argument called `session`. The `model()` function must return a single DataFrame. On Snowpark (Snowflake), this can be a Snowpark or pandas DataFrame. On BigQuery this can be BigFrames, pandas or Spark datafame. Via PySpark (Databricks), this can be a Spark, pandas, or pandas-on-Spark DataFrame. For more information about choosing between pandas and native DataFrames, see [DataFrame API + syntax](#dataframe-api-and-syntax). When you `dbt run --select python_model`, dbt will prepare and pass in both arguments (`dbt` and `session`). All you have to do is define the function. This is how every single Python model should look: models/my\_python\_model.py ```python def model(dbt, session): ... return final_df ``` ##### Referencing other models Python models participate fully in dbt's directed acyclic graph (DAG) of transformations. Use the `dbt.ref()` method within a Python model to read data from other models (SQL or Python). If you want to read directly from a raw source table, use `dbt.source()`. These methods return DataFrames pointing to the upstream source, model, seed, or snapshot. models/my\_python\_model.py ```python def model(dbt, session): # DataFrame representing an upstream model upstream_model = dbt.ref("upstream_model_name") # DataFrame representing an upstream source upstream_source = dbt.source("upstream_source_name", "table_name") ... ``` Of course, you can `ref()` your Python model in downstream SQL models, too: models/downstream\_model.sql ```sql with upstream_python_model as ( select * from {{ ref('my_python_model') }} ), ... ``` caution Referencing [ephemeral](https://docs.getdbt.com/docs/build/materializations.md#ephemeral) models is currently not supported (see [feature request](https://github.com/dbt-labs/dbt/issues/7288)) From dbt version 1.8, Python models also support dynamic configurations within Python f-strings. This allows for more nuanced and dynamic model configurations directly within your Python code. For example: models/my\_python\_model.py ```python # Previously, attempting to access a configuration value like this would result in None print(f"{dbt.config.get('my_var')}") # Output before change: None # Now you can access the actual configuration value # Assuming 'my_var' is configured to 5 for the current model print(f"{dbt.config.get('my_var')}") # Output after change: 5 ``` This also means you can use `dbt.config.get()` within Python models to ensure that configuration values are effectively retrievable and usable within Python f-strings. #### Configuring Python models Just like SQL models, there are three ways to configure Python models: 1. In `dbt_project.yml`, where you can configure many models at once 2. In a dedicated `.yml` file, within the `models/` directory 3. Within the model's `.py` file, using the `dbt.config()` method Calling the `dbt.config()` method will set configurations for your model within your `.py` file, similar to the `{{ config() }}` macro in `.sql` model files: models/my\_python\_model.py ```python def model(dbt, session): # setting configuration dbt.config(materialized="table") ``` There's a limit to how complex you can get with the `dbt.config()` method. It accepts *only* literal values (strings, booleans, and numeric types) and dynamic configuration. Passing another function or a more complex data structure is not possible. The reason is that dbt statically analyzes the arguments to `config()` while parsing your model without executing your Python code. If you need to set a more complex configuration, we recommend you define it using the [`config` property](https://docs.getdbt.com/reference/resource-properties/config.md) in a properties YAML file. ###### Accessing project context dbt Python models don't use Jinja to render compiled code. Python models have limited access to global project contexts compared to SQL models. That context is made available from the `dbt` class, passed in as an argument to the `model()` function. Out of the box, the `dbt` class supports: * Returning DataFrames referencing the locations of other resources: `dbt.ref()` + `dbt.source()` * Accessing the database location of the current model: `dbt.this()` (also: `dbt.this.database`, `.schema`, `.identifier`) * Determining if the current model's run is incremental: `dbt.is_incremental` * Accessing custom values stored in `meta`: `dbt.config.meta_get()` It is possible to extend this context by "getting" them with `dbt.config.get()` after they are configured in the [model's config](https://docs.getdbt.com/reference/model-configs.md). The `dbt.config.get()` method supports dynamic access to configurations within Python models, enhancing flexibility in model logic. This includes inputs such as `var`, `env_var`, and `target`. If you want to use those values for the conditional logic in your model, we require setting them through a dedicated properties YAML file config: models/config.yml ```yml models: - name: my_python_model config: materialized: table target_name: "{{ target.name }}" specific_var: "{{ var('SPECIFIC_VAR') }}" specific_env_var: "{{ env_var('SPECIFIC_ENV_VAR') }}" ``` Then, within the model's Python code, use the `dbt.config.get()` function to *access* values of configurations that have been set: models/my\_python\_model.py ```python def model(dbt, session): target_name = dbt.config.get("target_name") specific_var = dbt.config.get("specific_var") specific_env_var = dbt.config.get("specific_env_var") orders_df = dbt.ref("fct_orders") # limit data in dev if target_name == "dev": orders_df = orders_df.limit(500) ``` ###### Accessing custom meta values To store custom values, use the [`meta` config](https://docs.getdbt.com/reference/resource-configs/meta.md). For example, if you have a model named `my_python_model` and you want to store custom values, you can do the following: models/schema.yml ```yml models: - name: my_python_model config: meta: custom_value: "111" another_value: "abc" ``` Then access them in your Python model using the `dbt.config.meta_get()` method: models/my\_python\_model.py ```python def model(dbt, session): # Access custom values stored in meta directly custom_value = dbt.config.meta_get("custom_value") another_value = dbt.config.meta_get("another_value") # Use your custom values in your model logic orders_df = dbt.ref("fct_orders") ... ``` Alternative approach You can also retrieve meta values using `dbt.config.get("meta")`, which returns the entire meta dictionary. When using this approach, handle the case where `meta` might not be configured: ```python custom_value = dbt.config.get("meta", {}).get("custom_value") ``` ###### Dynamic configurations In addition to the existing methods of configuring Python models, you also have dynamic access to configuration values set with `dbt.config()` within Python models using f-strings. This increases the possibilities for custom logic and configuration management. models/my\_python\_model.py ```python def model(dbt, session): dbt.config(materialized="table") # Dynamic configuration access within Python f-strings, # which allows for real-time retrieval and use of configuration values. # Assuming 'my_var' is set to 5, this will print: Dynamic config value: 5 print(f"Dynamic config value: {dbt.config.get('my_var')}") ``` ##### Materializations Python models support these materializations: * `table` (default) * `incremental` Incremental Python models support all the same [incremental strategies](https://docs.getdbt.com/docs/build/incremental-strategy.md) as their SQL counterparts. The specific strategies supported depend on your adapter. As an example, incremental models are supported on BigQuery with Dataproc for the `merge` incremental strategy; the `insert_overwrite` strategy is not yet supported. Python models can't be materialized as `view` or `ephemeral`. Python isn't supported for non-model resource types (like tests and snapshots). For incremental models, like SQL models, you need to filter incoming tables to only new rows of data: ##### Snowpark models/my\_python\_model.py ```python import snowflake.snowpark.functions as F def model(dbt, session): dbt.config(materialized = "incremental") df = dbt.ref("upstream_table") if dbt.is_incremental: # only new rows compared to max in current table max_from_this = f"select max(updated_at) from {dbt.this}" df = df.filter(df.updated_at >= session.sql(max_from_this).collect()[0][0]) # or only rows from the past 3 days df = df.filter(df.updated_at >= F.dateadd("day", F.lit(-3), F.current_timestamp())) ... return df ``` ##### BigQuery DataFrames models/my\_python\_model.py ```python import datetime def model(dbt, session): dbt.config(materialized = "incremental") bdf = dbt.ref("upstream_table") if dbt.is_incremental: # only new rows compared to max in current table max_from_this = f"select max(updated_at) from {dbt.this}" bdf = bdf[bdf['updated_at'] >= bpd.read_gbq(max_from_this).values[0][0]] # or only rows from the past 3 days bdf = bdf[bdf['updated_at'] >= datetime.date.today() - datetime.timedelta(days=3)] ... return bdf ``` ##### PySpark models/my\_python\_model.py ```python import pyspark.sql.functions as F def model(dbt, session): dbt.config(materialized = "incremental") df = dbt.ref("upstream_table") if dbt.is_incremental: # only new rows compared to max in current table max_from_this = f"select max(updated_at) from {dbt.this}" df = df.filter(df.updated_at >= session.sql(max_from_this).collect()[0][0]) # or only rows from the past 3 days df = df.filter(df.updated_at >= F.date_add(F.current_timestamp(), F.lit(-3))) ... return df ``` #### Python-specific functionality ##### Defining functions In addition to defining a `model` function, the Python model can import other functions or define its own. Here's an example on Snowpark, defining a custom `add_one` function: models/my\_python\_model.py ```python def add_one(x): return x + 1 def model(dbt, session): dbt.config(materialized="table") temps_df = dbt.ref("temperatures") # warm things up just a little df = temps_df.withColumn("degree_plus_one", add_one(temps_df["degree"])) return df ``` Currently, Python functions defined in one dbt model can't be imported and reused in other models. Refer to [Code reuse](#code-reuse) for the potential patterns being considered. ##### Using PyPI packages You can also define functions that depend on third-party packages so long as those packages are installed and available to the Python runtime on your data platform. In this example, we use the `holidays` package to determine if a given date is a holiday in France. The code below uses the pandas API for simplicity and consistency across platforms. The exact syntax, and the need to refactor for multi-node processing, still vary. ##### Snowpark models/my\_python\_model.py ```python import holidays def is_holiday(date_col): # Chez Jaffle french_holidays = holidays.France() is_holiday = (date_col in french_holidays) return is_holiday def model(dbt, session): dbt.config( materialized = "table", packages = ["holidays"] ) orders_df = dbt.ref("stg_orders") df = orders_df.to_pandas() # apply our function # (columns need to be in uppercase on Snowpark) df["IS_HOLIDAY"] = df["ORDER_DATE"].apply(is_holiday) df["ORDER_DATE"].dt.tz_localize('UTC') # convert from Number/Long to tz-aware Datetime # return final dataset (Pandas DataFrame) return df ``` ##### BigQuery DataFrames models/my\_python\_model.py ```python import holidays def model(dbt, session): dbt.config(submission_method="bigframes") data = { 'id': [0, 1, 2], 'name': ['Brian Davis', 'Isaac Smith', 'Marie White'], 'birthday': ['2024-03-14', '2024-01-01', '2024-11-07'] } bdf = bpd.DataFrame(data) bdf['birthday'] = bpd.to_datetime(bdf['birthday']) bdf['birthday'] = bdf['birthday'].dt.date us_holidays = holidays.US(years=2024) return bdf[bdf['birthday'].isin(us_holidays)] ``` ##### PySpark models/my\_python\_model.py ```python import holidays def is_holiday(date_col): # Chez Jaffle french_holidays = holidays.France() is_holiday = (date_col in french_holidays) return is_holiday def model(dbt, session): dbt.config( materialized = "table", packages = ["holidays"] ) orders_df = dbt.ref("stg_orders") df = orders_df.to_pandas_on_spark() # Spark 3.2+ # df = orders_df.toPandas() in earlier versions # apply our function df["is_holiday"] = df["order_date"].apply(is_holiday) # convert back to PySpark df = df.to_spark() # Spark 3.2+ # df = session.createDataFrame(df) in earlier versions # return final dataset (PySpark DataFrame) return df ``` ###### Configuring packages We encourage you to configure required packages and versions so dbt can track them in project metadata. This configuration is required for the implementation on some platforms. If you need specific versions of packages, specify them. models/my\_python\_model.py ```python def model(dbt, session): dbt.config( packages = ["numpy==1.23.1", "scikit-learn"] ) ``` models/config.yml ```yml models: - name: my_python_model config: packages: - "numpy==1.23.1" - scikit-learn ``` ###### User-defined functions (UDFs) You can use the `@udf` decorator or `udf` function to define an "anonymous" function and call it within your `model` function's DataFrame transformation. This is a typical pattern for applying more complex functions as DataFrame operations, especially if those functions require inputs from third-party packages. * [Snowpark Python: Creating UDFs](https://docs.snowflake.com/en/developer-guide/snowpark/python/creating-udfs.html) * [BigQuery DataFrames UDFs](https://cloud.google.com/bigquery/docs/use-bigquery-dataframes#custom-python-functions) * [PySpark functions: udf](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.udf.html) tip You can also define [SQL or Python UDFs](https://docs.getdbt.com/docs/build/udfs.md) as first-class resources under `/functions` with a matching `YAML` file. dbt builds them as part of the DAG, and you reference them from SQL using `{{ function('my_udf') }}`. These UDFs are reusable across tools (BI, notebooks, SQL clients) because they live in your warehouse. ##### Snowpark models/my\_python\_model.py ```python import snowflake.snowpark.types as T import snowflake.snowpark.functions as F import numpy def register_udf_add_random(): add_random = F.udf( # use 'lambda' syntax, for simple functional behavior lambda x: x + numpy.random.normal(), return_type=T.FloatType(), input_types=[T.FloatType()] ) return add_random def model(dbt, session): dbt.config( materialized = "table", packages = ["numpy"] ) temps_df = dbt.ref("temperatures") add_random = register_udf_add_random() # warm things up, who knows by how much df = temps_df.withColumn("degree_plus_random", add_random("degree")) return df ``` **Note:** Due to a Snowpark limitation, it is not currently possible to register complex named UDFs within stored procedures and, therefore, dbt Python models. We are looking to add native support for Python UDFs as a project/DAG resource type in a future release. For the time being, if you want to create a "vectorized" Python UDF via the Batch API, we recommend either: * Writing [`create function`](https://docs.snowflake.com/en/developer-guide/udf/python/udf-python-batch.html) inside a SQL macro, to run as a hook or run-operation * [Registering from a staged file](https://docs.snowflake.com/en/developer-guide/snowpark/python/creating-udfs#creating-a-udf-from-a-python-source-file) within your Python model code ##### BigQuery DataFrames models/my\_python\_model.py ```python def model(dbt, session): dbt.config(submission_method="bigframes") # You can also use @bpd.udf @bpd.remote_function(dataset='jialuo_test_us') def my_func(x: int) -> int: return x * 1100 data = {"int": [1, 2], "str": ['a', 'b']} bdf = bpd.DataFrame(data=data) bdf['int'] = bdf['int'].apply(my_func) return bdf ``` ##### PySpark models/my\_python\_model.py ```python import pyspark.sql.types as T import pyspark.sql.functions as F import numpy # use a 'decorator' for more readable code @F.udf(returnType=T.DoubleType()) def add_random(x): random_number = numpy.random.normal() return x + random_number def model(dbt, session): dbt.config( materialized = "table", packages = ["numpy"] ) temps_df = dbt.ref("temperatures") # warm things up, who knows by how much df = temps_df.withColumn("degree_plus_random", add_random("degree")) return df ``` ###### Code reuse To re-use a Python function across multiple dbt models, you can define [Python UDFs](https://docs.getdbt.com/docs/build/udfs.md) under `/functions` with a matching YAML file. These UDFs live in your warehouse and can be reused across tools (BI, notebooks, SQL clients). In the future, we're considering also adding support for Private Python packages. In addition to importing reusable functions from public PyPI packages, many data platforms support uploading custom Python assets and registering them as packages. The upload process looks different across platforms, but your code’s actual `import` looks the same. ❓ dbt questions * How can dbt help users when uploading or initializing private Python assets? Is this a new form of `dbt deps`? * How can dbt support users who want to test custom functions? If defined as UDFs: "unit testing" in the database? If "pure" functions in packages: encourage adoption of `pytest`? 💬 Discussion: ["Python models: package, artifact/object storage, and UDF management in dbt"](https://github.com/dbt-labs/dbt/discussions/5741) ##### DataFrame API and syntax Over the past decade, most people writing [data transformations](https://www.getdbt.com/analytics-engineering/transformation/) in Python have adopted DataFrame as their common abstraction. dbt follows this convention by returning `ref()` and `source()` as DataFrames, and it expects all Python models to return a DataFrame. A DataFrame is a two-dimensional data structure (rows and columns). It supports convenient methods for transforming that data and creating new columns from calculations performed on existing columns. It also offers convenient ways for previewing data while developing locally or in a notebook. That's about where the agreement ends. There are numerous frameworks with their own syntaxes and APIs for DataFrames. The [pandas](https://pandas.pydata.org/docs/) library offered one of the original DataFrame APIs, and its syntax is the most common to learn for new data professionals. Most newer DataFrame APIs are compatible with pandas-style syntax, though few can offer perfect interoperability. This is true for BigQuery DataFrames, Snowpark, and PySpark, which have their own DataFrame APIs. When developing a Python model, you will find yourself asking these questions: **Why pandas?** — It's the most common API for DataFrames. It makes it easy to explore sampled data and develop transformations locally. You can “promote” your code as-is into dbt models and run it in production for small datasets. **Why *not* pandas?** — Performance. pandas runs "single-node" transformations, which cannot benefit from the parallelism and distributed computing offered by modern data warehouses. This quickly becomes a problem as you operate on larger datasets. Some data platforms support optimizations for code written using pandas DataFrame API, preventing the need for major refactors. For example, [pandas on PySpark](https://spark.apache.org/docs/latest/api/python/getting_started/quickstart_ps.html) offers support for 95% of pandas functionality, using the same API while still leveraging parallel processing. ❓ dbt questions * When developing a new dbt Python model, should we recommend pandas-style syntax for rapid iteration and then refactor? * Which open source libraries provide compelling abstractions across different data engines and vendor-specific APIs? * Should dbt attempt to play a longer-term role in standardizing across them? 💬 Discussion: ["Python models: the pandas problem (and a possible solution)"](https://github.com/dbt-labs/dbt/discussions/5738) #### Limitations Python models have capabilities that SQL models do not. They also have some drawbacks compared to SQL models: * **Time and cost.** Python models are slower to run than SQL models, and the cloud resources that run them can be more expensive. Running Python requires more general-purpose compute. That compute might sometimes live on a separate service or architecture from your SQL models. **However:** We believe that deploying Python models via dbt—with unified lineage, testing, and documentation—is, from a human standpoint, **dramatically** faster and cheaper. By comparison, spinning up separate infrastructure to orchestrate Python transformations in production and different tooling to integrate with dbt is much more time-consuming and expensive. * **Syntax differences** are even more pronounced. Over the years, dbt has done a lot, via dispatch patterns and packages such as `dbt_utils`, to abstract over differences in SQL dialects across popular data warehouses. Python offers a **much** wider field of play. If there are five ways to do something in SQL, there are 500 ways to write it in Python, all with varying performance and adherence to standards. Those options can be overwhelming. As the maintainers of dbt, we will be learning from state-of-the-art projects tackling this problem and sharing guidance as we develop it. * **These capabilities are very new.** As data warehouses develop new features, we expect them to offer cheaper, faster, and more intuitive mechanisms for deploying Python transformations. **We reserve the right to change the underlying implementation for executing Python models in future releases.** Our commitment to you is around the code in your model `.py` files, following the documented capabilities and guidance we're providing here. * **Lack of `print()` support.** The data platform runs and compiles your Python model without dbt's oversight. This means it doesn't display the output of commands such as Python's built-in [`print()`](https://docs.python.org/3/library/functions.html#print) function in dbt's logs. *  Alternatives to using print() in Python models The following explains other methods you can use for debugging, such as writing messages to a dataframe column: * Using platform logs: Use your data platform's logs to debug your Python models. * Return logs as a dataframe: Create a dataframe containing your logs and build it into the warehouse. * Develop locally with DuckDB: Test and debug your models locally using DuckDB before deploying them. Here's an example of debugging in a Python model: ```python def model(dbt, session): dbt.config( materialized = "table" ) df = dbt.ref("my_source_table").df() # One option for debugging: write messages to temporary table column # Pros: visibility # Cons: won't work if table isn't building for some reason msg = "something" df["debugging"] = f"My debug message here: {msg}" return df ``` As a general rule, if there's a transformation you could write equally well in SQL or Python, we believe that well-written SQL is preferable: it's more accessible to a greater number of colleagues, and it's easier to write code that's performant at scale. If there's a transformation you *can't* write in SQL, or where ten lines of elegant and well-annotated Python could save you 1000 lines of hard-to-read Jinja-SQL, Python is the way to go. --- ### Ratio metrics Ratio metrics allow you to create a ratio between two metrics. You specify a numerator and a denominator metric. You can optionally apply filters, names, and aliases to both the numerator and denominator when computing the metric. The parameters for ratio metrics are as follows: (Applies to dbt v1.12 and later) | Parameter | Description | Required | Type | | ------------- | --------------------------------------------------------------------------------------------------------------------------------------- | -------- | -------------- | | `name` | The name of the metric. | Required | String | | `description` | The description of the metric. | Optional | String | | `type` | The type of the metric (cumulative, derived, ratio, or simple). | Required | String | | `label` | Defines the display value in downstream tools. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | String | | `numerator` | The name of the metric used for the numerator. Can be a string (metric name) or a dict with `name`, `filter`, and `alias` properties. | Required | String or Dict | | `denominator` | The name of the metric used for the denominator. Can be a string (metric name) or a dict with `name`, `filter`, and `alias` properties. | Required | String or Dict | ###### Numerator/Denominator dictionary properties The following properties are available for the numerator and denominator dictionary: | Property | Description | Required | Type | | -------- | ------------------------------ | -------- | ------ | | `name` | Name of the metric. | Required | String | | `filter` | Filter to apply to the metric. | Optional | String | | `alias` | Alias for the metric. | Optional | String | The complete specification for ratio metrics is as follows: (Applies to dbt v1.12 and later) models/file\_name.yml ```yaml models: - name: file_name semantic_model: - enabled: true # required - name: my_semantic_model ... rest of config... metrics: - name: my_advanced_ratio_metric type: ratio numerator: name: my_simple_metric filter: "{{ Dimension('my_primary_entity__my_categorical_dimension_column') }} > 10" alias: joel_loves_data denominator: name: my_simple_metric_that_uses_the_other_time_dimension_but_is_also_from_another_semantic_model filter: "{{ Dimension('my_primary_entity__my_categorical_dimension_column') }} < 10" alias: joel_hates_data ``` For advanced data modeling, you can use `fill_nulls_with` and `join_to_timespine` to [set null metric values to zero](https://docs.getdbt.com/docs/build/fill-nulls-advanced.md), ensuring numeric values for every data row. #### Ratio metrics example These examples demonstrate how to create ratio metrics in your model. They cover basic and advanced use cases, including applying filters to the numerator and denominator metrics. ###### Example 1 This example is a basic ratio metric that calculates the ratio of food orders to total orders: (Applies to dbt v1.12 and later) models/file\_name.yml ```yaml metrics: - name: food_order_pct description: "The food order count as a ratio of the total order count" label: Food order ratio type: ratio numerator: food_orders denominator: orders ``` ###### Example 2 This example is a ratio metric that calculates the ratio of food orders to total orders, with a filter and alias applied to the numerator. Note that in order to add these attributes, you'll need to use an explicit key for the name attribute too. (Applies to dbt v1.12 and later) models/file\_name.yml ```yaml metrics: - name: food_order_pct description: "The food order count as a ratio of the total order count" label: Food order ratio by location type: ratio numerator: name: food_orders filter: location = 'New York' alias: ny_food_orders denominator: name: orders filter: location = 'New York' alias: ny_orders ``` #### Ratio metrics using different semantic models The system will simplify and turn the numerator and denominator into a ratio metric from different semantic models by computing their values in sub-queries. It will then join the result set based on common dimensions to calculate the final ratio. Here's an example of the SQL generated for such a ratio metric. ```sql select subq_15577.metric_time as metric_time, cast(subq_15577.mql_queries_created_test as double) / cast(nullif(subq_15582.distinct_query_users, 0) as double) as mql_queries_per_active_user from ( select metric_time, sum(mql_queries_created_test) as mql_queries_created_test from ( select cast(query_created_at as date) as metric_time, case when query_status in ('PENDING','MODE') then 1 else 0 end as mql_queries_created_test from prod_dbt.mql_query_base mql_queries_test_src_2552 ) subq_15576 group by metric_time ) subq_15577 inner join ( select metric_time, count(distinct distinct_query_users) as distinct_query_users from ( select cast(query_created_at as date) as metric_time, case when query_status in ('MODE','PENDING') then email else null end as distinct_query_users from prod_dbt.mql_query_base mql_queries_src_2585 ) subq_15581 group by metric_time ) subq_15582 on ( ( subq_15577.metric_time = subq_15582.metric_time ) or ( ( subq_15577.metric_time is null ) and ( subq_15582.metric_time is null ) ) ) ``` #### Add filter Users can define constraints on input metrics for a ratio metric by applying a filter directly to the input metric, like so: (Applies to dbt v1.12 and later) models/file\_name.yml ```yaml metrics: - name: frequent_purchaser_ratio description: Fraction of active users who qualify as frequent purchasers type: ratio numerator: name: distinct_purchasers filter: | "{{ Dimension('customer__is_frequent_purchaser') }}" alias: frequent_purchasers denominator: name: distinct_purchasers ``` Note the `filter` and `alias` parameters for the metric referenced in the numerator. * Use the `filter` parameter to apply a filter to the metric it's attached to. * The `alias` parameter is used to avoid naming conflicts in the rendered SQL queries when the same metric is used with different filters. * If there are no naming conflicts, the `alias` parameter can be left out. #### Related docs * [Fill null values for simple, derived, or ratio metrics](https://docs.getdbt.com/docs/build/fill-nulls-advanced.md) --- ### Release tracks in dbt platform dbt platform Since May 2024, new capabilities in the dbt framework are delivered continuously to dbt. Your projects and environments are upgraded automatically on a cadence that you choose, depending on your dbt plan. Previously, customers would pin to a minor version of dbt v1, and receive only patch updates during that specific version's active support period. Release tracks ensure that your project stays up-to-date with the modern capabilities of dbt and recent versions of dbt v1. This will require you to make one final update to your current jobs and environments. When that's done, you'll never have to think about managing, coordinating, or upgrading dbt versions again. By moving your environments and jobs to release tracks you can get all the functionality in dbt as soon as it's ready. On the **v1 Latest** release track, this includes access to features *before* they're available in final releases of dbt v1 OSS. #### Which release tracks are available? ##### dbt v2 release tracks The dbt platform has four dbt v2 release tracks available to give you control over stability vs. velocity. *`v2 Stable` is the default for all new dbt v2-powered projects.* | Release track | Cadence | Description | [Plan availability](https://www.getdbt.com/pricing) | API value | | ----------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------- | ----------------- | | **v2 Nightly** | Nightly | The latest nightly build. Includes early access to new features. | All plans | `fusion-nightly` | | **v2 Stable** (default) | Weekly | A weekly release that balances stability and feature access.<br /> | All plans | `fusion-stable` | | **v2 Extended** | Monthly | The previous month's final v2 Stable release. Designed for those who want maximum stability and additional testing time. | Enterprise, Enterprise+ | `fusion-extended` | | **v2 Fallback** | Monthly | The previous month's v2 Extended release. Emergency rollback option for account admins. | Enterprise+ | `fusion-fallback` | ##### dbt v1 release tracks | Release track | Description | Plan availability | API value | | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | ------------ | | **v1 Latest** | Provides a continuous release of the latest functionality in the dbt platform .<br /><br />Includes early access to new features of the dbt framework before they're available in dbt v1. | All plans | `latest` | | **v1 Compatible** | Provides a monthly release aligned with the most recent open source versions of dbt v1 and adapters, plus functionality exclusively available in the dbt platform.<br /><br />See [v1 Compatible track changelog](https://docs.getdbt.com/docs/dbt-versions/compatible-track-changelog.md) for more information. | Starter, Enterprise, Enterprise+ | `compatible` | | **v1 Extended** | The previous month's **v1 Compatible** release. | Enterprise, Enterprise+ | `extended` | | **v1 Fallback** | The previous month's **v1 Extended** release. | Enterprise+ | `fallback` | To configure an environment in the [dbt Admin API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md) or [Terraform](https://registry.terraform.io/providers/dbt-labs/dbtcloud/latest) to use a release track, set `dbt_version` to the release track name: * `fusion-nightly` * `fusion-stable` (formerly `latest-fusion`) * `fusion-extended` * `fusion-fallback` * `latest` * `compatible` * `extended` #### Which release track should I choose? ##### Choosing a dbt v2 release track Choose **v2 Stable** (the default for new projects) for a weekly release cadence that balances stability and feature access. This is the recommended starting point for most customers migrating to dbt v2. Choose **v2 Nightly** if you want the latest features and are comfortable with a daily release cadence. This track ideal for teams prioritizing the most recent features. Choose **v2 Extended** if your team needs maximum stability and more time to test before updates roll out. This track generally lags v2 Stable by one month and is available for Enterprise and Enterprise+ accounts. Choose **v2 Fallback** as an emergency measure only. This is a temporary rollback option for Enterprise+ accounts, not for ongoing use. Account admins can set the org-wide default release track for new projects via a toggle in **Account Settings**. Individual project owners can then override that default per environment. ##### Choosing a dbt v1 release track Choose the **v1 Latest** release track to continuously receive new features, fixes, performance improvements — latest & greatest dbt. This is the default for all customers on dbt. Choose the **v1 Compatible** and **v1 Extended** release tracks if you need a less-frequent release cadence, the ability to test new dbt releases before they go live in production, and/or ongoing compatibility with the latest open source releases of dbt v1. ##### Using the v2 Fallback release track The **v2 Fallback** release track provides an emergency rollback option for account admins if you suspect a regression in the "v2 Extended" track. This is available only for dbt v2 in Enterprise+ accounts. 1. Go to **Account settings** 2. Click the **Revert to Fallback** release track button in the dbt platform interface. This is not accessible through environment settings. 3. Fill in the details in the **Revert to Fallback** pop-up to confirm and share information with dbt Support. Switching to **v2 Fallback** alerts the dbt Support team, who may reach out to help resolve the issue. This track is meant only as a temporary safety option to unblock you and not for ongoing use. Return to "v2 Extended" or "v2 Stable" (dbt v2) / "v1 Compatible" (dbt v1) once the issue is resolved. [![v2 Fallback release track button in dbt platform](/img/docs/dbt-versions/rollback.png?v=2 "v2 Fallback release track button in dbt platform")](#)v2 Fallback release track button in dbt platform [![v2 Fallback release track popup in dbt platform](/img/docs/dbt-versions/rollback-popup.png?v=2 "v2 Fallback release track popup in dbt platform")](#)v2 Fallback release track popup in dbt platform ##### Common architectures ###### dbt v2 track architectures **Default** — All plans * Prioritize a weekly release cadence that balances stability and new features * Leave all environments on the **v2 Stable** release track (default for new projects at GA) **Velocity-focused** — All plans * Prioritize immediate access to the latest v2 features and fixes * Configure all environments to use the **v2 Nightly** release track **Cautious** — Enterprise, Enterprise+ * Prioritize maximum stability and additional testing time * Configure production environments to use **v2 Extended** (one month behind v2 Stable) * Configure development environments to use **v2 Stable** so developers get access to upcoming changes before they reach production ###### dbt v1 track architectures **Default** — Majority of customers on all plans * Prioritize immediate access to fixes and features * Leave all environments on the **v1 Latest** release track (default configuration) **Hybrid** — Starter, Enterprise, Enterprise+ * Prioritize ongoing compatibility between dbt and dbt v1 for development & deployment using both products in the same dbt projects * Configure all environments to use the **v1 Compatible** release track * Understand that new features will not be available until they are first released in dbt v1 OSS (several months after the **v1 Latest** release track) **Cautious** — Enterprise, Enterprise+, Business Critical * Prioritize "bake in" time for new features & fixes * Configure development & test environments to use the **v1 Compatible** release track * Configure pre-production & production environments to use the **v1 Extended** release track * Understand that new features will not be available until *a month after* they are first released in dbt v1 OSS and the v1 Compatible track. Developers (on **v1 Compatible**) will get access to new features before they can leverage those capabilities in production (on **v1 Extended**), and must be mindful of the additional delay. **Virtual Private dbt or Single Tenant** * Changes to all release tracks roll out as part of dbt instance upgrades once per week #### Upgrading from older versions ##### How to upgrade If you regularly develop your dbt project in dbt, and you're still running on a legacy version of dbt v1, dbt Labs recommends that you try upgrading your project in a development environment. [Override your dbt version in development](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md#override-dbt-version). Then, launch the Studio IDE or dbt platform CLI and do your development work as usual. Everything should work as you expect. If you do see something unexpected or surprising, revert back to the previous version and record the differences you observed. [Contact dbt support](https://docs.getdbt.com/docs/dbt-support.md#dbt-cloud-support) with your findings for a more detailed investigation. Next, we recommend that you try upgrading your project’s [deployment environment](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md#environments). If your project has a [staging deployment environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md#staging-environment), upgrade and try working with it for a few days before you proceed with upgrading the production environment. If your organization has multiple dbt projects, we recommend starting your upgrade with projects that are smaller, newer, or more familiar for your team. That way, if you do encounter any issues, it'll be easier and faster to troubleshoot those before proceeding to upgrade larger or more complex projects. ##### Considerations To learn more about how dbt Labs deploys stable dbt upgrades in a safe manner to dbt, we recommend that you read our blog post: [How we're making sure you can confidently switch to the "Latest" release track in dbt](https://docs.getdbt.com/blog/latest-dbt-stability). If you're running dbt version 1.6 or older, please know that your version of dbt v1 has reached [end-of-life (EOL)](https://docs.getdbt.com/docs/dbt-versions.md#eol-version-support) and is no longer supported. We strongly recommend that you update to a newer version as soon as reasonably possible. dbt Labs has extended the critical support period of dbt v1.7 for dbt Enterprise-tier customers to March 2025. At that point, we will be encouraging all customers to select a Release Track for ongoing updates in dbt.  I'm using an older version of dbt in the dbt platform. What should I do? What happens if I do nothing? If you're running dbt version v1.6 or older, please know that your version of dbt v1 has reached [end-of-life (EOL)](https://docs.getdbt.com/docs/dbt-versions.md#eol-version-support) and is no longer supported. We strongly recommend that you update to a newer version as soon as reasonably possible. dbt Labs has extended the "Critical Support" period of dbt v1.7 for dbt Enterprise-tier customers while we work through the migration with those customers to Release Tracks. In the meantime, this means that v1.7 will continue to be accessible in dbt for Enterprise customers, jobs and environments on v1.7 for those customers will not be automatically migrated to "Latest," and dbt Labs will continue to fix critical bugs and security issues. Starting in October 2024, dbt accounts on the Developer and Starter (formerly Teams) plans have been migrated to release tracks from older dbt v1 versions. If your account was migrated to the **v1 Latest** release track and you notice new failures in scheduled jobs, please [contact dbt support](https://docs.getdbt.com/docs/dbt-support.md#dbt-cloud-support) to report the problem or request an extension.  What are other known issues when upgrading from older v1 versions? If you are upgrading from a very old unsupported version of dbt v1, you may run into one of these edge cases after the upgrade to a newer version: * \[v1.1] Customers on BigQuery should be aware that dbt sets a default [per-model timeout](https://docs.getdbt.com/docs/local/connect-data-platform/bigquery-setup.md#job_execution_timeout_seconds) of 5 minutes. You may override this config in your connection details. Older versions of dbt (including v1.0) did not appropriately respect this timeout configuration. * \[v1.3] Customers with non-dbt `.py` files defined within their project directories, such as `models/`. Since v1.3, dbt expects these files be valid [Python models](https://docs.getdbt.com/docs/build/python-models.md). The customer needs to move these files out of their `models/` directory, or ignore them via `.dbtignore` * \[v1.5] Customers who have `--m` in their job definitions, instead of `-m` or `--models`. This autocompletion (`--m[odels]` for `--models`) has never been officially documented or supported. It was an implicit behavior of argparse (CLI library used in dbt-core v1.0-1.4) that is not supported by `click` (the CLI library used in dbt-core since v1.5+). * \[v1.5] Empty invalid `tests` config start raising a validation error]\(/docs/dbt-versions/dbt-upgrade/Older%20versions/upgrading-to-v1.5). Replace empty `tests` config with `tests: []` or remove it altogether. * \[v1.6] Performance optimization to `load_result` means you cannot call it on the same query result multiple times. Instead, save it to a local variable once, and reuse that variable (context: [dbt-labs/dbt#7371](https://github.com/dbt-labs/dbt/pull/7371) You should [contact dbt support](https://docs.getdbt.com/docs/dbt-support.md#dbt-cloud-support) to request an extension, during which you will need to make those updates.  I see that my account was migrated to v1 Latest. What should I do For the vast majority of customers, there is no further action needed. If you see new failures in your scheduled jobs now that they are running on a newer version of dbt, you may need to update your project code to account for one of the edge cases described on this page. You should [contact dbt support](https://docs.getdbt.com/docs/dbt-support.md#dbt-cloud-support) to request an extension, during which you will need to make those updates.  What about breaking changes to packages (maintained by dbt Labs or by others)? When we talk about *latest version*, we’re referring to the underlying runtime for dbt, not the versions of packages you’re installing. Our continuous release for dbt includes testing against several popular dbt packages. This ensures that updates we make to dbt-core, adapters, or anywhere else are compatible with the code in those packages. If a new version of a dbt package includes a breaking change (for example, a change to one of the macros in `dbt_utils`), you don’t have to immediately use the new version. In your `packages` configuration (in `dependencies.yml` or `packages.yml`), you can still specify which versions or version ranges of packages you want dbt to install. If you're not already doing so, we strongly recommend [checking `package-lock.yml` into version control](https://docs.getdbt.com/reference/commands/deps.md#predictable-package-installs) for predictable package installs in deployment environments and a clear change history whenever you install upgrades. If you upgrade to the **v1 Latest** release track, and immediately see something that breaks, please [contact support](https://docs.getdbt.com/docs/dbt-support.md#dbt-cloud-support) and, in the meantime, downgrade back to v1.7. If you’re already on the **v1 Latest** release track, and you observe a breaking change (like something worked yesterday, but today it isn't working, or works in a surprising/different way), please [contact support](https://docs.getdbt.com/docs/dbt-support.md#dbt-cloud-support) immediately. Depending on your contracted support agreement, the dbt Labs team will respond within our SLA time and we would seek to roll back the change and/or roll out a fix (just as we would for any other part of dbt). This is the same whether or not the root cause of the breaking change is in the project code or in the code of a package. If the package you’ve installed relies on *undocumented* functionality of dbt, it doesn't have the same guarantees as functionality that we’ve documented and tested. However, we will still do our best to avoid breaking them. If you have comments or concerns, we’re happy to help. If you’re an existing dbt customer, you may reach out to your account team or [contact support](https://docs.getdbt.com/docs/dbt-support.md#dbt-cloud-support). --- ### Retry your dbt jobs dbt platform If your dbt job run completed with a status of **Error**, you can rerun it from start or from the point of failure in dbt. #### Prerequisites * You have a [dbt account](https://www.getdbt.com/signup). * You must be using [dbt version](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) 1.6 or newer. * dbt can successfully parse the project and generate a [manifest](https://docs.getdbt.com/reference/artifacts/manifest-json.md) * The most recent run of the job hasn't completed successfully. The latest status of the run is **Error**. * The job command that failed in the run must be one that supports the [retry command](https://docs.getdbt.com/reference/commands/retry.md). #### Rerun an errored job 1. Select **Deploy** from the top navigation bar and choose **Run History.** 2. Choose the job run that has errored. 3. In the **Run Summary** tab on the job’s **Run** page, expand the run step that failed. An ❌ denotes the failed step. 4. Examine the error message and determine how to fix it. After you have made your changes, save and commit them to your [Git repo](https://docs.getdbt.com/docs/platform/git/git-version-control.md). 5. Return to your job’s **Run** page. In the upper right corner, click **Rerun** and choose **Rerun from start** or **Rerun from failure**. If you chose to rerun from the failure point, a **Rerun failed steps** modal opens. The modal lists the run steps that will be invoked: the failed step and any skipped steps. To confirm these run steps, click **Rerun from failure**. The job reruns from the failed command in the previously failed run. A banner at the top of the **Run Summary** tab captures this with the message, "This run resumed execution from last failed step". [![Example of the Rerun options in dbt](/img/docs/deploy/native-retry.gif?v=2 "Example of the Rerun options in dbt")](#)Example of the Rerun options in dbt #### Related content * [Retry a failed run for a job](https://docs.getdbt.com/dbt-cloud/api-v2#/operations/Retry%20Failed%20Job) API endpoint * [Run visibility](https://docs.getdbt.com/docs/deploy/run-visibility.md) * [Jobs](https://docs.getdbt.com/docs/deploy/jobs.md) * [Job commands](https://docs.getdbt.com/docs/deploy/job-commands.md) --- ### Run the self-hosted dbt MCP server Local development This quickstart walks you through connecting dbt MCP server to your local dbt project. This setup gives you dbt command tools (`run`, `build`, `test`, `compile`, and more) inside your AI assistant. If you'd like to connect to dbt platform with the CLI, see the [OAuth quickstart](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-oauth.md). To connect to dbt Wizard, see the [dbt Wizard quickstart](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md). No clone required You don't need to clone the dbt-mcp repository. Install [uv](https://docs.astral.sh/uv/getting-started/installation/) and run `uvx dbt-mcp` — it fetches and runs dbt-mcp for you. #### Prerequisites * [Install uv](https://docs.astral.sh/uv/getting-started/installation/) * A local dbt project (the folder containing your `dbt_project.yml` file) * dbt installed and available locally For the full list of environment variables and how to enable or disable toolsets, see the [Environment variables reference](https://docs.getdbt.com/docs/dbt-ai/mcp-environment-variables.md). #### Step 1: Find your paths You need two values before configuring your MCP client: * `DBT_PROJECT_DIR` — the full path to your dbt project folder (where `dbt_project.yml` lives). For example, if your project name is `jaffle_shop`, the path should be `/Users/yourname/dbt-projects/jaffle_shop`. * `DBT_PATH` — the full path to your dbt executable. ##### macOS/Linux ```bash # Find DBT_PATH which dbt # Example output: /opt/homebrew/bin/dbt # Find DBT_PROJECT_DIR — run from inside your project folder pwd # Example output: /Users/yourname/projects/my_dbt_project ``` ##### Windows ```bash # Find DBT_PATH where dbt # Example output: C:\Python39\Scripts\dbt.exe # Find DBT_PROJECT_DIR — run from inside your project folder cd # Example output: C:\Users\yourname\projects\my_dbt_project ``` Note: Use forward slashes in your configuration: `C:/Python39/Scripts/dbt.exe` #### Step 2: Add to your MCP client Replace the paths below with the values from Step 1: ##### Claude Desktop 1. In Claude Desktop, go to **Settings** → **Developer** tab → **Edit Config**. 2. Paste the following configuration, replacing the paths with your actual values: ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_PROJECT_DIR": "/path/to/your/dbt/project", "DBT_PATH": "/path/to/your/dbt/executable" } } } } ``` 3. Save and restart Claude Desktop. Config file location: * macOS: `~/Library/Application Support/Claude/claude_desktop_config.json` * Windows: `%APPDATA%\Claude\claude_desktop_config.json` ##### Claude Code Run this command, replacing the paths with your actual values: ```shell claude mcp add dbt \ -e DBT_PROJECT_DIR=/path/to/your/dbt/project \ -e DBT_PATH=/path/to/your/dbt/executable \ -- uvx dbt-mcp ``` ##### Cursor Click the link below with Cursor open to auto-configure: [Add dbt v1 or dbt v2 to Cursor](cursor://anysphere.cursor-deeplink/mcp/install?name=dbt\&config=eyJlbnYiOnsiREJUX1BST0pFQ1RfRElSIjoiL3BhdGgvdG8veW91ci9kYnQvcHJvamVjdCIsIkRCVF9QQVRIIjoiL3BhdGgvdG8veW91ci9kYnQvZXhlY3V0YWJsZSJ9LCJjb21tYW5kIjoidXZ4IiwiYXJncyI6WyJkYnQtbWNwIl19) After clicking: 1. Update `DBT_PROJECT_DIR` with the full path to your dbt project. 2. Update `DBT_PATH` with the full path to your dbt executable (from Step 1). 3. Save the configuration. ##### VS Code 1. Open **Settings** → **Features** → **Chat** and ensure **MCP** is enabled. 2. Open the Command Palette (`Ctrl/Cmd + Shift + P`) and select **MCP: Open User Configuration**. 3. Add the following configuration to `mcp.json`, replacing the paths with your actual values: VS Code uses `"servers"`, not `"mcpServers"` ```json { "servers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_PROJECT_DIR": "/path/to/your/dbt/project", "DBT_PATH": "/path/to/your/dbt/executable" } } } } ``` 4. Save the file. #### Step 3: Test your setup Ask your AI assistant to run a dbt command (for example, *"Run `dbt compile` on my project"* or *"List all models in my project"*). If dbt MCP is working, the assistant will execute the command against your local project.  Optional: verify from the command line ```bash uvx dbt-mcp ``` If there are no errors, your configuration is correct. Press `Ctrl+C` to stop the server. #### What's available With CLI-only setup, your AI assistant can use: * `dbt run`, `dbt build`, `dbt test`, `dbt compile`, `dbt list`, `dbt parse`, `dbt show` * Model lineage and node details from your local project * Codegen tools (when enabled — see [Environment variables reference](https://docs.getdbt.com/docs/dbt-ai/mcp-environment-variables.md)) Platform features like Semantic Layer, Discovery API, and metadata queries require a dbt platform account. To add them, see [Connect to dbt platform](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-oauth.md). #### Troubleshooting  Can't find the uvx executable  Can't find the uvx executable **Symptoms:** Error messages like `Could not connect to MCP server dbt-mcp`, `Error: spawn uvx ENOENT`, or `spawn uvx ENOENT` in your MCP client. **Cause:** Your MCP client (like Claude desktop) can't find `uvx` in its PATH because it starts with a limited environment. **Solution:** Use the full path to `uvx` in your configuration. 1. Find the full path: * macOS/Linux: Run `which uvx` in Terminal. * Windows: Run `where uvx` in Command Prompt or PowerShell. 2. Replace `"command": "uvx"` with the full path: ```json { "mcpServers": { "dbt": { "command": "/full/path/to/uvx", "args": ["dbt-mcp"], "env": { } } } } ``` Example on macOS with Homebrew: `"command": "/opt/homebrew/bin/uvx"` For VS Code (`mcp.json`), the same fix applies — replace `uvx` with its full path in the `command` field.  Server not starting  Server not starting **Symptoms:** The MCP server shows as disconnected or unavailable in your client. **Diagnosis:** Check the server logs: * **VS Code:** Open the Command Palette (`Ctrl/Cmd + Shift + P`) → `MCP: List Servers` → click the dbt server to see detailed logs. * **Claude Desktop:** Check `~/Library/Logs/Claude` (macOS) or `%APPDATA%\Claude\logs` (Windows). * **All clients:** Set `DBT_MCP_LOG_LEVEL=DEBUG` in your environment variables to get more verbose output. **Common causes:** * Missing or incorrect `DBT_PROJECT_DIR` or `DBT_PATH` — verify the paths exist and are absolute paths. * Invalid or expired authentication tokens — generate a new token and update your config. * Missing required environment variables for the toolset you're trying to use — see [Tool requirements](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#tool-requirements-at-a-glance). For all troubleshooting topics, see [MCP troubleshooting](https://docs.getdbt.com/docs/dbt-ai/mcp-troubleshooting.md). #### Next steps * Add dbt platform features: see [Connect to dbt platform](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-oauth.md) * Configure toolsets or disable specific tools: see the [Environment variables reference](https://docs.getdbt.com/docs/dbt-ai/mcp-environment-variables.md) --- ### Run visibility dbt platform You can view the history of your runs and the model timing dashboard to help identify where improvements can be made to jobs. #### Run history The **Run history** dashboard in dbt helps you monitor the health of your dbt project. It provides a detailed overview of all your project's job runs and empowers you with a variety of filters that enable you to focus on specific aspects. You can also use it to review recent runs, find errored runs, and track the progress of runs in progress. You can access it from the top navigation menu by clicking **Deploy** and then **Run history**. The dashboard displays your full run history, including job name, status, associated environment, job trigger, commit SHA, schema, and timing info. dbt developers can access their run history for the last 365 days through the dbt user interface (UI) and API. dbt Labs limits self-service retrieval of run history metadata to 365 days to improve dbt's performance. [![Run history dashboard allows you to monitor the health of your dbt project and displays jobs, job status, environment, timing, and more.](/img/docs/dbt-platform/deployment/run-history.png?v=2 "Run history dashboard allows you to monitor the health of your dbt project and displays jobs, job status, environment, timing, and more.")](#)Run history dashboard allows you to monitor the health of your dbt project and displays jobs, job status, environment, timing, and more. #### Job run details From the **Run history** dashboard, select a run to view complete details about it. The job run details page displays job trigger, commit SHA, time spent in the scheduler queue, all the run steps and their [logs](#access-logs), [model timing](#model-timing), and more. Click **Rerun now** to rerun the job immediately. An example of a completed run with a configuration for a [job completion trigger](https://docs.getdbt.com/docs/deploy/deploy-jobs.md#trigger-on-job-completion): [![Example of run details](/img/docs/dbt-platform/deployment/example-job-details.png?v=2 "Example of run details")](#)Example of run details ##### Run summary tab You can view and download in-progress and historical logs for your dbt runs. This makes it easier for you to debug errors more efficiently. For in-progress steps, dbt platform only displays the tail of the log output — up to the last 1,000 lines or 0.5 MB, whichever comes first. This applies to both console and debug logs. * When logs are truncated, a notice appears at the top of the log. Because only the tail is displayed, a resource that is still running may not appear in the logs until it completes and its output reaches the tail. * When a step is complete, the full log is available. (Applies to dbt v2.0 and later) When a job on dbt v2 finishes, selecting a step displays a structured logs view showing the status of each resource. Nodes are classified into the following categories, and you can expand each node to view its log details: * **Success** * **Reused** * **Failed** * **Warning** * **Running** * **Skipped** * **No-op** For more information about each status, refer to [dbt v2 telemetry and observability](https://docs.getdbt.com/reference/telemetry-observability.md#node-outcome). [![Structured logs in v2](/img/docs/dbt-platform/deployment/fusion-logs.png?v=2 "Structured logs in v2")](#)Structured logs in v2 ###### Downloading logs * To download logs for an individual step, select the step in the **Run summary** tab and click **Download** > **Download logs**. * Note that when viewing debug logs, the log output is truncated. To view and export all debug logs for an individual step, click **Download** > **Download all debug logs**. * You can download OpenTelemetry (OTel) logs for dbt v2 job command steps as a Parquet file. The file contains structured step-level log data that you can query or inspect outside dbt. To download it, go to the **Run summary** tab in the job, select a step and click **Download** > **Download OTel log** to export a Parquet file This option only appears when the step emitted an OTel log artifact. Some steps, such as `dbt deps`, don't produce one. For analysis examples, refer to [Querying telemetry data](https://docs.getdbt.com/reference/telemetry-observability.md#querying-telemetry-data). [![Download logs](/img/docs/dbt-platform/deployment/download-logs.png?v=2 "Download logs")](#)Download logs ###### Log size limits dbt enforces cumulative log size limits on run endpoints. If a single step's logs or the total run logs exceed this limit, dbt omits the logs. When dbt omits logs due to size, it displays a **Run logs are too large** banner and shows a message where the logs would usually appear. The run step also displays an **Unknown** status. You can still download omitted logs. If the log file is too large, the download may fail. If that happens, you can [reach out to support](mailto:support@getdbt.com). ##### Lineage tab View the lineage graph associated with the job run so you can better understand the dependencies and relationships of the resources in your project. To view a node's metadata directly in [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md), select it (double-click) from the graph. [![Example of accessing dbt Catalog from the Lineage tab](/img/docs/collaborate/dbt-explorer/explorer-from-lineage.gif?v=2 "Example of accessing dbt Catalog from the Lineage tab")](#)Example of accessing dbt Catalog from the Lineage tab ##### Model timing tab [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") The **Model timing** tab displays the composition, order, and time each model takes in a job run. This helps you identify bottlenecks in your runs so you can investigate them and potentially make changes to improve performance. You can find it on the [job's run details](#job-run-details). The tab includes the following sections: * [Metric tiles](#metric-tiles) * [Execution timeline](#execution-timeline) * [Concurrency over time](#concurrency-over-time) * [Resource details](#resource-details) ###### Metric tiles Six metric tiles appear at the top of the tab: * **Est. critical path**: The estimated duration of the longest chain of dependent models, and what percentage of total run time it represents. * **Peak concurrency**: The maximum number of models running simultaneously, and at what point in the run it occurred. * **Avg active models**: The average number of models active at any point over the run duration. * **Longest model**: The duration and name of the slowest model in the run. * **Wall clock**: The total elapsed time of the run. * **Latest start**: The name and start time of the last model to start in the run. [![Metric tiles showing key run statistics like estimated critical path, peak concurrency, longest model, and more.](/img/docs/dbt-platform/deployment/model-timing-metric-tiles.png?v=2 "Metric tiles showing key run statistics like estimated critical path, peak concurrency, longest model, and more.")](#)Metric tiles showing key run statistics like estimated critical path, peak concurrency, longest model, and more. ###### Execution timeline A Gantt-style timeline of all resources in the run. Hover over bars to see details. You can customize the view using: * **Group by**: Controls how resources are grouped in the timeline: * **Resource type**: Groups by node type: Model, Test, Snapshot, or Exposure. * **Folder**: Groups by the folder path of the resource in your project. * **Execution phase**: Groups resources into phases 0 through 4 based on execution start time — Phase 0 started earliest, while Phase 4 started latest. * **Thread**: Groups by the dbt execution thread that ran each resource. dbt uses multiple threads to run models in parallel; this view shows which thread handled which resources. * **No grouping**: Lists all resources without any grouping. * **Highlight**: Changes how bars are colored to help you focus on what matters: * **Est. critical path**: Highlights resources on the estimated critical path by graying out all others. * **All equal**: Shows all bars in their resource type color with no additional emphasis. The legend shows the color for each type: Model, Test, Snapshot, and Exposure. * **By duration**: Grays out shorter-running resources and shows longer-running ones in color. * **Search resources**: Filters the timeline to resources matching your search term. [![Execution timeline showing a Gantt-style view of all resources in the run](/img/docs/dbt-platform/deployment/model-timing-timeline.png?v=2 "Execution timeline showing a Gantt-style view of all resources in the run")](#)Execution timeline showing a Gantt-style view of all resources in the run ###### Concurrency over time A stacked bar chart showing model activity over the run duration. Each bar is split into **Active models** and **Queued / ready**, so you can see how many models were running versus waiting at any point in time. It also displays the peak concurrency reached during the run. [![Concurrency over time chart showing active models and queued/ready models throughout the run](/img/docs/dbt-platform/deployment/model-timing-concurrency.png?v=2 "Concurrency over time chart showing active models and queued/ready models throughout the run")](#)Concurrency over time chart showing active models and queued/ready models throughout the run ###### Resource details A paginated, searchable table listing all resources in the run. It includes the following columns: * **Model**: The resource name. * **Start**: The time the resource started. * **End**: The time the resource finished. * **Duration**: How long the resource took to run. * **Execution phase**: A number from 0 to 4 indicating when the resource started relative to others in the run — Phase 0 started earliest, while Phase 4 started latest. * **Est. critical path**: Whether the resource is on the estimated critical path. * **Type**: The resource type (Model, Test, Snapshot, or Exposure), displayed as a color-coded badge. * **Folder**: The folder path of the resource. [![Resource details table showing each model's start time, end time, duration, execution phase, critical path status, type, and folder](/img/docs/dbt-platform/deployment/model-timing-resource-details.png?v=2 "Resource details table showing each model's start time, end time, duration, execution phase, critical path status, type, and folder")](#)Resource details table showing each model's start time, end time, duration, execution phase, critical path status, type, and folder ##### Artifacts tab This provides a list of the artifacts generated by the job run. The files are saved and available for download. [![Example of the Artifacts tab](/img/docs/dbt-platform/example-artifacts-tab.png?v=2 "Example of the Artifacts tab")](#)Example of the Artifacts tab ##### Explain tab The **Explain** tab appears on job runs that used [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md). It shows why dbt State rebuilt, reused, or cloned each resource, so you can investigate unexpected behavior or verify that State is working as expected. The tab is available while a run is in progress and updates as resources finish. The tab displays an **Explain results** table with one row per resource. You can search by resource name and download the full results as a text file. Expand a row to see the full decision details. Not all analyses apply to every resource type: | Field | Description | | --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | **Resource name** | The name of the resource. | | **Resource type** | The resource type: model, seed, snapshot, test, and so on. | | **Decision** | The reason dbt State rebuilt, reused, or cloned this resource. | | **Run step** | The job command that ran this resource (for example, `dbt build --exclude tag:ml_pipeline`). | | **Table analysis** | Whether the target table already exists in the schema. | | **Query analysis** | Whether the resource query or its upstream queries have changed. | | **Data freshness analysis** | Whether upstream data is fresh or within the configured [`lag_tolerance`](https://docs.getdbt.com/reference/resource-configs/lag-tolerance.md). | [![Explain tab showing the decision breakdown](/img/docs/dbt-platform/deployment/explain-tab.png?v=2 "Explain tab showing the decision breakdown")](#)Explain tab showing the decision breakdown ##### Compare tab [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") The **Compare** tab is shown for [CI job runs](https://docs.getdbt.com/docs/deploy/ci-jobs.md) with the **Run compare changes** setting enabled. It displays details about [the changes from the comparison dbt performed](https://docs.getdbt.com/docs/deploy/advanced-ci.md#compare-changes) between what's in your production environment and the pull request. To help you better visualize the differences, dbt highlights changes to your models in red (deletions) and green (inserts). From the **Modified** section, you can view the following: * **Overview**: High-level summary about the changes to the models such as the number of primary keys that were added or removed. * **Primary keys**: Details about the changes to the records. * **Modified rows**: Details about the modified rows. Click **Show full preview** to display all columns. * **Columns**: Details about the changes to the columns. To view the dependencies and relationships of the resources in your project more closely, click **View in Catalog** to launch [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md). [![Example of the Compare tab](/img/docs/dbt-platform/example-ci-compare-changes-tab.png?v=2 "Example of the Compare tab")](#)Example of the Compare tab [![Example of Modified rows tab which shows you details of the modified rows.](/img/docs/dbt-platform/advanced-ci-modified-rows.png?v=2 "Example of Modified rows tab which shows you details of the modified rows.")](#)Example of Modified rows tab which shows you details of the modified rows. --- ### Saved queries Saved queries are a way to save commonly used queries in MetricFlow. You can group metrics, dimensions, and filters that are logically related into a saved query. Saved queries are nodes and visible in the dbt DAG. Saved queries serve as the foundational building block, allowing you to [configure exports](#configure-exports) in your saved query configuration. Exports takes this functionality a step further by enabling you to [schedule and write saved queries](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md) directly within your data platform using [dbt's job scheduler](https://docs.getdbt.com/docs/deploy/job-scheduler.md). #### Parameters To create a saved query, refer to the following table parameters. tip Note that we use dot notation (`.`) to indicate whether a parameter is nested within another parameter. For example, `query_params.metrics` means the `metrics` parameter is nested under `query_params`. (Applies to dbt v1.9 and later) | Parameter | Type | Required | Description | | -------------------------- | ----------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `name` | String | Required | Name of the saved query object. | | `description` | String | Required | A description of the saved query. | | `label` | String | Required | The display name for your saved query. This value will be shown in downstream tools. | | `config` | String | Optional | Use the [`config`](https://docs.getdbt.com/reference/resource-properties/config.md) property to specify configurations for your saved query. Supports `cache`, [`enabled`](https://docs.getdbt.com/reference/resource-configs/enabled.md), `export_as`, [`group`](https://docs.getdbt.com/reference/resource-configs/group.md), [`meta`](https://docs.getdbt.com/reference/resource-configs/meta.md), [`tags`](https://docs.getdbt.com/reference/resource-configs/tags.md), and [`schema`](https://docs.getdbt.com/reference/resource-configs/schema.md) configurations. | | `config.cache.enabled` | Object | Optional | An object with a sub-key used to specify if a saved query should populate the [cache](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-cache.md). Accepts sub-key `true` or `false`. Defaults to `false` | | `limit` | Integer | Optional | The maximum number of rows to return. | | `order_by` | String | Optional | The metrics and group bys to order the query by. | | `query_params` | Structure | Required | Contains the query parameters. | | `query_params.metrics` | List or String | Optional | A list of the metrics to be used in the query as specified in the command line interface. | | `query_params.group_by` | List or String | Optional | A list of the Entities and Dimensions to be used in the query, which include the `Dimension` or `TimeDimension`. | | `query_params.where` | List or String | Optional | A list of strings that may include the `Dimension` or `TimeDimension` objects. | | `exports` | List or Structure | Optional | A list of exports to be specified within the exports structure. | | `exports.name` | String | Required | Name of the export object. | | `exports.config` | List or Structure | Required | A [`config`](https://docs.getdbt.com/reference/resource-properties/config.md) property for any parameters specifying the export. | | `exports.config.export_as` | String | Required | The type of export to run. Options include table or view currently and cache in the near future. | | `exports.config.schema` | String | Optional | The [schema](https://docs.getdbt.com/reference/resource-configs/schema.md) for creating the table or view. This option cannot be used for caching. | | `exports.config.alias` | String | Optional | The table [alias](https://docs.getdbt.com/reference/resource-configs/alias.md) used to write to the table or view. This option cannot be used for caching. | If you use multiple metrics in a saved query, then you will only be able to reference the common dimensions these metrics share in the `group_by` or `where` clauses. Use the entity name prefix with the Dimension object, like `Dimension('user__ds')`. #### Configure saved query Use saved queries to define and manage common Semantic Layer queries in YAML, including metrics and dimensions. Saved queries enable you to organize and reuse common MetricFlow queries within dbt projects. For example, you can group related metrics together for better organization, and include commonly used dimensions and filters. In your saved query config, you can also leverage [caching](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-cache.md) with the dbt job scheduler to cache common queries, speed up performance, and reduce compute costs. In the following example, you can set the saved query in the `semantic_model.yml` file: semantic\_model.yml (Applies to dbt v1.9 and later) ```yaml saved_queries: - name: test_saved_query description: "{{ doc('saved_query_description') }}" label: Test saved query config: cache: enabled: true | false tags: 'my_tag' query_params: metrics: - simple_metric group_by: - "Dimension('user__ds')" where: - "{{ Dimension('user__ds', 'DAY') }} <= now()" - "{{ Dimension('user__ds', 'DAY') }} >= '2023-01-01'" exports: - name: my_export config: export_as: table alias: my_export_alias schema: my_export_schema_name ``` Note that you can set `export_as` to both the saved query and the exports [config](https://docs.getdbt.com/reference/resource-properties/config.md), with the exports config value taking precedence. If a key isn't set in the exports config, it will inherit the saved query config value. ###### Where clause Use the following syntax to reference entities, dimensions, time dimensions, or metrics in filters and refer to [Metrics as dimensions](https://docs.getdbt.com/docs/build/ref-metrics-in-filters.md) for details on how to use metrics as dimensions with metric filters: ```yaml filter: | {{ Entity('entity_name') }} filter: | {{ Dimension('primary_entity__dimension_name') }} filter: | {{ TimeDimension('time_dimension', 'granularity') }} filter: | {{ Metric('metric_name', group_by=['entity_name']) }} ``` ###### Project-level saved queries To enable saved queries at the project level, you can set the `saved-queries` configuration in the [`dbt_project.yml` file](https://docs.getdbt.com/reference/dbt_project.yml.md). This saves you time in configuring saved queries in each file: dbt\_project.yml ```yaml saved-queries: my_saved_query: +cache: enabled: true ``` For more information on `dbt_project.yml` and config naming conventions, see the [dbt\_project.yml reference page](https://docs.getdbt.com/reference/dbt_project.yml.md#naming-convention). To build `saved_queries`: * Make sure you set the right [environment variable](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md#set-environment-variable) in your environment. * Run the command `dbt build --resource-type saved_query` using the [`--resource-type` flag](https://docs.getdbt.com/reference/global-configs/resource-type.md). #### Configure exports Exports are an additional configuration added to a saved query. They define *how* to write a saved query, along with the schema and table name. Once you've configured your saved query and set the foundation block, you can now configure exports in the `saved_queries` YAML configuration file (the same file as your metric definitions). This will also allow you to [run exports](#run-exports) automatically within your data platform using [dbt's job scheduler](https://docs.getdbt.com/docs/deploy/job-scheduler.md). The following is an example of a saved query with an export: semantic\_model.yml (Applies to dbt v1.9 and later) ```yaml saved_queries: - name: order_metrics description: Relevant order metrics config: tags: - order_metrics query_params: metrics: - orders - large_order - food_orders - order_total group_by: - Entity('order_id') - TimeDimension('metric_time', 'day') - Dimension('customer__customer_name') - ... # Additional group_by where: - "{{TimeDimension('metric_time')}} > current_timestamp - interval '1 week'" - ... # Additional where clauses exports: - name: order_metrics config: export_as: table # Options available: table, view alias: my_export_alias # Optional - defaults to Export name schema: my_export_schema_name # Optional - defaults to deployment schema ``` #### Run exports Once you've configured exports, you can now take things a step further by running exports to automatically write saved queries within your data platform using [dbt's job scheduler](https://docs.getdbt.com/docs/deploy/job-scheduler.md). This feature is only available with the [dbt's Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md). For more information on how to run exports, refer to the [Exports](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md) documentation. #### FAQs  Can I have multiple exports in a single saved query? Yes, this is possible. However, the difference would be the name, schema, and materialization strategy of the export.  How can I select saved\_queries by their resource type? To include all saved queries in the dbt build run, use the [`--resource-type` flag](https://docs.getdbt.com/reference/global-configs/resource-type.md) and run the command `dbt build --resource-type saved_query`. #### Related docs * [Validate semantic nodes in a CI job](https://docs.getdbt.com/docs/deploy/ci-jobs.md#semantic-validations-in-ci) * Configure [caching](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-cache.md) --- ### Semantic models Tip Use [dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-overview.md) to generate semantic models in the dbt platform or locally in the CLI. Semantic models are the foundation for data definition in MetricFlow, which powers the Semantic Layer: (Applies to dbt v1.12 and later) * Think of semantic models as nodes connected by entities in a semantic graph. You can configure this graph by making metadata annotations on your dbt models to describe their usage in metric calculations. * MetricFlow uses YAML configuration files to create this graph for querying metrics. * Each semantic model corresponds to a dbt model in your DAG, requiring a unique YAML configuration for each semantic model. * Each dbt model can define one semantic model via a `semantic_model` block. Use the optional `name` field if you need a different display name. * Configure semantic models in a YAML file within your dbt project directory, embedded within your model definitions rather than as separate configurations. * You can also define semantic models using [Apache Ossie](https://github.com/apache/ossie) documents, an alternative to dbt's native YAML configuration. For more information, refer to [Ossie semantic layer documents](https://docs.getdbt.com/docs/build/ossie-semantic-models.md). 📹 Learn about the dbt Semantic Layer with on-demand video courses! Explore our [dbt Semantic Layer on-demand course](https://learn.getdbt.com/courses/semantic-layer) to learn how to define and query metrics in your dbt project. Additionally, dive into mini-courses for querying the dbt Semantic Layer in your favorite tools: [Tableau](https://courses.getdbt.com/courses/tableau-querying-the-semantic-layer), [Excel](https://learn.getdbt.com/courses/querying-the-semantic-layer-with-excel), [Hex](https://courses.getdbt.com/courses/hex-querying-the-semantic-layer), and [Mode](https://courses.getdbt.com/courses/mode-querying-the-semantic-layer). Here we describe the Semantic model components with examples: (Applies to dbt v1.12 and later) | Component | Description | Required | Type | | ------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------ | | [Name](#name) | Choose a name for the semantic model. Avoid using double underscores (\_\_) in the name as they're not supported. Defaults to value of the model name. | Optional | String | | [Time dimension](#time-dimension) | Only `agg_time_dimension` is supported. | Required | Dict | | [Entities](#entities) | Define entities at the column level. For any column that serves as a join key, add an entity block under the column with `type` set to primary, foreign, unique, or natural. Optionally include name, description, label, config, or use expr when the join key differs from the column. | Required | List | | [Primary Entity](#primary-entity) | If a column declares an entity with `type: primary`, you don’t need `primary_entity`. If no column is primary, set the top-level `primary_entity` to name the model’s primary entity. | Optional | String | | [Dimensions](#dimensions) | Different ways to group or slice data for a metric, they can be `time` or `categorical`. | Required | List | | [Derived semantics](#derived_semantics-in-dimensions-and-entities) | Use to create dimensions and entities that don’t match a single, physical column. The `expr` field is required. | Optional | List | | [Simple metrics](#measures) | Aggregations applied to columns in your data model. They can be the final metric or used as building blocks for more complex metrics. | Optional | List | | [Label](#label) | The display name for your semantic model `node`, `dimension`, `entity`, and/or simple metrics. | Optional | String | | `config` | Use the [`config`](https://docs.getdbt.com/reference/resource-properties/config.md) property to specify configurations for your metric. Supports [`meta`](https://docs.getdbt.com/reference/resource-configs/meta.md), [`group`](https://docs.getdbt.com/reference/resource-configs/group.md), and [`enabled`](https://docs.getdbt.com/reference/resource-configs/enabled.md) configs. | Optional | Dict | #### Semantic models components The complete spec for semantic models is below: (Applies to dbt v1.12 and later) ```yaml models: - name: the_name_of_the_model # Model name semantic_model: enabled: true name: the_name_of_the_semantic_model # Optional: semantic model name, defaults to model name agg_time_dimension: dimension_name # Required # Entities, dimensions defined at column level columns: - name: entity_column_name entity: type: primary # or foreign name: entity_name description: entity description - name: dimension_column_name dimension: type: categorical # or time name: dimension_name description: dimension description # Simple metrics replace measures metrics: - name: metric_name description: metric description type: simple label: Metric Label agg: sum # Required expr: column_name # Optional, defaults to metric name # Other properties like fill_nulls_with, join_to_timespine can go here ``` The following example displays a complete configuration and detailed descriptions of each field: (Applies to dbt v1.12 and later) ```yaml models: - name: fact_transactions # Model name description: "Transaction fact table at the transaction level. This table contains one row per transaction and includes the transaction timestamp." semantic_model: enabled: true agg_time_dimension: transaction_date columns: - name: transaction_id entity: # Entities included in the table are defined here. MetricFlow will use these columns as join keys. type: primary name: transaction - name: customer_id entity: type: foreign name: customer - name: transaction_date granularity: day dimension: # Dimensions are qualitative values such as names, dates, or geographical data. They provide context to metrics and allow "metric by group" data slicing. type: time - name: order_country dimension: type: categorical name: transaction_location metrics: # Simple metrics are columns we perform an aggregation over. These are inputs to metrics. - name: transaction_total description: "The total value of the transaction." type: simple label: Transaction Total agg: sum expr: transaction_total - name: average_transaction_total description: "The average total sale of the transaction." type: simple label: Average Transaction Total agg: average expr: average_transaction_total - name: median_sales description: "The median sale of the transaction." type: simple label: Median Sales agg: median expr: transaction_total - name: dim_customers # Another model description: "A customer dimension table." semantic_model: enabled: true columns: - name: customer_id entity: type: primary name: customer - name: first_name dimension: type: categorical name: first_name ``` (Applies to dbt v1.12 and later) Semantic models support [`meta`](https://docs.getdbt.com/reference/resource-configs/meta.md), [`group`](https://docs.getdbt.com/reference/resource-configs/group.md), and [`enabled`](https://docs.getdbt.com/reference/resource-configs/enabled.md) [`config`](https://docs.getdbt.com/reference/resource-properties/config.md) property in the schema file: ```yml models: - name: orders semantic_model: enabled: true | false group: some_group config: meta: some_key: some_value ``` (Applies to dbt v1.12 and later) ##### Name Define the name of the semantic model. If not provided, this defaults to the value of the model name. Avoid using double underscores (\_\_) in the name as they're not supported. (Applies to dbt v1.12 and later) ##### Time dimension `agg_time_dimension` represents the default time dimensions for simple metrics. This can be overridden by adding the `agg_time_dimension` key directly to a simple metric - see [Dimensions](https://docs.getdbt.com/docs/build/dimensions.md) for examples. (Applies to dbt v1.12 and later) ##### Entities To specify the [entities](https://docs.getdbt.com/docs/build/entities.md) in your model, add an `entity` block at the column level with `type` set to primary, foreign, unique, or natural. Optionally, provide `name` and `expr` when the join key’s name differs from the column. ##### Primary entity (Applies to dbt v1.12 and later) MetricFlow requires that all dimensions be tied to an entity to guarantee unique dimension names. If your data source doesn't have a primary entity, you need to assign the entity under the column that serves as the key by adding `type: primary`. If no column is marked as primary, set a top-level `primary_entity` to name the model’s primary entity. The primary entity doesn't necessarily have to map to a column in that table and assigning the name doesn't affect query generation. You can define a primary entity using the following configs: (Applies to dbt v1.12 and later) ```yaml models: - name: bookings_monthly_source semantic_model: enabled: true agg_time_dimension: ds columns: - name: booking_id entity: type: primary name: booking_id - name: ds dimension: type: time name: ds metrics: - name: bookings_monthly description: "Sum of bookings monthly" type: simple label: "Bookings Monthly" agg: sum expr: bookings_monthly ``` ##### Entity types Here are the types of keys: * **Primary** — Only one record per row in the table, and it includes every record in the data platform. * **Unique** — Only one record per row in the table, but it may have a subset of records in the data platform. Null values may also be present. * **Foreign** — Can have zero, one, or multiple instances of the same record. Null values may also be present. * **Natural** — A column or combination of columns in a table that uniquely identifies a record based on real-world data. For example, the `sales_person_id` can serve as a natural key in a `sales_person_department` dimension table. ##### Sample config This example shows a semantic model with three entities and their entity types: `transaction` (primary), `order` (foreign), and `user` (foreign). To reference a desired column, use the actual column name from the model in the `name` parameter. You can also use `name` as an alias to rename the column, and the `expr` parameter to refer to the original column name or a SQL expression of the column. ```yaml entity: - name: transaction type: primary - name: order type: foreign expr: id_order - name: user type: foreign expr: substring(id_order FROM 2) ``` You can refer to entities (join keys) in a semantic model using the `name` parameter. Entity names must be unique within a semantic model, and identifier names can be non-unique across semantic models since MetricFlow uses them for [joins](https://docs.getdbt.com/docs/build/join-logic.md). ##### Dimensions [Dimensions](https://docs.getdbt.com/docs/build/dimensions.md) are different ways to organize or look at data. They are effectively the group by parameters for metrics. For example, you might group data by things like region, country, or job title. (Applies to dbt v1.12 and later) MetricFlow takes a dynamic approach when making dimensions available for metrics. Instead of trying to figure out all the possible groupings ahead of time, MetricFlow lets you ask for the dimensions you need and constructs any joins necessary to reach the requested dimensions at query time. The advantage of this approach is that you don't need to set up a system that pre-materializes every possible way to group data, which can be time-consuming and prone to errors. Instead, you define dimensions within the semantic model by nesting them under columns, and they will automatically be made available for valid metrics. Dimensions have the following characteristics: * There are two types of dimensions: categorical and time. Categorical dimensions are for things you can't measure in numbers, while time dimensions represent dates and timestamps. Time dimensions require a column-level granularity. * Dimensions are bound to the primary entity of the semantic model in which they are defined. For example, if a dimension called `full_name` is defined in a model with `user` as a primary entity, then `full_name` is scoped to the `user` entity. To reference this dimension, you would use the fully qualified dimension name `user__full_name`. * The naming of dimensions must be unique in each semantic model with the same primary entity. Dimension names can be repeated if defined in semantic models with a different primary entity. For time groups For semantic models that define metrics, include at least one time dimension column with a `granularity` and set the model’s `agg_time_dimension`. You can override the aggregation time dimension on individual metrics if needed. (Applies to dbt v1.12 and later) ##### `derived_semantics` in dimensions and entities Use the `derived_semantics` key in the model YAML entry when you want to create dimensions and entities that don’t match a single, physical column. The `expr` field is required when using `derived_semantics`. For more information, see [Dimensions](https://docs.getdbt.com/docs/build/dimensions.md#derived_semantics-in-dimensions) and [Entities](https://docs.getdbt.com/docs/build/entities.md#derived_semantics-in-entities). ##### Simple metrics Simple metrics are direct aggregations over columns in your data warehouse using different aggregation types. They serve as building blocks for more complex metrics and can be filtered by dimensions. Simple metrics have various parameters which are listed in a table along with their descriptions and types. For more information, see [Simple metrics](https://docs.getdbt.com/docs/build/simple.md). #### Dependencies (Applies to dbt v1.12 and later) Metric nodes will reflect dependencies on semantic models based on their simple metrics. However, dependencies based on filters should not be reflected in: * [dbt selection syntax](https://docs.getdbt.com/reference/node-selection/syntax.md) * Visualization of the DAG in dbt-docs and the [integrated development environment](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) (IDE). This is because metrics need to source nodes for their `depends_on` attribute from a few different places: * `RATIO` metrics depend on `numerator` and `denominator` metrics, while `DERIVED` metrics depend on `input_metrics`. * `SIMPLE` type metrics depend on their semantic\_model. For example, when you run the command `dbt list --select my_semantic_model+`, it will show you the metrics that belong to the specified semantic model. But there's a condition: Only the metrics that actually use simple metrics or derived metrics from that semantic model will be included in the list. In other words, if a metric only uses a dimension from the semantic model in its filters, it won't be considered as part of that semantic model. #### Related docs (Applies to dbt v1.12 and later) * [About MetricFlow](https://docs.getdbt.com/docs/build/about-metricflow.md) * [Dimensions](https://docs.getdbt.com/docs/build/dimensions.md) * [Entities](https://docs.getdbt.com/docs/build/entities.md) * [Simple metrics](https://docs.getdbt.com/docs/build/simple.md) * [Semantic Layer best practices guide](https://docs.getdbt.com/best-practices/how-we-build-our-metrics/semantic-layer-1-intro.md) --- ### Set up automatic exposures in Tableau dbt platform | Enterprise, Enterprise+ Set up and automatically populate downstream exposures for supported BI tool integrations, like Tableau. Visualize and orchestrate them through [dbt Catalog](https://docs.getdbt.com/docs/explore/explore-projects) and the [dbt job scheduler](https://docs.getdbt.com/docs/deploy/job-scheduler.md) for a richer experience. As a data team, it’s critical that you have context into the downstream use cases and users of your data products. By leveraging automatic downstream [exposures](https://docs.getdbt.com/docs/build/exposures.md), you can: * Gain a better understanding of how models are used in downstream analytics, improving governance and decision-making. * Reduce incidents and optimize workflows by linking upstream models to downstream dependencies. * Automate exposure tracking for supported BI tools, ensuring lineage is always up to date. * [Orchestrate exposures](https://docs.getdbt.com/docs/platform-integrations/orchestrate-exposures.md) to refresh the underlying data sources during scheduled dbt jobs, improving timeliness and reducing costs. Orchestrating exposures is a way to ensure that your BI tools are updated regularly using the [dbt job scheduler](https://docs.getdbt.com/docs/deploy/job-scheduler.md). See the [previous page](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures.md) for more information. In dbt, you can configure downstream exposures in two ways: * Manually — Declared [explicitly](https://docs.getdbt.com/docs/build/exposures.md#declaring-an-exposure) in your project’s YAML files. * Automatic — dbt [creates and visualizes downstream exposures](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures.md) automatically for supported integrations, removing the need for manual YAML definitions. These downstream exposures are stored in dbt’s metadata system, appear in [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md), and behave like manual exposures. However, they don’t exist in YAML files. Tableau Server If you're using Tableau Server, you need to add the [dbt IP addresses for your region](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) to your allowlist. #### Prerequisites To configure automatic downstream exposures, you should meet the following: 1. Your environment and jobs are on a supported [dbt release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). 2. You have a dbt account on the [Enterprise or Enterprise+ plan](https://www.getdbt.com/pricing/). 3. You have set up a [production](https://docs.getdbt.com/docs/deploy/deploy-environments.md#set-as-production-environment) deployment environment for each project you want to explore, with at least one successful job run. 4. You have [proper permissions](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) to edit dbt project or production environment settings. 5. Use Tableau as your BI tool and enable metadata permissions or work with an admin to do so. Compatible with Tableau Cloud or Tableau Server with the Metadata API enabled. 6. You have configured a [Tableau personal access token (PAT)](https://help.tableau.com/current/server/en-us/security_personal_access_tokens.htm) whose creator has permission to view data sources. The PAT inherits the permissions of its creator, so ensure the Tableau user who created the token has [Connect permissions](https://help.tableau.com/current/api/rest_api/en-us/REST/rest_api_concepts_permissions.htm). ##### Considerations Configuring automatic downstream exposures with Tableau have the following considerations: * You can only connect to a single Tableau site on the same server. * If you're using Tableau Server, you need to [allowlist dbt's IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for your dbt region. * Tableau dashboards built using custom SQL queries aren't supported. * Downstream exposures sync automatically *once per day* or when a user updates the selected collections. *  The database fully qualified names (FQNs) in Tableau must match those in the dbt build. Tableau's database FQNs (fully qualified names) must match those in the dbt build. To view all expected dependencies in your exposure, the FQNs must match but aren't case-sensitive. For example: | Tableau FQN | dbt FQN | Result | | ---------------------------------- | --------------------------------------- | ---------------------------------------------------------------- | | `analytics.dbt_data_team.my_model` | `analytics.dbt_data_team.my_model` | ✅ Matches and dependencies will display as expected. | | `analytics.dbt_data_team.my_model` | `prod_analytics.dbt_data_team.my_model` | ❌ Doesn't match and not all expected dependencies will display. | To troubleshoot this: 1. In dbt, download the `manifest.json` from the most recent production run that includes the missing dependencies by clicking on the **Artifacts** tab and scrolling to `manifest.json`. 2. Run the following [GraphiQl](https://help.tableau.com/current/api/metadata_api/en-us/docs/meta_api_start.html#explore-the-metadata-api-schema-using-graphiql) query. Make sure to run the query at `your_tableau_server/metadata/graphiql`, where `your_tableau_server` is the value you provided for the Server URL when [setting up your Tableau integration](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md#set-up-in-tableau): ```jsx query { workbooks { name uri id luid projectLuid projectName upstreamTables { id name schema database { name connectionType } } } } ``` 3. Compare database FQNs between `manifest.json` and the GraphiQL response. Make sure that `{database}.{schema}.{name}` matches in both. The following images are examples of FQNs that *match* in both `manifest.json` and the GraphiQL response and aren't case-sensitive: [![manifest.json example with lowercase FQNs.](/img/docs/platform-integrations/auto-exposures/manifest-json-example.png?v=2 "manifest.json example with lowercase FQNs.")](#)manifest.json example with lowercase FQNs. [![GraphiQl response example with uppercase FQNs.](/img/docs/platform-integrations/auto-exposures/graphiql-example.png?v=2 "GraphiQl response example with uppercase FQNs.")](#)GraphiQl response example with uppercase FQNs. 4. If the FQNs don't match, update your Tableau FQNs to match the dbt FQNs. 5. If you're still experiencing issues, please contact [dbt Support](mailto:support@getdbt.com) and share the results with them. #### Set up downstream exposures Set up downstream exposures in [Tableau](#set-up-in-tableau) and [dbt](#set-up-in-dbt-cloud) to ensure that your BI tool's extracts are updated automatically. ##### Set up in Tableau This section explains the steps to configure the integration in Tableau. A Tableau site admin must complete these steps. Once configured in both Tableau and [dbt](#set-up-in-dbt-cloud), you can [view downstream exposures](#view-downstream-exposures) in Catalog. 1. Enable [personal access tokens (PATs)](https://help.tableau.com/current/server/en-us/security_personal_access_tokens.htm) for your Tableau account. [![Enable PATs for the account in Tableau](/img/docs/platform-integrations/auto-exposures/tableau-enable-pat.jpg?v=2 "Enable PATs for the account in Tableau")](#)Enable PATs for the account in Tableau 2. Create a PAT to add to dbt to pull in Tableau metadata for the downstream exposures. When creating the token, you must have permission to access collections/folders, as the PAT only grants access matching the creator's existing privileges. [![Create PATs for the account in Tableau](/img/docs/platform-integrations/auto-exposures/tableau-create-pat.jpg?v=2 "Create PATs for the account in Tableau")](#)Create PATs for the account in Tableau 3. Copy the **Secret** and the **Token name** for use in a later step in dbt. The secret is only displayed once, so store it in a safe location (like a password manager). [![Copy the secret and token name to enter them in dbt](/img/docs/platform-integrations/auto-exposures/tableau-copy-token.jpg?v=2 "Copy the secret and token name to enter them in dbt")](#)Copy the secret and token name to enter them in dbt 4. Copy the **Server URL** and **Sitename**. You can find these in the URL while logged into Tableau. [![Locate the Server URL and Sitename in Tableau](/img/docs/platform-integrations/auto-exposures/tablueau-serverurl.jpg?v=2 "Locate the Server URL and Sitename in Tableau")](#)Locate the Server URL and Sitename in Tableau For example, if the full URL is: `10az.online.tableau.com/#/site/dbtlabspartner/explore`: * The **Server URL** is the fully qualified domain name, in this case: `10az.online.tableau.com` * The **Sitename** is the path fragment right after `site` in the URL, in this case: `dbtlabspartner` 5. With the following items copied, you are now ready to set up downstream exposures in dbt: * ServerURL * Sitename * Token name * Secret ##### Set up in dbt 1. In dbt, navigate to the **Dashboard** of the project you want to add the downstream exposure to and then select **Settings**. 2. Under the **Exposures** section, select **Add lineage integration** to add the Tableau connection. [![Select Add lineage integration to add the Tableau connection.](/img/docs/platform-integrations/auto-exposures/cloud-add-integration.png?v=2 "Select Add lineage integration to add the Tableau connection.")](#)Select Add lineage integration to add the Tableau connection. 3. Enter the details for the exposure connection you collected from Tableau in the [previous step](#set-up-in-tableau) and click **Continue**. Note that all fields are case-sensitive. [![Enter the details for the exposure connection.](/img/docs/platform-integrations/auto-exposures/cloud-integration-details.png?v=2 "Enter the details for the exposure connection.")](#)Enter the details for the exposure connection. 4. Select the collections you want to include for the downstream exposures and click **Save**. [![Select the collections you want to include for the downstream exposures.](/img/docs/platform-integrations/auto-exposures/cloud-select-collections.png?v=2 "Select the collections you want to include for the downstream exposures.")](#)Select the collections you want to include for the downstream exposures. info dbt automatically imports and syncs any workbook within the selected collections. New additions to the collections will appear in the lineage in dbt once per day — after the daily sync and a job run. dbt immediately starts a sync when you update the selected collections list, capturing new workbooks and removing irrelevant ones. 5. dbt imports everything in the collection(s) and you can continue to [view them](#view-downstream-exposures) in Catalog. [![View from the dbt Catalog in your Project lineage view, displayed with the Tableau icon.](/img/docs/platform-integrations/auto-exposures/explorer-lineage2.png?v=2 "View from the dbt Catalog in your Project lineage view, displayed with the Tableau icon.")](#)View from the dbt Catalog in your Project lineage view, displayed with the Tableau icon. #### View downstream exposures After setting up downstream exposures in dbt, you can view them in [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) for a richer experience. Navigate to Catalog by selecting **Catalog** from the top-level navigation. You can view downstream exposures from a couple of places: * [Exposures menu](#exposures-menu) * [Exposure detail page](#exposure-detail-page) * [Project lineage](#project-lineage) ##### Exposures menu View all downstream exposures for a project from the Catalog sidebar: 1. In the sidebar, select your project. 2. Under the project, select **Exposure**. You will only see this option if you set up downstream exposures in [Tableau](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md#set-up-in-tableau) and [dbt](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md#set-up-in-dbt-cloud). The **Exposures** table lists every exposure in the project so you can quickly access and manage them: * **Name**: The name of the exposure. * **Health**: The [data health signal](https://docs.getdbt.com/docs/explore/data-health-signals.md) of the exposure. * **Type**: The type of exposure, such as `dashboard` or `notebook`. * **Owner**: The owner of the exposure. * **Owner email**: The email address of the owner of the exposure. * **Integration**: The BI tool that the exposure is integrated with. * **Exposure mode**: The type of exposure defined: **Auto** or **Manual**. Auto exposures from Tableau appear alongside manual exposures [![View the Exposures table from the Catalog sidebar.](/img/docs/platform-integrations/auto-exposures/explorer-view-resources.png?v=2 "View the Exposures table from the Catalog sidebar.")](#)View the Exposures table from the Catalog sidebar. ##### Exposure detail page After you open the **Exposures** table ([previous section](#exposures-menu)), select an exposure name to open its detail page. On the exposure detail page, you can review metadata, [data health signals](https://docs.getdbt.com/docs/explore/data-health-signals.md), description, and lineage. For Tableau auto exposures, use **Open in Dashboard** to open the workbook in Tableau, or **Modify integration** to update your Tableau connection settings. [![View an exposure detail page in Catalog.](/img/docs/platform-integrations/auto-exposures/explorer-view-exposure-detail.png?v=2 "View an exposure detail page in Catalog.")](#)View an exposure detail page in Catalog. ##### Project lineage You can also view exposures from the **Project lineage** view, separate from the **Exposures** table: 1. In the sidebar, select your project. 2. Click **View lineage**. 3. Select an exposure node with the Tableau icon to view its details in the side panel. This view visualizes the dependencies and relationships in your project. For Tableau auto exposures, use **View in Tableau** or **Modify integration** from the side panel. [![View from the dbt Catalog in your Project lineage view, displayed with the Tableau icon.](/img/docs/platform-integrations/auto-exposures/explorer-lineage2.png?v=2 "View from the dbt Catalog in your Project lineage view, displayed with the Tableau icon.")](#)View from the dbt Catalog in your Project lineage view, displayed with the Tableau icon. [![View from the dbt Catalog in your Project lineage view, displayed with the Tableau icon.](/img/docs/platform-integrations/auto-exposures/explorer-lineage.png?v=2 "View from the dbt Catalog in your Project lineage view, displayed with the Tableau icon.")](#)View from the dbt Catalog in your Project lineage view, displayed with the Tableau icon. #### Orchestrate exposures [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") [Orchestrate exposures](https://docs.getdbt.com/docs/platform-integrations/orchestrate-exposures.md) using the dbt [Cloud job scheduler](https://docs.getdbt.com/docs/deploy/job-scheduler.md) to proactively refresh the underlying data sources (extracts) that power your Tableau Workbooks. This beta capability requires deployment jobs on [**v1 Latest**](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) with dbt v1 and *isn't* available on [**v2 Stable**](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). For setup details, refer to [Orchestrate exposures](https://docs.getdbt.com/docs/platform-integrations/orchestrate-exposures.md). * Orchestrating exposures with a `dbt build` job ensures that downstream exposures, like Tableau extracts, are updated regularly and automatically. * You can control the frequency of these refreshes by configuring environment variables. To set up and proactively run exposures with the dbt job scheduler, refer to [Orchestrate exposures](https://docs.getdbt.com/docs/platform-integrations/orchestrate-exposures.md). --- ### Set up Cost Insights dbt platform | Enterprise, Enterprise+ This guide walks you through setting up Cost Insights to track warehouse compute costs and cost reductions from dbt State and state-aware orchestration across your dbt projects and models. #### Prerequisites Before setting up Cost Insights, ensure you have: * An administrator role. * A supported data warehouse: * Snowflake * BigQuery * Databricks * Amazon Redshift [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") To set up Cost Insights, follow these steps: 1. [Assign required permissions.](#assign-required-permissions) 2. [Configure platform metadata credentials.](#configure-platform-metadata-credentials) 3. [(Optional) Configure Cost Insights settings.](#configure-cost-insights-settings-optional) 4. [(Optional) Enable dbt State or state-aware orchestration in your job settings.](#enable-dbt-state-or-state-aware-orchestration-optional) After completing these setup steps, you can view cost and optimization data across multiple areas of the dbt platform. Refer to [Explore cost data](https://docs.getdbt.com/docs/explore/explore-cost-data.md) to learn more about the Cost Insights section and how to use it. #### Assign required permissions Users with the following [permission sets](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) can view cost data by default: * Account Admin * Account Viewer * Cost Insights Admin * Cost Insights Viewer * Database Admin * Git Admin * Job Admin * Project Creator * Team Admin For more information on how to assign permissions to users, refer to [About user access](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md). #### Configure platform metadata credentials 1. Click your account name at the bottom of the left-side menu and click **Account settings**. 2. Under **Settings**, go to **Connections**. 3. Select an existing connection or create a new connection for the project where you want to enable Cost Insights. 4. Configure platform metadata credentials for your connection. 1. Go to the **Platform metadata credentials** section. Depending on your current configuration, you may see one of the following: * If no platform metadata credentials are configured yet, the credentials form is immediately in edit mode. * If you previously canceled the form and see **Add credentials**, click the button to reopen the form. * If you have multiple connections that reference the same account identifier, you will only be prompted to add platform metadata credentials to one of them. Other connections using the same account identifier will display a message indicating that platform metadata credentials are already configured. 2. Add credentials with permissions to the warehouse tables. Expand each connection to see the permissions required.  Snowflake * `read` permissions to the [`ORGANIZATION_USAGE`](https://docs.snowflake.com/en/sql-reference/organization-usage) and [`ACCOUNT_USAGE`](https://docs.snowflake.com/en/sql-reference/account-usage) schemas * A Snowflake database role assigned the following access: * `ACCOUNT_USAGE.QUERY_HISTORY` * `ACCOUNT_USAGE.QUERY_ATTRIBUTION_HISTORY` * `ACCOUNT_USAGE.ACCESS_HISTORY` * `ACCOUNT_USAGE.WAREHOUSE_METERING_HISTORY` * `ORGANIZATION_USAGE.USAGE_IN_CURRENCY_DAILY` (Optional) * `ACCOUNT_USAGE.QUERY_METERING_HISTORY` (Optional; required for [Adaptive Warehouse](https://docs.snowflake.com/en/user-guide/warehouses-adaptive) cost attribution) If `QUERY_METERING_HISTORY` access is not granted, Adaptive Warehouse queries appear as $0 in Cost Insights and a warning is shown in the connection test. For more information, refer to the [Snowflake documentation](https://docs.snowflake.com/en/sql-reference/account-usage/query_metering_history).  BigQuery * `bigquery.datasets.get` * `bigquery.jobs.create` * `bigquery.jobs.listAll`  Databricks * Access to a [Unity Catalog workspace](https://docs.databricks.com/aws/en/admin/system-tables/#requirements) * `USE` permissions on the catalog and schema * `SELECT` permissions on the following system tables: * [`system.billing.usage`](https://docs.databricks.com/aws/en/admin/system-tables/billing) * [`system.billing.list_prices`](https://docs.databricks.com/aws/en/admin/system-tables/pricing) * [`system.query.history`](https://docs.databricks.com/aws/en/admin/system-tables/query-history) For more information, refer to the Databricks documentation on [granting access to system tables](https://docs.databricks.com/aws/en/admin/system-tables/#grant-access-to-system-tables).  Amazon Redshift[Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") By default, Redshift users can only view their own queries. dbt must be able to query all users' queries in `SYS_QUERY_HISTORY` to attribute costs across your dbt runs. Grant one of the following permissions to the platform metadata credentials user: * **`sys:monitor` role** (recommended): ```sql GRANT ROLE sys:monitor TO <user>; ``` * **Unrestricted syslog access**: ```sql ALTER USER <user> SYSLOG ACCESS UNRESTRICTED; ``` For more information, refer to the [Amazon Redshift documentation on enhanced query monitoring permissions](https://docs.aws.amazon.com/redshift/latest/mgmt/metrics-enhanced-query-monitoring.html#metrics-enhanced-query-monitoring-permissions). dbt verifies cross-user visibility during the connection test. If the credentials can only see their own queries, the test fails and cost data will not be processed for that environment. 5. Verify that **Cost insights** is enabled under **Features**. This feature is enabled by default when you configure platform metadata credentials. 6. Click **Save**. #### Configure Cost Insights settings (optional) **Note:** This step is required for Amazon Redshift users. Without a configured price, costs will appear as $0. By default, dbt uses standard warehouse pricing, which you can override if you have custom pricing agreements. Databricks and Amazon Redshift do not have default values. The default values vary by warehouse: | Warehouse | Default values | | ---------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | | [Snowflake](https://www.snowflake.com/en/pricing-options/) | `price_per_credit` = $3 | | [BigQuery](https://cloud.google.com/bigquery/pricing) | `price_per_slot_hour` = $0.04, `price_per_tib` = $6.25 | | [Databricks](https://docs.databricks.com/aws/en/admin/system-tables/pricing) | dbt queries the `list_prices` system table directly, so there is no default value. | | [Amazon Redshift Serverless](https://aws.amazon.com/redshift/pricing/) | `rpu_price_per_hour` — no default value; costs appear as $0 until configured. | | [Amazon Redshift Provisioned](https://aws.amazon.com/redshift/pricing/) | `node_price_per_hour` — no default value; costs appear as $0 until configured. | <br /> To change the default value: 1. Click your account name at the bottom of the left-side menu and click **Account settings**. 2. Under **Settings**, go to **Connections**. 3. Select the connection where you want to configure Cost Insights settings. 4. Go to the **Cost Insights settings** section. 5. Enter your custom value in the **Price per credit** field. 6. Click **Save**. These custom values will apply to all future cost calculations for this connection. If you clear these values, they will reset to the default warehouse pricing. #### Enable dbt State (optional) State-aware orchestration is now dbt State [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) works with all engines and environments: dbt v1, dbt platform, and dbt v2 If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). Cost Insights displays cost data for your dbt models and jobs without dbt State. However, to understand the impact of optimizations and see cost reductions from model and test reuse, you must enable dbt State in your jobs. For steps on how to enable it, see [Setting up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md). Cost Insights also reflects cost reductions and efficiency gains from state-aware orchestration if you have it enabled. note For accounts already using dbt State or state-aware orchestration before Cost Insights is enabled, at least one full model build must occur within the last 10 days to establish a baseline for cost reduction calculations. If you don't see cost reduction data, try running a full build to establish the baseline. #### Disable Cost Insights To disable Cost Insights, you must have an administrator role. 1. Click your account name at the bottom of the left-side menu and click **Account settings**. 2. Under **Settings**, go to **Connections**. 3. Select the connection where you want to disable Cost Insights. 4. Go to **Platform metadata credentials** and click **Edit**. 5. Go to the **Features** section and clear the **Cost Insights** option. 6. Click **Save**. --- ### Set up remote MCP dbt platform The remote MCP server uses an HTTP connection and makes calls to dbt-mcp hosted on the cloud-based dbt platform. The self-hosted installation is not required for remote MCP use and is ideal for data consumption use cases. [![Remote dbt MCP server architecture](/img/mcp/remote-dbt-mcp.jpg?v=2 "Remote dbt MCP server architecture")](#)Remote dbt MCP server architecture #### When to use remote MCP The remote MCP server is the ideal choice when: * You don't want to or are restricted from installing additional software (`uvx`, `dbt-mcp`) on your system. * Your primary use case is *consumption-based*: querying metrics, exploring metadata, viewing lineage. * You need access to Semantic Layer, Administrative, and Discovery APIs without maintaining a local dbt project. * You don't need to execute CLI commands. Remote MCP does not support self-hosted dbt CLI commands (`dbt run`, `dbt build`, `dbt test`, and more). If you need to execute dbt commands, use the [self-hosted MCP server](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md) instead. info Only [`text_to_sql`](https://docs.getdbt.com/docs/dbt-ai/mcp-available-tools.md) consumes your dbt Copilot action allotment. Other MCP tools do not. When your account runs out of dbt Copilot actions, the remote MCP server blocks every tool that runs through it, including tools invoked from a self-hosted MCP server and [proxied](https://github.com/dbt-labs/dbt-mcp/blob/main/src/dbt_mcp/tools/toolsets.py#L24) to remote MCP, such as SQL and remote dbt v2 tools. If you reach your dbt Copilot actions limit, remote MCP tools remain unavailable until the limit resets. If you need help, contact your account manager. #### Choose your auth method | If you need... | Use... | | ------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------- | | Fastest first-time setup and your MCP client supports OAuth for HTTP servers (including `execute_sql`) | **[OAuth (remote)](#oauth-remote-mcp)**<br />Available in public beta for Starter, Enterprise, and Enterprise+ accounts | | `execute_sql` with token-based auth, automation, shared setup, or clients without OAuth | **Token-based** with a [personal access token (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) | | Shared or team setup | **Service token** (token-based) | | CI or automation | **Service token** (token-based) | Service tokens do not work for `execute_sql` If you connect with a token instead of OAuth, `execute_sql` only works with a [personal access token (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md). Service tokens won't work. To run SQL without creating a PAT, use [OAuth (remote MCP)](#oauth-remote-mcp) instead. #### OAuth (remote MCP) [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles")[Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") OAuth lets you connect to the remote MCP server without copying API tokens into your MCP client, when your client supports OAuth for HTTP-based MCP servers. ##### Prerequisites * Starter, Enterprise, or Enterprise+ account * An MCP client that supports OAuth for remote (HTTP) MCP servers. * Your **MCP URL** from **Account settings** → **Access URLs** → **MCP Endpoint URL** in dbt platform. Check out the next section [MCP URL](#mcp-url) for more information. AI features are enabled by default. Admins can [turn them off or back on anytime](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md). Turning them off doesn't disable remote MCP — it only hides the [`text_to_sql` tool](https://docs.getdbt.com/docs/dbt-ai/mcp-available-tools.md#tools-that-require-ai-features). ##### MCP URL You can copy your full **MCP URL** from **Account settings** → **Access URLs** → **MCP Endpoint URL** in dbt platform, and paste it directly into your AI tool.  Build your own MCP URL We recommend using the MCP URL from **Account settings** → **Access URLs** → **MCP Endpoint URL** in dbt platform. However, if you want to build your own MCP URL, use your **Access URL** from **Account settings** in dbt platform. The remote MCP endpoint is `https://YOUR_DBT_HOST_URL/api/ai/v1/mcp`. Replace `YOUR_DBT_HOST_URL` with your hostname only (no `https://`). For default hosts, multi-cell accounts, and regions, see [Access, Regions, & IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). ##### How it works 1. In your MCP client, navigate to the connector or integrations settings and add your MCP URL (refer to [MCP URL](#mcp-url)). For example, if you used Claude or ChatGPT, you would go to: * **Claude (web)**: **Customize** → **Connectors** → **+** → **Add custom connector** * **ChatGPT**: **Settings** → **Apps** → **Create App** * For Claude Desktop and Claude Code, refer to [Integrate Claude with dbt MCP](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-claude.md). 2. When prompted in your MCP client, complete sign-in in the browser and approve the requested scopes on the consent screen. 3. Return to your MCP client; subsequent requests use the OAuth session according to your client's behavior. You can use `execute_sql` with OAuth. Add your MCP URL to your client and sign in when your MCP client prompts you. You don't need a personal access token or extra headers. ##### Register your MCP client OAuth requires every client to be registered with dbt platform. There are two paths: * **Dynamic registration (default)** — Clients that implement [Dynamic Client Registration (RFC 7591)](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#dynamic-registration) self-register the first time a user connects. No admin action required. Most modern MCP clients (such as Claude Desktop, Claude Code, Cursor, and VS Code) support this. * **Manual registration** — For clients that don't support dynamic registration, an account admin registers the client in **Account settings → Integrations → App integrations**. Manually registered clients use [PKCE (RFC 7636)](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#manual-registration) instead of a client secret. Refer to [Manual registration](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#manual-registration) for the admin walkthrough. Both types appear in **App integrations** in dbt platform, where admins can review and audit each connected client. ##### Scopes you'll consent to The first time you connect, dbt shows a consent screen listing the scopes (the specific permissions the client is allowed to use) the MCP client is requesting. Scopes act as a **filter on your existing permissions** — they don't grant new access. You can also choose whether the client gets access to all projects or only selected projects. For the full list of scopes and what each one allows, refer to [Scopes and consent](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md#scopes-and-consent). For sessions, refresh tokens, revoking access, and audit logging, refer to [Connect apps with OAuth](https://docs.getdbt.com/docs/platform/manage-access/connect-apps-oauth.md). ##### Limitations * Remote MCP doesn't support self-hosted dbt CLI commands (like `dbt run`, `dbt build`, `dbt test`, and more) or local project access; use the [self-hosted MCP server](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md) for those workflows. For client-specific steps, refer to [Integrate Claude with MCP](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-claude.md), [Integrate Cursor with MCP](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-cursor.md), [Integrate Snowflake Cortex with MCP](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-snowflake-cortex.md), or [Integrate VS Code with MCP](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-vscode.md). #### Token-based authentication Token-based authentication lets you connect to the remote MCP server without OAuth by passing a PAT or service token in your MCP client config. Use it when your client doesn't support OAuth for HTTP-based MCP servers, when you need a shared or CI setup, or when you need `execute_sql`, which requires a PAT. ##### Setup instructions AI features are enabled by default. Admins can [turn them off or back on anytime](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md). Turning them off doesn't disable remote MCP — it only hides the [`text_to_sql` tool](https://docs.getdbt.com/docs/dbt-ai/mcp-available-tools.md#tools-that-require-ai-features). 1. Obtain the following information from dbt platform: * **dbt platform host**: Use this to form the full URL. For example, replace `YOUR_DBT_HOST_URL` here: `https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/`. It may look like: `https://cloud.getdbt.com/api/ai/v1/mcp/`. If you have a multi-cell account, the host URL will be in the `ACCOUNT_PREFIX.us1.dbt.com` format. For more information, refer to [Access, Regions, & IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). * **Production environment ID**: From **Orchestration** → **Environments** in dbt platform. Use this to set an `x-dbt-prod-environment-id` header. Refer to [How to find your dbt MCP IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md#dbt-prod-env-id) for step-by-step instructions. * **Token**: Generate either a personal access token or a service token. To fully utilize remote MCP, the token must have Semantic Layer and Developer permissions. * If you plan to use `execute_sql` with token-based auth, you must use a [Personal Access Token (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md). Service tokens *do not* work for this tool. For other tools that require `x-dbt-user-id`, a PAT is also required. 2. For the remote MCP, you will pass on headers through the JSON blob to configure required fields: ###### Configuration for APIs and SQL tools | Header | Required | Description | | ------------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Authorization | Required | Your [personal access token (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) or [service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) from the dbt platform.<br />**Note**: When using the Semantic Layer, we recommended to use a PAT. If you're using a service token, make sure that it has at least `Semantic Layer Only`, `Metadata Only`, and `Developer` permissions.<br /><br />The value must be in the format `Token YOUR_DBT_ACCESS_TOKEN` or `Bearer YOUR_DBT_ACCESS_TOKEN`, replacing `YOUR_DBT_ACCESS_TOKEN` with your actual token. | | x-dbt-prod-environment-id | Required | Your dbt platform production environment ID. Refer to [How to find your dbt MCP IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md#dbt-prod-env-id) for step-by-step instructions. | ###### Additional configuration for SQL tools | Header | Required | Description | | ------------------------ | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | x-dbt-dev-environment-id | Required for `execute_sql` | Your dbt platform development environment ID. Refer to [How to find your dbt MCP IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md#dbt-dev-env-id) for step-by-step instructions. | | x-dbt-user-id | Required for `execute_sql` | Your dbt platform user ID. Refer to [Where can I find my user ID?](https://docs.getdbt.com/faqs/Accounts/find-user-id.md) for details. | ###### Additional configuration for dbt v2 tools By default, dbt v2 tools use the environment you set in `x-dbt-prod-environment-id` for model and table metadata. | Header | Required | Description | | -------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | x-dbt-dev-environment-id | Required | Your dbt platform development environment ID. Refer to [How to find your dbt MCP IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md#dbt-dev-env-id) for step-by-step instructions. | | x-dbt-user-id | Required | Your dbt platform user ID. Refer to [Where can I find my user ID?](https://docs.getdbt.com/faqs/Accounts/find-user-id.md) for details. | | x-dbt-fusion-disable-defer | Optional | Default: `false`. When set to `true`, dbt v2 tools will not defer to the production environment and use the models and table metadata from the development environment (`x-dbt-dev-environment-id`) instead. | ###### Configuration to disable tools | Header | Required | Description | | ---------------------- | -------- | ---------------------------------------------------------------------------------------------------- | | x-dbt-disable-tools | Optional | A comma-separated list of tools to disable. For instance: `get_all_models,text_to_sql,list_entities` | | x-dbt-disable-toolsets | Optional | A comma-separated list of toolsets to disable. For instance: `semantic_layer,sql,discovery` | 3. After establishing which headers you need, you can follow the [examples](https://github.com/dbt-labs/dbt-mcp/tree/main/examples) to create your own agent. #### Examples The MCP protocol is programming language and framework agnostic, so use whatever helps you build agents. If you use [OAuth (remote MCP)](#oauth-remote-mcp) in Beta, you only need your MCP URL. If you use token-based authentication, add the headers in the examples below. Configuration varies by client — select your tool in the following tabs and replace the placeholder values with your own: ##### OAuth ##### Claude Code ```json { "mcpServers": { "dbt": { "type": "http", "url": "https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/" } } } ``` ##### Cursor ```json { "mcpServers": { "dbt": { "url": "https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/" } } } ``` ##### Token-based ##### Claude Code ```json { "mcpServers": { "dbt": { "type": "http", "url": "https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/", "headers": { "Authorization": "Token YOUR_DBT_ACCESS_TOKEN", "x-dbt-prod-environment-id": "DBT_PROD_ENV_ID", "x-dbt-user-id": "DBT_USER_ID", "x-dbt-dev-environment-id": "DBT_DEV_ENV_ID" } } } } ``` ##### Cursor ```json { "mcpServers": { "dbt": { "url": "https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/", "headers": { "Authorization": "Token YOUR_DBT_ACCESS_TOKEN", "x-dbt-prod-environment-id": "DBT_PROD_ENV_ID", "x-dbt-user-id": "DBT_USER_ID", "x-dbt-dev-environment-id": "DBT_DEV_ENV_ID" } } } } ``` ##### Gemini ```json { "mcpServers": { "dbt": { "httpUrl": "https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/", "headers": { "Authorization": "Token YOUR_DBT_ACCESS_TOKEN", "x-dbt-prod-environment-id": "DBT_PROD_ENV_ID", "x-dbt-user-id": "DBT_USER_ID", "x-dbt-dev-environment-id": "DBT_DEV_ENV_ID" } } } } ``` Use numeric IDs, not full URLs Header values like `x-dbt-prod-environment-id` and `x-dbt-user-id` expect numeric IDs, not full URLs. The host in the `url` field should include `https://`, but ID headers must be integers only: ```bash # ✅ Correct "url": "https://cloud.getdbt.com/api/ai/v1/mcp" "x-dbt-prod-environment-id": "54321" "x-dbt-user-id": "123" # ❌ Wrong — don't paste full URLs into ID headers "x-dbt-prod-environment-id": "https://cloud.getdbt.com/deploy/12345/projects/67890/environments/54321" "x-dbt-user-id": "https://cloud.getdbt.com/settings/profile" ``` For other MCP clients (Codex, Windsurf, and so on), refer to your client's MCP configuration docs for the correct key format. For self-hosted MCP, use environment variables to configure your setup; refer to the [Environment variables reference](https://docs.getdbt.com/docs/dbt-ai/mcp-environment-variables.md). #### Related docs Step-by-step client setup (including Cursor, VS Code, and Claude) is in: * [Integrate Cursor with MCP](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-cursor.md) * [Integrate VS Code with MCP](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-vscode.md) * [Integrate Claude with MCP](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-claude.md) * [Integrate Snowflake Cortex with MCP](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-snowflake-cortex.md) --- ### Set up self-hosted MCP Local development [The self-hosted dbt MCP server](https://github.com/dbt-labs/dbt-mcp) runs on your machine and supports dbt v1, dbt v2, and dbt platform CLI. You can use it with or without a dbt platform account. No clone required You don't need to clone the dbt-mcp repository to use self-hosted MCP. [Install uv](https://docs.astral.sh/uv/getting-started/installation/) and run `uvx dbt-mcp`, which fetches and runs dbt-mcp for you. If you'd like to contribute to dbt MCP, clone the [dbt-mcp repo](https://github.com/dbt-labs/dbt-mcp) and contribute away! #### Tool requirements at a glance Use this table to understand what each toolset needs and whether it works with or without a dbt platform account: | Toolset | Required variables | Works with dbt platform | Works without dbt platform | | ----------------------------- | -------------------------------------------------------------- | ----------------------- | -------------------------- | | dbt CLI | `DBT_PROJECT_DIR`, `DBT_PATH` | Yes | Yes | | Semantic Layer | `DBT_HOST`, `DBT_TOKEN`, `DBT_PROD_ENV_ID` | Yes | No | | Discovery API | `DBT_HOST`, `DBT_TOKEN`, `DBT_PROD_ENV_ID` | Yes | No | | Admin API | `DBT_HOST`, `DBT_TOKEN`, `DBT_ACCOUNT_ID` | Yes | No | | SQL execution (`execute_sql`) | Personal access token, `DBT_DEV_ENV_ID`, `DBT_USER_ID` | Yes | No | | Codegen | `DBT_PROJECT_DIR`, `DBT_PATH`, and `DISABLE_DBT_CODEGEN=false` | Yes | Yes | | LSP / dbt v2 | `DBT_PROJECT_DIR`, `DBT_PATH`, and the dbt VS Code extension | Yes | Yes | Toolsets auto-disable when required variables are missing If a required variable is not set, dbt-mcp will automatically disable that toolset rather than error. For example, if `DBT_HOST` is not configured, the Semantic Layer, Discovery, and Admin API toolsets won't be available. To confirm which toolsets are active, set `DBT_MCP_LOG_LEVEL=DEBUG` in your environment and check the [server logs](#debug-configurations). #### Prerequisites * [Install uv](https://docs.astral.sh/uv/getting-started/installation/) to be able to run `dbt-mcp` and [related dependencies](https://github.com/dbt-labs/dbt-mcp/blob/main/pyproject.toml) into an isolated virtual environment. * Have a local dbt project (if you want to use dbt CLI commands). #### Choose your auth method If you're connecting to dbt platform features (Semantic Layer, Discovery API, Admin API, or SQL execution), you need to authenticate. Use this table to choose the right method: | If you need... | Use... | | ------------------------ | ------------------------------------------------------------------------------- | | Fastest first-time setup | **OAuth** | | `execute_sql` tool | **Personal Access Token (PAT)**. Service tokens *do not* work for `execute_sql` | | Shared or team setup | **Service token** | | CI or automation | **Service token** | `execute_sql` requires a PAT The `execute_sql` tool does *not* work with service tokens. You must use a [Personal Access Token (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) for `DBT_TOKEN` when using this tool. #### Setup options Choose the setup method that best fits your workflow: ##### OAuth authentication with dbt platform [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") This method uses OAuth to authenticate with your dbt platform account. It's the simplest setup and doesn't require managing tokens or environment variables manually. Static subdomains required Only accounts with static subdomains (for example, `abc123` in `abc123.us1.dbt.com`) can use OAuth with MCP servers. Follow [these](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) instructions to find your account subdomain. If your account does not have a subdomain, contact support for more information. ###### Configuration options ##### dbt platform only This option is for users who only want dbt platform features (Discovery API, Semantic Layer, job management) without self-hosted dbt platform CLI commands. When you use only the dbt platform, the CLI tools are automatically disabled. You can find the `DBT_HOST` field value in your dbt platform account information under **Access URLs**. ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "YOUR-ACCESS-URL" } } } } ``` **Note:** Replace `YOUR-ACCESS-URL` with your Access URL hostname (for example, `abc123.us1.dbt.com`). Both `abc123.us1.dbt.com` and `https://abc123.us1.dbt.com` are accepted. This enables OAuth authentication without requiring self-hosted dbt installation. ##### dbt platform + CLI This option is for users who want both dbt platform CLI commands and dbt platform features (Discovery API, Semantic Layer, job management). The `DBT_PROJECT_DIR` and `DBT_PATH` fields are required for CLI access. You can find the `DBT_HOST` field value in your dbt platform account information under **Access URLs**. ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "YOUR-ACCESS-URL", "DBT_PROJECT_DIR": "/path/to/project", "DBT_PATH": "/path/to/dbt/executable" } } } } ``` **Note:** Replace `YOUR-ACCESS-URL` with your Access URL hostname (for example, `abc123.us1.dbt.com`). Both `abc123.us1.dbt.com` and `https://abc123.us1.dbt.com` are accepted. This enables OAuth authentication. Once configured, your session connects to the dbt platform account, starts the OAuth authentication workflow, and then opens your account where you can select the project you want to reference. [![Select your dbt platform project](/img/mcp/select-project.png?v=2 "Select your dbt platform project")](#)Select your dbt platform project After completing OAuth setup, skip to [Test your configuration](#optional-test-your-configuration). ##### CLI only (no dbt platform) This option runs the MCP server locally and connects it to your local dbt project using `DBT_PROJECT_DIR` and `DBT_PATH`. If you're using dbt v1 or dbt v2 and don't need access to dbt platform features (Discovery API, Semantic Layer, Administrative API), you can set up local MCP with just your dbt project information. Add this configuration to your MCP client (refer to the specific [integration guides](#set-up-your-mcp-client) for exact file locations): ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_PROJECT_DIR": "/path/to/your/dbt/project", "DBT_PATH": "/path/to/your/dbt/executable" } } } } ``` ###### Locating your paths Follow the appropriate instructions for your OS to locate your path:  macOS/Linux * **DBT\_PROJECT\_DIR**: The full path to your dbt project folder * Example: `/Users/yourname/dbt-projects/my_project` * This is the folder containing your `dbt_project.yml` file. * **DBT\_PATH**: Find your dbt executable path by running in terminal: ```bash which dbt ``` * Example output: `/opt/homebrew/bin/dbt` * Use this exact path in your configuration.  Windows * **DBT\_PROJECT\_DIR**: The full path to your dbt project folder * Example: `C:\Users\yourname\dbt-projects\my_project` * This is the folder containing your `dbt_project.yml` file. * Use forward slashes or escaped backslashes: `C:/Users/yourname/dbt-projects/my_project` * **DBT\_PATH**: Find your dbt executable path by running in Command Prompt or PowerShell: ```bash where dbt ``` * Example output: `C:\Python39\Scripts\dbt.exe` * Use forward slashes or escaped backslashes: `C:/Python39/Scripts/dbt.exe` After completing this setup, skip to [Test your configuration](#optional-test-your-configuration). ##### Environment variable configuration If you need to configure multiple environment variables or prefer to manage them separately, you can use an `.env` file or pass them inline. You only need to supply the variables relevant to your setup — dbt CLI variables for CLI-only use, or dbt platform variables for platform features. For the complete list of variables, enabling or disabling toolsets, and logging options, see the [Environment variables reference](https://docs.getdbt.com/docs/dbt-ai/mcp-environment-variables.md). Where to put the `.env` file Create the `.env` file in your *dbt project root* (the same folder as `dbt_project.yml`). When referencing it with `--env-file`, always *use an absolute path* so your MCP client can find it reliably. For example, `/absolute/path/to/your-dbt-project/.env`. Pick the `.env` example that matches your setup. Only include the variables you need: ##### CLI only Use this if you're running dbt commands locally and don't need dbt platform features (Discovery API, Semantic Layer, etc.): ```code DBT_PROJECT_DIR=/path/to/your/dbt/project DBT_PATH=/path/to/your/dbt/executable ``` ##### dbt platform only Use this if you only need dbt platform features and won't run dbt commands: ```code DBT_HOST=cloud.getdbt.com DBT_TOKEN=dbtc_your_token DBT_PROD_ENV_ID=12345 ``` ##### CLI and dbt platform (most common) Use this if you want both dbt CLI commands and dbt platform features: ```code DBT_PROJECT_DIR=/path/to/your/dbt/project DBT_PATH=/path/to/your/dbt/executable DBT_HOST=cloud.getdbt.com DBT_TOKEN=dbtc_your_token DBT_PROD_ENV_ID=12345 ``` ##### All variables A complete reference of all available variables. Most setups only need a subset of these — refer to the [API and SQL tool settings](#api-and-sql-tool-settings) and [dbt CLI settings](#dbt-cli-settings) tables for details on each variable. ```code DBT_HOST=cloud.getdbt.com DBT_PROD_ENV_ID=your-production-environment-id DBT_DEV_ENV_ID=your-development-environment-id DBT_USER_ID=your-user-id DBT_ACCOUNT_ID=your-account-id DBT_TOKEN=your-service-token DBT_PROJECT_DIR=/path/to/your/dbt/project DBT_PATH=/path/to/your/dbt/executable ``` ###### How to pass environment variables to dbt-mcp Here are some of the ways to pass environment variables. We recommend using an `.env` file with `--env-file` for most setups: ##### .env file with \`--env-file\` **1. `.env` file with `--env-file` (recommended)** (make sure to use an absolute path in `args`): Reference the file using an absolute path in your MCP client config: ```json { "mcpServers": { "dbt": { "command": "uvx", "args": [ "--env-file", "/absolute/path/to/your-dbt-project/.env", "dbt-mcp" ] } } } ``` Replace `/absolute/path/to/your-dbt-project` with the full path to the folder containing your `dbt_project.yml`. ##### Inline in the MCP client config **2. Inline in the MCP client config**: Pass variables directly in the `env` field, replacing the values with your actual ones. This keeps everything in one file but means tokens are stored in your client config: ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "cloud.getdbt.com", "DBT_TOKEN": "your-token-here", "DBT_PROD_ENV_ID": "12345", "DBT_PROJECT_DIR": "/path/to/project", "DBT_PATH": "/path/to/dbt" } } } } ``` ##### Shell environment variables **3. Shell environment variables** Export variables in your terminal before starting the MCP client, replacing the values with your actual ones: ```bash export DBT_HOST=cloud.getdbt.com export DBT_TOKEN=your-token-here export DBT_PROJECT_DIR=/path/to/project uvx dbt-mcp ``` #### API and SQL tool settings | Environment variable | Required | Description | | -------------------- | -------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `DBT_HOST` | Required | Your dbt platform [instance hostname](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). The default is `cloud.getdbt.com`. For multi-cell and multi-tenant accounts with a static subdomain, use the full hostname — for example, `abc123.us1.dbt.com`. | | `DBT_TOKEN` | Required | Your personal access token or service token from the dbt platform.<br />**Note**: The `execute_sql` tool requires a [Personal Access Token (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) — service tokens do not work for this tool. For Semantic Layer use, a PAT is also recommended. If you're using a service token for other toolsets, make sure it has at least `Semantic Layer Only`, `Metadata Only`, and `Developer` permissions. | | `DBT_ACCOUNT_ID` | Required for Administrative API tools and PAT-based auth | Your [dbt account ID](https://docs.getdbt.com/faqs/Accounts/find-user-id.md). Also required when using a Personal Access Token (PAT) as your `DBT_TOKEN`. | | `DBT_PROD_ENV_ID` | Required | Your dbt platform production environment ID | | `DBT_DEV_ENV_ID` | Optional | Your dbt platform development environment ID | | `DBT_USER_ID` | Optional | Your dbt platform user ID ([docs](https://docs.getdbt.com/faqs/Accounts/find-user-id.md)) | Use values only, not full URLs A common mistake is pasting a full URL instead of the value. These variables expect hostnames or numeric IDs: ```bash # ✅ Correct DBT_HOST=cloud.getdbt.com # https://cloud.getdbt.com also works DBT_PROD_ENV_ID=54321 DBT_USER_ID=123 # ❌ Wrong — IDs must be numeric, not full URLs DBT_PROD_ENV_ID=https://cloud.getdbt.com/deploy/12345/projects/67890/environments/54321 DBT_USER_ID=https://cloud.getdbt.com/settings/profile ``` **Subdomain prefix configuration example:** ✅ **Correct configuration:** ```bash DBT_HOST=abc123.us1.dbt.com # Use the full hostname including the prefix DBT_ACCOUNT_ID=12345 # Required when using PAT-based auth ``` You don't need to set `MULTICELL_ACCOUNT_PREFIX` or `DBT_HOST_PREFIX`. #### Environment variables The self-hosted dbt-mcp supports all flavors of dbt, including dbt v1 and dbt v2. | Environment variable | Required | Description | Example | | -------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | | `DBT_PROJECT_DIR` | Required | The full path to where the repository of your dbt project is hosted locally. This is the folder containing your `dbt_project.yml` file. | macOS/Linux: `/Users/myname/reponame`<br />Windows: `C:/Users/myname/reponame` | | `DBT_PATH` | Required | The full path to your dbt executable (dbt v1/dbt v2/dbt platform CLI). See the next section for how to find this. | macOS/Linux: `/opt/homebrew/bin/dbt`<br />Windows: `C:/Python39/Scripts/dbt.exe` | | `DBT_CLI_TIMEOUT` | Optional | Configure the number of seconds before your agent will timeout dbt commands. | Defaults to 60 seconds. | ##### Locating your `DBT_PATH` Follow the instructions for your OS to locate your `DBT_PATH`:  macOS/Linux Run this command in your Terminal: ```bash which dbt ``` Example output: `/opt/homebrew/bin/dbt`  Windows Run this command in Command Prompt or PowerShell: ```bash where dbt ``` Example output: `C:\Python39\Scripts\dbt.exe` **Note:** Use forward slashes in your configuration: `C:/Python39/Scripts/dbt.exe` **Additional notes:** * You can set any environment variable supported by your dbt executable, like [the ones supported in dbt v1](https://docs.getdbt.com/reference/global-configs/about-global-configs.md#available-flags). * dbt MCP respects the standard environment variables and flags for usage tracking mentioned [here](https://docs.getdbt.com/reference/global-configs/usage-stats.md). * `DBT_WARN_ERROR_OPTIONS='{"error": ["NoNodesForSelectionCriteria"]}'` is automatically set so that the MCP server knows if no node is selected when running a dbt command. You can overwrite it if needed, but it provides a better experience when calling dbt from the MCP server, ensuring the tool selects valid nodes. #### Controlling tool access dbt-mcp has two modes for controlling which tools are available. Pick one approach: * disable mode (turn off what you don't want) * enable mode (turn on only what you want) Which mode should I use? * Disable mode — Use when you want *most* tools available and just need to turn off a few. This is the default behavior. * Enable mode — Use when you want *only* a small set of tools available (allowlist approach). **Do not mix both modes** for the same toolset. For example, don't set both `DISABLE_SEMANTIC_LAYER=true` and `DBT_MCP_ENABLE_SEMANTIC_LAYER=true` together — the behavior may be unpredictable. ##### Disable mode (default) All tools are available by default. Set any of these to `true` to turn off a toolset: | Name | Default | Description | | ----------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `DISABLE_DBT_CLI` | `false` | Disable dbt v1, dbt platform CLI, and dbt v2 MCP tools. | | `DISABLE_SEMANTIC_LAYER` | `false` | Disable dbt Semantic Layer MCP tools. | | `DISABLE_DISCOVERY` | `false` | Disable dbt Discovery API MCP tools. | | `DISABLE_ADMIN_API` | `false` | Disable dbt Administrative API MCP tools. | | `DISABLE_SQL` | `true` | SQL MCP tools are disabled by default. Set to `false` to enable. | | `DISABLE_DBT_CODEGEN` | `true` | [dbt codegen MCP tools](https://docs.getdbt.com/docs/dbt-ai/mcp-available-tools.md#codegen-tools) are disabled by default. Set to `false` to enable (requires dbt-codegen package). | | `DISABLE_LSP` | `false` | Disable dbt LSP/dbt v2 MCP tools. | | `DISABLE_MCP_SERVER_METADATA` | `true` | MCP server metadata tools (like `get_mcp_server_version`) are disabled by default. Set to `false` to enable. | | `DISABLE_TOOLS` | `""` | A comma-separated list of specific tool names to disable. | ##### Enable mode Use `DBT_MCP_ENABLE_*` variables when you want to explicitly allowlist which toolsets are available. When an enable variable is set, only the enabled toolsets will be active: | Name | Default | Description | | ------------------------------- | ------- | -------------------------------------------------------- | | `DBT_MCP_ENABLE_DBT_CLI` | Not set | Set to `true` to enable dbt CLI tools. | | `DBT_MCP_ENABLE_SEMANTIC_LAYER` | Not set | Set to `true` to enable Semantic Layer tools. | | `DBT_MCP_ENABLE_DISCOVERY` | Not set | Set to `true` to enable Discovery API tools. | | `DBT_MCP_ENABLE_ADMIN_API` | Not set | Set to `true` to enable Administrative API tools. | | `DBT_MCP_ENABLE_SQL` | Not set | Set to `true` to enable SQL tools. | | `DBT_MCP_ENABLE_DBT_CODEGEN` | Not set | Set to `true` to enable dbt codegen tools. | | `DBT_MCP_ENABLE_LSP` | Not set | Set to `true` to enable LSP/dbt v2 tools. | | `DBT_MCP_ENABLE_TOOLS` | Not set | A comma-separated list of specific tool names to enable. | ##### Precedence When multiple variables are set, they are evaluated in this order (highest priority first): 1. `DBT_MCP_ENABLE_TOOLS` (enable specific tools by name) 2. `DISABLE_TOOLS` (disable specific tools by name) 3. Toolset enable (`DBT_MCP_ENABLE_*=true`) 4. Toolset disable (`DISABLE_*=true`) 5. Default behavior #### (Optional) Test your configuration In your command line tool, run the following to test your setup: **If using the `env` field in JSON:** ```bash export DBT_PROJECT_DIR=/path/to/project export DBT_PATH=/path/to/dbt uvx dbt-mcp ``` **If using an `.env` file:** (use an absolute path, for example to `.env` in your dbt project root) ```bash uvx --env-file /absolute/path/to/your-dbt-project/.env dbt-mcp ``` If there are no errors, your configuration is correct. #### Set up your MCP client After completing your configuration, follow the specific integration guide for your chosen tool: * [Claude](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-claude.md) * [Cursor](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-cursor.md) * [VS Code](https://docs.getdbt.com/docs/dbt-ai/integrate-mcp-vscode.md) #### Debug configurations These settings allow you to customize the MCP server’s logging level to help with diagnosing and troubleshooting. | Name | Default | Description | | ------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------ | | `DBT_MCP_LOG_LEVEL` | `INFO` | Environment variable to override the MCP server log level. Options are: `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`. | To see more detail about what’s happening inside the MCP server and help debug issues, you can temporarily set the log level to `DEBUG`. We recommend setting it temporarily to avoid filling up disk space with logs. #### Troubleshooting  Can't find the uvx executable  Can't find the uvx executable **Symptoms:** Error messages like `Could not connect to MCP server dbt-mcp`, `Error: spawn uvx ENOENT`, or `spawn uvx ENOENT` in your MCP client. **Cause:** Your MCP client (like Claude desktop) can't find `uvx` in its PATH because it starts with a limited environment. **Solution:** Use the full path to `uvx` in your configuration. 1. Find the full path: * macOS/Linux: Run `which uvx` in Terminal. * Windows: Run `where uvx` in Command Prompt or PowerShell. 2. Replace `"command": "uvx"` with the full path: ```json { "mcpServers": { "dbt": { "command": "/full/path/to/uvx", "args": ["dbt-mcp"], "env": { } } } } ``` Example on macOS with Homebrew: `"command": "/opt/homebrew/bin/uvx"` For VS Code (`mcp.json`), the same fix applies — replace `uvx` with its full path in the `command` field.  OAuth login not initiating  OAuth login not initiating **Symptoms:** The OAuth browser window never opens, or authentication appears to hang. **Cause:** dbt MCP uses a lock file to avoid repeated authentication. If a previous session left the lock file in place, it can block new authentication attempts. **Solution:** 1. Close your MCP client (Claude Desktop, Cursor, VS Code, etc.). 2. Delete the self-hosted dbt MCP config files: * macOS/Linux: `rm -f ~/.dbt/mcp.yml ~/.dbt/mcp.lock` * Windows: `Remove-Item -Force $env:USERPROFILE\.dbt\mcp.yml, $env:USERPROFILE\.dbt\mcp.lock` 3. Restart your client and try connecting again. If these steps don't resolve the issue, confirm that AI features are enabled on your account. An account admin can enable them in **Account settings** → **Edit** → toggle on **Enable account access to dbt Wizard features**. Refer to [Enable dbt Wizard](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md). For all troubleshooting topics, see [MCP troubleshooting](https://docs.getdbt.com/docs/dbt-ai/mcp-troubleshooting.md). --- ### Setting up dbt State for non-interactive environments Preview ### Setting up dbt State for non-interactive environments [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Login required | Usage-based In a non-interactive environment, dbt runs without a person available to complete authentication manually — for example, CI/CD pipelines (such as GitHub Actions, GitLab CI, and Jenkins) and production orchestration tools (such as Airflow and Prefect). Browser-based authentication isn't possible in these environments. Instead, dbt State authenticates using credentials provided through environment variables, allowing it to continue caching state and optimizing your builds. dbt State automatically detects when it's running in a non-interactive environment. If valid credentials are not provided, dbt State disables itself and displays a warning, allowing your dbt commands to continue without caching. There are two authentication methods depending on your setup: * [**Service account token**](#service-token-dbt-platform) for dbt platform users * [**OAuth client credentials**](#oauth-client-credentials) for standalone dbt State app users #### Service account token For dbt platform users, you can authenticate dbt State with a [service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md). ##### Prerequisites Before you begin, make sure you have: * A dbt platform account. * **Owner** or **Account Admin** permissions to create a service token. * dbt State installed and configured. Refer to [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) for more information. ##### Creating a service token To create a service account token in dbt platform, refer to [Generate service account tokens](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md#generate-service-account-tokens). When adding permissions for the token, assign at least one of the following: * **Owner** * **Account Admin** * **Job Admin** * **Job Creator** * **Job Runner** (recommended; provides the minimum access required for dbt State) * **Developer** ##### Configuring authentication Set the following environment variables in your orchestration environment: ```bash DBT_CLOUD_TOKEN=YOUR_SERVICE_TOKEN DBT_CLOUD_ACCOUNT_HOST=YOUR_ACCOUNT_HOST DBT_CLOUD_ACCOUNT_ID=YOUR_ACCOUNT_ID ``` Replace `YOUR_SERVICE_TOKEN` with your service token, `YOUR_ACCOUNT_HOST` with your [account host](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) (for example, `abc123.us1.dbt.com`), and `YOUR_ACCOUNT_ID` with your numeric account ID. Go to **Account settings** > **Account** to find your account ID and account host (the hostname from the **Access URL** field). #### OAuth client credentials dbt State app retirement The standalone dbt State app (`app.state.dbt.com`) is being retired and is no longer accepting new users. If you have an existing dbt State app account, [create a free dbt platform account](http://us1.dbt.com/register?_dbtsrc=dbt-state) to continue using dbt State — your free trial will be extended upon account creation. If you're using the standalone [dbt State web app](https://app.state.dbt.com/), authenticate with OAuth client credentials. ##### Prerequisites * dbt State installed and configured. Refer to [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) for more information. * A standalone dbt State account at [app.state.dbt.com](https://app.state.dbt.com/). * An **Admin** or **Owner** role in your dbt State organization. Refer to [Roles and tab access](#roles-and-tab-access) for details. #### Roles and tab access The dbt State web app has four tabs under **Organization**: | Tab | Description | | ----------- | --------------------------------------------------------------------------------- | | **Usage** | View your project reuses and compute time saved once dbt State is enabled. | | **Users** | Invite team members and grant or revoke access. | | **Billing** | View daily active target tables (DATTs) for the current billing period. | | **Clients** | Create and manage OAuth clients for CI/CD and other non-interactive environments. | <br /> Your role determines which tabs you can access. | Role | Access | Notes | | ------------- | ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Owner** | Usage, Users, Billing, Clients | The user who created the organization is the Owner by default. An Owner can transfer their role to another user, which demotes the original Owner to Admin. | | **Admin** | Usage, Users, Billing, Clients | — | | **Developer** | Usage | Default role when users are added. | An existing **Owner** or **Admin** can grant or revoke admin access from the **Users** tab. ##### Creating an OAuth client 1. In the [dbt State web app](https://app.state.dbt.com/), navigate to the **Clients** tab. 2. Click **Add OAuth Client**. 3. Enter a name and description for the new client and click **Create**. 4. Copy the client ID and secret to use in your environment configuration. ##### Configuring OAuth authentication Once you have the client ID and secret, set the following environment variables in your environment. Using environment variables is the recommended approach as it keeps sensitive credentials out of your code repository. ```bash DBT_ENGINE_STATE_OAUTH_CLIENT_ID=YOUR_CLIENT_ID DBT_ENV_SECRET_STATE_OAUTH_CLIENT_SECRET=YOUR_CLIENT_SECRET ``` Replace `YOUR_CLIENT_ID` and `YOUR_CLIENT_SECRET` with the values from your OAuth client. #### Verifying dbt State is active 1. Run any dbt transformation job in your orchestrated environment. 2. Check the log output. You should see a message like this, then the specific dbt State step status: ```text dbt State adapter: dbt-state v2.10.1 is enabled ``` #### Related docs * [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) * [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) * [dbt State configs](https://docs.getdbt.com/reference/resource-configs/dbt-state-configs.md) * [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md) --- ### Setting up dbt State Preview ### Setting up dbt State [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Login required | Usage-based This page walks you through setting up dbt State across dbt v1, dbt platform, and dbt v2. #### Prerequisites Before you set up dbt State, make sure you have: * **A supported dbt version**: dbt State is natively available in dbt platform and dbt v2. It's also available as a plugin for dbt v1.7–1.12. * **A supported data platform**: Snowflake, Databricks, BigQuery, or Redshift. More warehouses are on the roadmap. * **A dbt platform account**: Refer to [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md#signing-up-for-dbt-state) for sign-up details, and [dbt State usage and pricing](https://docs.getdbt.com/docs/platform/billing/dbt-state-usage.md) for pricing details. Note that dbt State isn't available on [legacy Starter](https://docs.getdbt.com/docs/platform/billing/plans-and-billing.md#legacy-plans) plan. Please [contact dbt Labs](https://www.getdbt.com/contact) if that applies to you. #### Setting up dbt State Set up dbt State either in the dbt platform or self-hosted dbt by using the following steps depending on how you're using dbt: * Already logged in to dbt platform and managing your account? Use **dbt Account settings**. * Signing up or logging in through the [dbt platform sign-up page](http://us1.dbt.com/register?_dbtsrc=dbt-state)? Use **dbt platform sign-up**. * Using the CLI? Use **dbt v2** or **dbt v1.7-1.12**. ##### dbt Account settings ###### Enabling dbt State on your account **Prerequisite**: You must be an admin in your dbt platform account. To enable dbt State: 1. In your dbt platform account, click your account name in the lower-left corner above your username and click **Account settings**. 2. Under **Settings**, go to **Billing & Usage** > **Usage-based features**. 3. Under the **State** tab, click **Start free trial**. Once started, you cannot pause the trial. After 30 days, you must add a credit card or enterprise contract to continue. For information about how the trial period and billing work, refer to [dbt State trial and billing](https://docs.getdbt.com/docs/deploy/dbt-state-trial.md). Extended trial for state-aware orchestration users If you were using state-aware orchestration prior to June 1, 2026, your dbt State trial will be extended beyond 30 days. If the extension isn’t applied to your account, contact your account team. 4. Review and agree to the terms of service. 5. Click **Start 30-day trial**. 6. Click **Enable dbt State**. 7. Select the jobs to enable dbt State for. You can either enable: * **By environment**: Enables dbt State on all existing jobs within the selected environment at once. New jobs created in that environment will have dbt State enabled automatically. * **By specific jobs**: Enables dbt State on individual jobs. To enable it on additional jobs later, refer to [Enabling dbt State on individual jobs](https://docs.getdbt.com/docs/deploy/dbt-state-enable-jobs.md). 8. Click **Enable dbt State**. For next steps, see: * [Enable dbt State on individual jobs](https://docs.getdbt.com/docs/deploy/dbt-state-enable-jobs.md) * [Enable dbt State in Studio](https://docs.getdbt.com/docs/deploy/dbt-state-enable-studio.md) ##### dbt platform sign-up 1. Go to the [dbt platform sign-up page](http://us1.dbt.com/register?_dbtsrc=dbt-state) to create a new account or log in to an existing one. * If you're new to dbt platform, enter your email address and click **Continue**. 1. Enter your name and password, and agree to the Terms of Service. 2. Click **Continue**. 3. Verify your email address. * If you already have a dbt platform account, log in with your existing credentials. 2. Agree to the dbt State Terms of Service and click **Start 30-day trial**. Once started, you cannot pause the trial. After 30 days, you must add a credit card or enterprise contract to continue. For information about how the trial period and billing work, refer to [dbt State trial and billing](https://docs.getdbt.com/docs/deploy/dbt-state-trial.md). 3. Go to **Orchestration** to create your environments and jobs. For next steps, see: * [Enable dbt State on individual jobs](https://docs.getdbt.com/docs/deploy/dbt-state-enable-jobs.md) * [Enable dbt State in Studio](https://docs.getdbt.com/docs/deploy/dbt-state-enable-studio.md) ##### v2 1. Navigate to your project: ```bash cd to/your/project ``` 2. Log in to dbt platform: ```bash dbt login ``` This opens a browser window where you can log in to or create a dbt platform account. For details on authentication behavior, refer to [`dbt login` with dbt State](https://docs.getdbt.com/reference/commands/login.md?version=2.0#dbt-login-with-dbt-state). 3. If prompted to choose your goal, select **Set up dbt State**. The **Start your dbt State trial** dialog appears. 4. Agree to the dbt State Terms of Service and click **Start 30-day trial**. Once started, you cannot pause the trial. After 30 days, you must add a credit card or enterprise contract to continue. For information about how the trial period and billing work, refer to [dbt State trial and billing](https://docs.getdbt.com/docs/deploy/dbt-state-trial.md). dbt State is now enabled and will run automatically on every `dbt run` or `dbt build`. You can also enable or disable dbt State per run using [CLI flags](https://docs.getdbt.com/reference/global-configs/about-global-configs.md): `--manage-state` or `--no-manage-state`, or set the `DBT_ENGINE_MANAGE_STATE` environment variable. To enable dbt State for everyone on your project, add [`manage_state: true`](https://docs.getdbt.com/reference/global-configs/about-global-configs.md) to the `flags:` block in `dbt_project.yml`: ```yaml flags: manage_state: true ``` ##### dbt v1.7-1.12 dbt State is available as a plugin for dbt v1.7+. If you are running on dbt v1.9 or older, we encourage you to upgrade to a [more recent version with ongoing support](https://docs.getdbt.com/docs/dbt-versions.md#latest-releases). To install the plugin: 1. Navigate to your project: ```bash cd to/your/project ``` 2. Create and activate a virtual environment: ```bash python3 -m venv .venv source .venv/bin/activate ``` 3. Install the dbt State plugin: ```bash pip install dbt-state ``` 4. Run `dbt run` or `dbt build` The first time you execute `dbt run` or `dbt build`, a browser window opens where you can log in to or create a dbt platform account. 5. If prompted to choose your goal, select **Set up dbt State**. The **Start your dbt State trial** dialog appears. 6. Agree to the dbt State Terms of Service and click **Start 30-day trial**. Once started, you cannot pause the trial. After 30 days, you must add a credit card or enterprise contract to continue. For information about how the trial period and billing work, refer to [dbt State trial and billing](https://docs.getdbt.com/docs/deploy/dbt-state-trial.md). dbt State is now enabled and will run automatically on every `dbt run` or `dbt build`. The CLI flags `--manage-state` and `--no-manage-state` are not available in older dbt v1 versions. Use the environment variable (`DBT_ENGINE_ENABLE_STATE`) or project flag (`enable_state`) to enable or disable dbt State. To see how dbt State optimizes your runs, refer to [dbt State usage examples](https://docs.getdbt.com/docs/deploy/dbt-state-examples.md). #### Configuring lag tolerance Lag tolerance allows you to set a tolerance level for older data at the project, environment, or model level. If not configured, `lag_tolerance` defaults to `45m`. We recommend starting with the following Jinja expression: dbt\_project.yml ```yaml models: +state: lag_tolerance: "{{ '4h' if target.name == 'prod' else '7d' }}" ``` In this example, models in the `prod` target rebuild only when upstream data is more than 4 hours old. In all other environments, models wait 7 days before rebuilding. For more details, refer to the [`lag_tolerance` config reference](https://docs.getdbt.com/reference/resource-configs/lag-tolerance.md). tip To help you tune `lag_tolerance` values, the **dbt State** page on the dbt platform provides [lag tolerance recommendations](https://docs.getdbt.com/docs/deploy/dbt-state-interface.md#lag-tolerance-recommendations) based on your models' 30-day build history, so you can see which models would benefit from a higher tolerance. #### Inviting team members The more team members you have using dbt State, the better it gets; more team members means more opportunities to clone existing nodes rather than rebuilding them. To invite colleagues: * From the v2 CLI: Have your colleagues run [`dbt login`](https://docs.getdbt.com/reference/commands/login.md?version=2.0) after dbt State is enabled on the account. * From dbt platform: Go to **Account settings** > **Users** and click **Invite users**. For more information, refer to [Invite new users](https://docs.getdbt.com/docs/platform/manage-access/invite-users.md#invite-new-users). #### Debugging dbt State If dbt State is behaving unexpectedly, you can prepend your run command with the `DBT_ENGINE_MANAGE_STATE` environment variable to isolate the issue: ```bash DBT_ENGINE_MANAGE_STATE=0 dbt run --target dev --select "customers" ``` To see which decision dbt State made for each node after a run and why, you can run the (Applies to dbt v2.0 and later) [`dbt state explain`](https://docs.getdbt.com/reference/commands/state-explain.md) command. If you use the dbt platform, the same information is available without running a command — go to the [**Explain** tab](https://docs.getdbt.com/docs/deploy/run-visibility.md#explain-tab) on the job run details page to see the full decision breakdown for each node. #### Next steps * [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md) * [`dbt login` with dbt State](https://docs.getdbt.com/reference/commands/login.md?version=2.0#dbt-login-with-dbt-state) * [Configure deferral](https://docs.getdbt.com/docs/deploy/dbt-state-deferral.md) * [Non-interactive environment setup](https://docs.getdbt.com/docs/deploy/dbt-state-cicd.md) * [dbt State configs](https://docs.getdbt.com/reference/resource-configs/dbt-state-configs.md) * [Monitor dbt State activity](https://docs.getdbt.com/docs/deploy/dbt-state-interface.md) --- ### Setting up state-aware orchestration Private preview ### Setting up state-aware orchestration [Private preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt platform | Enterprise, Enterprise+ Set up state-aware orchestration to automatically determine which models to build by detecting changes in code or data and only building the changed models each time a job is run. State-aware orchestration is now dbt State [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) works with all engines and environments: dbt v1, dbt platform, and dbt v2 If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). important dbt v2 is currently available for installation in: * [Local command line interface (CLI) tools](https://docs.getdbt.com/docs/local/install-dbt.md?version=2) [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") * [VS Code and Cursor with the dbt extension](https://docs.getdbt.com/docs/install-dbt-extension.md) [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") * [dbt platform environments](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md#dbt-v2) Join the conversation in our Community Slack channel [`#dbt-fusion-engine`](https://getdbt.slack.com/archives/C088YCAB6GH). #### Prerequisites To use state-aware orchestration, make sure you meet these prerequisites: * You must have a dbt [Enterprise and Enterprise+ accounts](https://www.getdbt.com/signup/) and a [Developer seat license](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md). * You have updated the environment that will run state-aware orchestration to dbt v2. For more information, refer to [Upgrading to dbt v2](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md). * Your account must have access to state-aware orchestration. Contact your account manager to request access. * You must have a dbt project connected to a [data platform](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md). * You must have [access permission](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md) to view, create, modify, or run jobs. * You must set up a [deployment environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md) that is production or staging only. * You must use a deploy job. Continuous integration (CI) and merge jobs currently do not support state-aware orchestration. * (Optional) To customize behavior, you have configured your model or source data with [advanced configurations](#advanced-configurations). info State-aware orchestration is available for SQL models only. Python models are not supported. #### Default settings Once your account has access to state-aware orchestration, any new deploy job you create in a dbt v2 environment is automatically state-aware. Without additional configuration, the job only builds models when code has changed or new data exists in a source. #### Create a job New jobs are state-aware by default For existing jobs, select **Enable dbt v2 cost optimization features** in the **Job settings** page to enable state-aware orchestration. To create a state-aware job: 1. From your deployment environment page, click **Create job** and select **Deploy job**. 2. Options in the **Job settings** section: * **Job name**: Specify the name, for example, `Daily build`. * (Optional) **Description**: Provide a description of what the job does (for example, what the job consumes and what the job produces). * **Environment**: By default, it’s set to the deployment environment you created the state-aware job from. 3. Options in the **Execution settings** and **Triggers** sections: [![Example of Triggers on the Deploy Job page](/img/docs/dbt-platform/using-dbt-platform/example-triggers-section.png?v=2 "Example of Triggers on the Deploy Job page")](#)Example of Triggers on the Deploy Job page * **Execution settings** section: * **Commands**: By default, it includes the `dbt build` command. Click **Add command** to add more [commands](https://docs.getdbt.com/docs/deploy/job-commands.md) that you want to be invoked when the job runs. * **Generate docs on run**: Enable this option if you want to [generate project docs](https://docs.getdbt.com/docs/build/documentation.md) when this deploy job runs. * **Enable dbt v2 cost optimization features**: Select this option to enable **State-aware orchestration**. **Efficient testing** is disabled by default. You can expand **More options** to enable or disable individual settings. * **Triggers** section: * **Run on schedule**: Run the deploy job on a set schedule. * **Timing**: Specify whether to [schedule](#schedule-days) the deploy job using **Intervals** that run the job every specified number of hours, **Specific hours** that run the job at specific times of day, or **Cron schedule** that run the job specified using [cron syntax](#cron-schedule). * **Days of the week**: By default, it’s set to every day when **Intervals** or **Specific hours** is chosen for **Timing**. * **Run when another job finishes**: Run the deploy job when another *upstream* deploy [job completes](#trigger-on-job-completion). * **Project**: Specify the parent project that has that upstream deploy job. * **Job**: Specify the upstream deploy job. * **Completes on**: Select the job run status(es) that will [enqueue](https://docs.getdbt.com/docs/deploy/job-scheduler.md#scheduler-queue) the deploy job. 6. (Optional) Options in the **Advanced settings** section: * **Environment variables**: Define [environment variables](https://docs.getdbt.com/docs/build/environment-variables.md) to customize the behavior of your project when the deploy job runs. * **Target name**: Define the [target name](https://docs.getdbt.com/docs/build/custom-target-names.md) to customize the behavior of your project when the deploy job runs. Environment variables and target names are often used interchangeably. * **Run timeout**: Cancel the deploy job if the run time exceeds the timeout value. * **Compare changes against**: By default, it’s set to **No deferral**. Select either **Environment** or **This Job** to let dbt know what it should compare the changes against. 7. Click **Save**. You can see which models dbt builds in the run summary logs. Models that weren't rebuilt during the run are tagged as **Reused** with context about why dbt skipped rebuilding them (and saving you unnecessary compute!). You can also see the reused models under the **Reused** tab. [![Example logs for state-aware orchestration](/img/docs/dbt-platform/using-dbt-platform/SAO_logs_view.png?v=2 "Example logs for state-aware orchestration")](#)Example logs for state-aware orchestration #### Delete a job To delete a job or multiple jobs in dbt: 1. Click **Deploy** on the navigation header. 2. Click **Jobs** and select the job you want to delete. 3. Click **Settings** on the top right of the page and then click **Edit**. 4. Scroll to the bottom of the page and click **Delete job** to delete the job. <br /> [![Delete a job](/img/docs/dbt-platform/platform-configuring-dbt-platform/delete-job.png?v=2 "Delete a job")](#)Delete a job 5. Confirm your action in the pop-up by clicking **Confirm delete** in the bottom right to delete the job immediately. This action cannot be undone. However, you can create a new job with the same information if the deletion was made in error. 6. Refresh the page, and the deleted job should now be gone. If you want to delete multiple jobs, you'll need to perform these steps for each job. If you're having any issues, feel free to [contact us](mailto:support@getdbt.com) for additional help. #### Advanced configurations By default, we use the warehouse metadata to check if sources (or upstream models in the case of Mesh) are fresh. For more advanced use cases, dbt provides other options that enable you to specify what gets run by state-aware orchestration. You can use the following optional parameters to customize your state-aware orchestration: | Parameter | Description | Allowed values | Supports Jinja | | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- | | `loaded_at_field` | Specifies a specific column to use from the data. | Name of timestamp column. For example, `created_at`, `"CAST(created_at AS TIMESTAMP)"`. | ✅ | | `loaded_at_query` | Defines a custom freshness condition in SQL to account for partial loading or streaming data. | SQL string. For example, `"select {{ current_timestamp() }}"`. For a multi-line query, see the example after this table. | ✅ | | `build_after.count` | Determines how many units of time must pass before a model can be rebuilt to help reduce build frequency. | A positive integer or a Jinja expression. For example, `4` or `"{{ var('build_after_count', 4) }}"`. | ✅ | | `build_after.period` | The time unit for the count to define the build interval. | `minute`, `hour`, `day`, or a Jinja expression (for example, `"{{ var('build_after_period', 'day') }}"`). | ✅ | | `build_after.updates_on` | Determines whether a model rebuild is triggered when any upstream dependency has fresh data or only when all upstream dependencies are fresh. | \* `any` (default) — Use this value when you want a downstream model to rebuild if *any* of its upstream dependencies receives fresh data, even if others haven’t.\* `all` — Use this value when you want to trigger a rebuild only when *all* upstream dependencies are fresh — minimizing unnecessary builds and reducing compute cost. Recommended to use in state-aware orchestration. | ❌ | Some notes when using `loaded_at_field` or `loaded_at_query`: * You can either define `loaded_at_field` or `loaded_at_query` but not both. * To use a multi-line SQL query for a `loaded_at_query` configuration, include your query as a YAML block so dbt can execute it as the custom freshness query. For example: ```yaml loaded_at_query: | select max(ingested_at) from {{ this }} where ingested_at >= current_timestamp - interval '3 days' ``` * If a source is a view in the data warehouse, the available metadata is usually insufficient to discern freshness, and dbt emits a warning during freshness checks. To determine freshness for sources that are views, add a `loaded_at_field` or `loaded_at_query` to your configuration. To learn more about model freshness and `build_after`, refer to [model `freshness` config](https://docs.getdbt.com/reference/resource-configs/freshness.md). To learn more about source and upstream model freshness configs, refer to [resource `freshness` config](https://docs.getdbt.com/reference/resource-properties/freshness.md). ##### Customizing behavior You can optionally configure state-aware orchestration when you want to fine-tune orchestration behavior for these reasons: * **Defining source freshness:** By default, dbt uses metadata from the data warehouse to automatically detect when source data changes. Freshness configuration is not required for state-aware orchestration to work. You can optionally configure source freshness if you want to: * Receive alerts when sources don't update within your expected Service Level Agreement (SLA) using `warn_after`/`error_after`. * Specify a custom column using `loaded_at_field`. * Specify a custom SQL statement using `loaded_at_query` to define what freshness means. Not all source freshness is equal — especially with partial ingestion pipelines. You may want to delay a model build until your sources have received a larger volume of data or until a specific time window has passed. You can define what "fresh" means on a source-by-source basis using a custom freshness query. This lets you: * Add a time difference to account for late-arriving data * Delay freshness detection until a threshold is reached (for example, number of records or hours of data) The following examples show how to configure a source so that state-aware orchestration detects new upstream data only when your custom condition is met. ##### loaded\_at\_field State-aware orchestration treats the source as fresh when the maximum value of the `loaded_at_field` column changes since the previous run: models/sources.yml ```yaml sources: - name: jaffle_shop config: freshness: warn_after: {count: 12, period: hour} error_after: {count: 24, period: hour} loaded_at_field: _etl_loaded_at ``` ##### loaded\_at\_query To define freshness with custom SQL, use `loaded_at_query`. State-aware orchestration runs the query to get a single timestamp. When that value changes compared to the previous run, the source is considered fresh. models/sources.yml ```yaml sources: - name: raw_orders tables: - name: orders loaded_at_query: | select max(ingested_at) from {{ this }} where ingested_at >= current_timestamp - interval '3 days' ``` In this example, dbt runs the custom `loaded_at_query` to get a single timestamp — the latest `ingested_at` within the last three days. On each run, dbt compares this new maximum timestamp to the value from the previous run. If the maximum timestamp is newer, state-aware orchestration considers the source to have fresh data and may trigger rebuilds. * **Reducing model build frequency** Some models don’t need to be rebuilt every time their source data is updated. To control this: * Set a refresh interval on models, folders, or the project to define how often they should be rebuilt at most * This helps avoid overbuilding and reduces costs by only running what's really needed * **Changing the default from `any` to `all`** Based on what a model depends on upstream, you may want to wait until all upstream models have been refreshed rather than going as soon as there is any new data. * Change what orchestration waits on from any to all for models, folders, or the project to wait until all upstream models have new data * This helps avoid overbuilding and reduces costs by building models once everything has been refreshed To configure and customize behavior, you can do so in the following places using the `build_after` config: * `dbt_project.yml` at the project level in YAML * `model/properties.yml` at the model level in YAML * `model/model.sql` at the model level in SQL These configurations are powerful because you can define a sensible default at the project level or for specific model folders, and override it for individual models or model groups that require more frequent updates. ##### Handling late-arriving data If your incremental models use a lookback window to capture [late-arriving data](https://docs.getdbt.com/best-practices/materializations/4-incremental-models.md#late-arriving-facts), make sure your freshness logic aligns with that window. When you use a `loaded_at_field` or `loaded_at_query`, state-aware orchestration uses that value to determine whether new data has arrived. When the `loaded_at` value reflects an event timestamp (for example, `event_date`), late-arriving records may not update this value if the event occurred in the past. In these cases, state-aware orchestration may not trigger a rebuild, even though your incremental model’s lookback window would normally include those rows. To ensure late-arriving data is detected by state-aware orchestration, use `loaded_at_query` and make sure it aligns with the same lookback window used in your incremental filter. See the following samples of a lookback window and its corresponding `loaded_at_query` value: ##### Lookback window ```sql {{ config( materialized='incremental', unique_key='order_id' ) }} select * from {{ source('raw_orders', 'orders') }} {% if is_incremental() %} where ingested_at > (select max(ingested_at) from {{ this }}) - interval '3 days' {% endif %} ``` ##### loaded\_at\_query ```yaml loaded_at_query: | select max(ingested_at) from {{ this }} where ingested_at >= current_timestamp - interval '3 days' ``` #### Example Let's use an example to illustrate how to customize our project so a model and its parent model are rebuilt only if they haven't been refreshed in the past 4 hours — even if a job runs more frequently than that. A Jaffle shop has recently expanded globally and wanted to make savings. To reduce spend, they found out about dbt's state-aware orchestration and want to rebuild models only when needed. Maggie — the analytics engineer — wants to configure her dbt `jaffle_shop` project to only rebuild certain models if they haven't been refreshed in the last 4 hours, even if a job runs more often than that. To do this, she uses the model `freshness` config. This config helps state-aware orchestration decide *when* a model should be rebuilt. Note that for every `freshness` config, you're required to set values for both `count` and `period`. This applies to all `freshness` types: `freshness.warn_after`, `freshness.error_after`, and `freshness.build_after`. Refer to the following examples for using the `freshness` config in the model file, in the project YAML file, and in the config block of the `model.sql` file: ##### Model YAML models/model.yml ```yaml models: - name: dim_wizards config: freshness: build_after: count: 4 # how long to wait before rebuilding period: hour # unit of time updates_on: all # only rebuild if all upstream dependencies have new data - name: dim_worlds config: freshness: build_after: count: 4 period: hour updates_on: all ``` ##### Project YAML file dbt\_project.yml ```yaml models: <resource-path>: +freshness: build_after: count: 4 period: hour updates_on: all ``` ##### SQL file config models/\<filename>.sql ```jinja {{ config( freshness={ "build_after": { "count": 4, "period": "hour", "updates_on": "all" } } ) }} ``` With this config, dbt: * Checks if there's new data in the upstream sources * Checks when `dim_wizards` and `dim_worlds` were last built If any new data is available *and* at least 4 hours have passed, dbt rebuilds the models. You can override freshness rules set at higher levels in your dbt project. For example, in the project YAML file, you set: dbt\_project.yml ```yml models: +freshness: build_after: count: 4 period: hour jaffle_shop: # this needs to match your project `name:` in dbt_project.yml staging: +materialized: view marts: +materialized: table ``` This configuration means that every model in the project has a `build_after` of 4 hours. To change this for specific models or groups of models, you could set: dbt\_project.yml ```yml models: +freshness: build_after: count: 4 period: hour marts: # only applies to models inside the marts folder +freshness: build_after: count: 1 period: hour ``` If you want to exclude a model from the freshness rule set at a higher level, set `freshness: null` for that model. With freshness disabled, state-aware orchestration falls back to its default behavior and builds the model whenever there’s an upstream code or data change. ##### Differences between `all` and `any` * Since Maggie configured `updates_on: all`, this means *both* models must have new upstream data to trigger a rebuild. If only one model has fresh data and the other doesn't, nothing is built -- which will massively reduce unnecessary compute costs and save time. * If Maggie wanted these models to rebuild more often (for example, if *any* upstream source has new data), she would then use `updates_on: any` instead: models/model.yml ```yaml freshness: build_after: count: 1 period: hour updates_on: any ``` This way, if either `dim_wizards` or `dim_worlds` has fresh upstream data and enough time passed, dbt rebuilds the models. This method helps when the need for fresher data outweighs the costs. #### Related docs * [State-aware orchestration configuration](https://docs.getdbt.com/docs/deploy/state-aware-about.md) * [Artifacts](https://docs.getdbt.com/docs/deploy/artifacts.md) * [Continuous integration (CI) jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md) * [`freshness`](https://docs.getdbt.com/reference/resource-configs/freshness.md) * [About dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) * [Setting up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) * [Set up dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md) * [dbt State configs](https://docs.getdbt.com/reference/resource-configs/dbt-state-configs.md) * [Migrate to dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md) --- ### Sign in or register Preview ### Sign in or register [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Local development The dbt VS Code extension comes with a suite of features that are available to all users for 14 days. After the 14-day trial, sign in or register for a dbt platform account to keep using all features, including advanced features such as [live preview for models and CTEs](https://docs.getdbt.com/docs/dbt-extension-features.md#live-preview-for-models-and-ctes), [column-level lineage](https://docs.getdbt.com/docs/dbt-extension-features.md#rich-lineage-in-context), and more. Refer to [VS Code extension features](https://docs.getdbt.com/docs/dbt-extension-features.md#feature-availability) for the full list of features and their availability. Most features remain available without signing in — only advanced features prompt you to sign in after the trial ends. (Applies to dbt v1.13 and later) To continue using all the features, register or log in from the command line using [`dbt login`](https://docs.getdbt.com/reference/commands/login.md?version=2.0), available with v2 and later. Your login state can be shared across dbt features, including the dbt VS Code extension and, in supported versions, dbt State. Run [`dbt login status`](https://docs.getdbt.com/reference/commands/login.md?version=2.0#dbt-login-status) to view your currently authenticated status. #### Key points * The extension is free for organizations for up to 15 users. Refer to the [acceptable use policy](https://www.getdbt.com/dbt-assets/vscode-plugin-aup). * Registration links your editor to your registered dbt platform account so you can keep using advanced features after the 14-day trial. * You can authenticate with `dbt login` or register from inside the extension. If you don't have a dbt platform account, you can create a free account during authentication to unlock advanced features. * If a valid [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md) file exists on your machine, the extension can use it automatically. If you've registered before, you won't need to authenticate again unless your session expired. #### Session expiry and re-authorization Your sign-in session stays active across editor restarts. dbt automatically renews your session while you're using the extension or the CLI. You may need to re-authorize in a few specific cases: * **Session expired after 7 days of inactivity.** The extension shows the message: "Your dbt extension session expired. Sign in again to continue using the extension." Click the prompt or run **dbt: Register dbt extension** from the command palette to sign in again. * **A feature needs broader access than your initial sign-in granted.** [`dbt login`](https://docs.getdbt.com/reference/commands/login.md?version=2.0) and the dbt VS Code extension may request different sets of permissions. If you signed in via `dbt login` and later use an extension feature that needs more access (for example, the catalog tab or running a job), the extension prompts you to re-authorize. You'll re-authorize with the same dbt platform account — you don't need a second account. * **You authenticated with `dbt_cloud.yml` instead of OAuth.** File-based credentials don't expire on the 7-day inactivity rule. You only re-authenticate if you regenerate the file or it's removed. When re-authorizing, the **Authorize dbt login** consent screen lists the access being requested. Click **Allow access** to continue. #### Choose a sign-in path There are a couple of different ways to sign in or register for a dbt platform account. Choose the best path for you: | If you... | Use this path | | --------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Starting authentication from the terminal | [`dbt login`](https://docs.getdbt.com/docs/sign-in-dbt-extension.md?registration-path=dbt-login#choose-a-sign-in-path) from the command line. | | Are new to dbt and don't have a dbt platform account | [First-time registration](https://docs.getdbt.com/docs/sign-in-dbt-extension.md?registration-path=first-time#choose-a-sign-in-path) from the dbt VS Code extension. | | Can't sign in or need to reset access | [Recover your login](https://docs.getdbt.com/docs/sign-in-dbt-extension.md?registration-path=recover-login#choose-a-sign-in-path) from the dbt VS Code extension. | | Prefer file-based authentication to remove need for re-authenticating | [`dbt_cloud.yml`](https://docs.getdbt.com/docs/sign-in-dbt-extension.md?registration-path=dbt-cloud-yml#choose-a-sign-in-path) from the dbt VS Code extension. | ##### dbt login ##### dbt login Use this path if you prefer the terminal or want a single auth flow that works across the CLI and the dbt VS Code extension. Running `dbt login` opens the same browser flow that the extension uses when you select **dbt: Register dbt extension** from the command palette, so completing it once authenticates both. 1. From your terminal, run: ```shell dbt login ``` 2. Complete the sign-in or sign-up flow in your browser. * Refer to [First-time registration](https://docs.getdbt.com/docs/sign-in-dbt-extension.md?registration-path=first-time#first-time-registration) if you've never registered * Refer to [Existing account sign-in](https://docs.getdbt.com/docs/sign-in-dbt-extension.md?registration-path=existing-account#existing-dbt-account) if you already have a dbt account. * Refer to [Recover your login](https://docs.getdbt.com/docs/sign-in-dbt-extension.md?registration-path=recover-login#recover-your-login) if you've forgotten your password or your account is locked. 3. On the **Authorize dbt login** screen, click **Allow access**. dbt saves your credentials to your local dbt configuration directory and uses them for future CLI commands: * macOS and Linux: `~/.dbt/` * Windows: `C:\Users\[username]\.dbt\` [![The Authorize VS Code Extension consent screen.](/img/docs/extension/signin-authorize-extension.png?v=2 "The Authorize VS Code Extension consent screen.")](#)The Authorize VS Code Extension consent screen. 4. Return to your editor and restart the dbt VS Code extension, or open a new editor window. 5. The extension detects the credentials and shows **Registered**. Authorizing additional access When you register in the dbt VS Code extension, you might be prompted to authorize more access than you granted when signing in through the CLI (or the other way around). This is expected — some extension features need broader access than your initial sign-in. You can authorize the request with the same dbt platform account you already use, so a single account covers both the CLI and the dbt VS Code extension. For more details on when re-authorization is triggered, refer to [Session expiry and re-authorization](https://docs.getdbt.com/docs/sign-in-dbt-extension.md#session-expiry-and-re-authorization). You've now unlocked the full feature set of the dbt VS Code extension. For details on what `dbt login` unlocks across tools, refer to the [`dbt login` reference](https://docs.getdbt.com/reference/commands/login.md?version=2.0). ##### First-time registration ##### First-time registration Register to unlock the full [feature set](https://docs.getdbt.com/docs/dbt-extension-features.md) of the dbt VS Code extension. Use this path if you've never registered before. The extension and the CLI open the same browser flow, so you can start from either entry point: * **From the extension:** Click the registration prompt, or open the command palette (Windows/Linux: Ctrl + Shift + P, macOS: Cmd + Shift + P) and run **dbt: Register dbt extension**. [![The extension registration prompt in VS Code.](/img/docs/extension/registration-prompt.png?v=2 "The extension registration prompt in VS Code.")](#)The extension registration prompt in VS Code. * **From the CLI:** Run `dbt login` in your terminal. After you start the flow, complete the browser steps: 1. On the **Log in to dbt platform** page, enter your email and click **Continue**. 2. On the **Create your dbt platform** page, enter your name, create a password, then click **Continue**. dbt creates a free account for you. 3. On the **Authorize dbt login** screen, review the requested access. Leave **Keep session alive** on if you want dbt to renew your session automatically, then click **Allow access**. [![The Authorize dbt login consent screen.](/img/docs/extension/signin-authorize-extension.png?v=2 "The Authorize dbt login consent screen.")](#)The Authorize dbt login consent screen. 4. Return to your editor. The extension shows **Registered**. If you started from the CLI, your credentials are also saved for future CLI commands. 5. Continue with the [Get started](https://docs.getdbt.com/docs/install-dbt-extension.md#getting-started) workflow. Authorizing additional access When you register in the dbt VS Code extension, you might be prompted to authorize more access than you granted when signing in through the CLI (or the other way around). This is expected — some extension features need broader access than your initial sign-in. You can authorize the request with the same dbt platform account you already use, so a single account covers both the CLI and the dbt VS Code extension. For more details on when re-authorization is triggered, refer to [Session expiry and re-authorization](https://docs.getdbt.com/docs/sign-in-dbt-extension.md#session-expiry-and-re-authorization). ##### Existing account sign-in ##### Existing account sign-in Use this path if you've registered before, including with an older or inactive registration. The extension and the CLI open the same browser flow, so you can start from either entry point: * **From the extension:** [Update the VS Code extension](https://code.visualstudio.com/docs/setup/setup-overview#_update-cadence) to the latest version, restart your editor, then click the registration prompt or run **dbt: Register dbt extension** from the command palette. * **From the CLI:** Run `dbt login` in your terminal. After you start the flow, complete the browser steps: 1. On the **Sign in to dbt** page, enter the email you used to register, agree to the terms of service, complete the "I'm not a robot" check, then click **Continue**. 2. On the **Verify your email** page, enter the verification code dbt sent to your email and click **Continue**. 3. If your email is linked to more than one account, select the account you want to use on the account selection page. 4. If your account uses an authenticator app, choose a method to verify your identity, then complete the prompt. [![The verify-your-identity screen for accounts that use an authenticator app.](/img/docs/extension/signin-verify-identity.png?v=2 "The verify-your-identity screen for accounts that use an authenticator app.")](#)The verify-your-identity screen for accounts that use an authenticator app. 5. On the **Authorize dbt login** screen, review the requested access and click **Allow access**. **When you might still need a [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md):** * You want a file-based credential for automations or defer. * You use platform features that require `dbt_cloud.yml`, such as Mesh or auto-deferral. Authorizing additional access When you register in the dbt VS Code extension, you might be prompted to authorize more access than you granted when signing in through the CLI (or the other way around). This is expected — some extension features need broader access than your initial sign-in. You can authorize the request with the same dbt platform account you already use, so a single account covers both the CLI and the dbt VS Code extension. For more details on when re-authorization is triggered, refer to [Session expiry and re-authorization](https://docs.getdbt.com/docs/sign-in-dbt-extension.md#session-expiry-and-re-authorization). ##### Recover your login ##### Recover your login Use this path if the login flow says you have an account but you don't remember your password. 1. On the **Sign in to dbt** page, click **Reset password**. 2. On the reset-password screen, enter the email you used to register. 3. Check your inbox for the password-reset email and follow the link. 4. Set a new password. 5. Return to your terminal and run `dbt login` again. The password reset opens a separate browser session from the original terminal login session, so you need to restart the login flow. 6. On the **Sign in to dbt** page, sign in with the email and new password you just set, agree to the terms of service, complete the "I'm not a robot" check, then click **Continue**. 7. If your email is linked to more than one account, select the account you want to use. 8. On the **Authorize dbt login** screen, click **Allow access**. When authorization completes, the flow returns to your terminal and the dbt VS Code extension. If you still can't sign in, your account may be locked. Contact [dbt Support](mailto:support@getdbt.com) to unlock your account, then continue with the **Existing account sign-in** flow. Authorizing additional access When you register in the dbt VS Code extension, you might be prompted to authorize more access than you granted when signing in through the CLI (or the other way around). This is expected — some extension features need broader access than your initial sign-in. You can authorize the request with the same dbt platform account you already use, so a single account covers both the CLI and the dbt VS Code extension. For more details on when re-authorization is triggered, refer to [Session expiry and re-authorization](https://docs.getdbt.com/docs/sign-in-dbt-extension.md#session-expiry-and-re-authorization). ##### dbt\_cloud.yml ##### Register with `dbt_cloud.yml` Use this path if your workflow requires a credential file or if you want to remove the need to re-authenticate when sessions expire. `dbt_cloud.yml` is a separate, complementary authentication path. You may still need `dbt_cloud.yml` for dbt platform features such as Mesh and auto-deferral. 1. Log in to dbt platform. 2. Go to **Account settings** → **VS Code extension**. 3. In the **Set up your credentials** section, click **Download credentials** to get your [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md) file. [![Download the dbt\_cloud.yml file from your dbt platform account.](/img/docs/extension/download-registration-2.png?v=2 "Download the dbt_cloud.yml file from your dbt platform account.")](#)Download the dbt\_cloud.yml file from your dbt platform account. 4. Move the file into your `.dbt` directory: * macOS and Linux: `~/.dbt/dbt_cloud.yml` * Windows: `C:\Users\[username]\.dbt\dbt_cloud.yml` 5. Return to your editor, open the command palette, and run **dbt: Register dbt extension**. 6. The extension detects the credential file and continues the registration flow. **Behavior details:** * If a [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md) file exists, the extension uses it automatically. * If the file is missing, the extension prompts you to sign in or add the file. If your dbt project uses environment variables, configure them locally so the extension can use the same values as your dbt environment. For more information, refer to [Configure environment variables](https://docs.getdbt.com/docs/configure-dbt-extension.md). #### Troubleshooting If you run into any issues, check out the troubleshooting section below.  How to create a .dbt directory in root and move dbt\_cloud.yml file If you've never had a `.dbt` directory, you should perform the following recommended steps to create one. If you already have a `.dbt` directory, move the `dbt_cloud.yml` file into it. Some information about the `.dbt` directory: * A `.dbt` directory is a hidden folder in the root of your filesystem. It's used to store your dbt configuration files. The `.` prefix is used to create a hidden folder, which means it's not visible in Finder or File Explorer by default. * To view hidden files and folders, press Command + Shift + G on macOS or Ctrl + Shift + G on Windows. This opens the "Go to Folder" dialog where you can search for the `.dbt` directory. ##### Create a .dbt directory 1. Clone your dbt project repository locally. 2. Use the `mkdir` command followed by the name of the folder you want to create. * If using macOS, add the `~` prefix to create a `.dbt` folder in the root of your filesystem: * macOS: `mkdir ~/.dbt` * Windows: `mkdir %USERPROFILE%\.dbt` ##### Move the dbt\_cloud.yml file You can move the `dbt_cloud.yml` file into the `.dbt` directory using the `mv` command or by dragging and dropping the file into the `.dbt` directory by opening the Downloads folder using the "Go to Folder" dialog and then using drag-and-drop in the UI. To move the file using the terminal, use the `mv/move` command. This command moves the `dbt_cloud.yml` from the `Downloads` folder to the `.dbt` folder. If your `dbt_cloud.yml` file is located elsewhere, adjust the path accordingly. ###### Mac or Linux In your command line, use the `mv` command to move your `dbt_cloud.yml` file into the `.dbt` directory. If you've just downloaded the `dbt_cloud.yml` file and it's in your Downloads folder, the command might look something like this: ```bash mv ~/Downloads/dbt_cloud.yml ~/.dbt/dbt_cloud.yml ``` ###### Windows In your command line, use the move command. Assuming your file is in the Downloads folder, the command might look like this: ```bash move %USERPROFILE%\Downloads\dbt_cloud.yml %USERPROFILE%\.dbt\dbt_cloud.yml ```  I can't see the lineage tab in Cursor If you're using the dbt VS Code extension in Cursor, the lineage tab works best in Editor mode and doesn't render in Agent mode. If you're in Agent mode and the lineage tab isn't rendering, just switch to Editor mode to view your project's table and column lineage.  The extension gets stuck in a loading state If the extension is attempting to activate during startup and locks into a permanent loading state, check that: * Your dbt VS Code extension is on the latest version. * Your IDE is on the latest version. * You have a valid `dbt_cloud.yml` file configured and in the [correct location](#register-with-dbt_cloudyml). If you're still experiencing issues, try these steps before contacting dbt Support: * Delete and download a new copy of your `dbt_cloud.yml` file. * Delete and reinstall the dbt VS Code extension.  dbt platform configurations If you're a cloud-based dbt platform user who has the `dbt-cloud:` config in the `dbt_project.yml` file and are also using dbt Mesh, you must have the project ID configured: ```yaml dbt-cloud: project-id: 12345 # Required ``` If you don’t configure this correctly, cross-platform references will not resolve properly, and you will encounter errors executing dbt commands.  dbt extension not activating If the dbt extension has activated successfully, you will see the **dbt Extension** label in the status bar at the bottom left of your editor. You can view diagnostic information about the dbt extension by clicking the **dbt Extension** button. If the **dbt Extension** label is not present, then it is likely that the dbt extension was not installed successfully. If this happens, try uninstalling the extension, restarting your editor, and then reinstalling the extension. **Note:** It is possible to "hide" status bar items in VS Code. Double-check if the dbt Extension status bar label is hidden by right-clicking on the status bar in your editor. If you see dbt Extension in the right-click menu, then the extension has installed successfully.  Missing dbt LSP features If you receive a `no active LSP for this workspace` error message or aren't seeing dbt Language Server (LSP) features in your editor (like autocomplete, go-to-definition, or hover text), start by first following the general troubleshooting steps mentioned earlier. If you've confirmed the dbt extension is installed correctly but don't see LSP features, try the following: 1. Check extension version — Ensure that you're using the latest available version of the dbt extension by: * Opening the **Extensions** page in your editor, or * Going to the **Output** tab and looking for the version number, or * Running `dbtf --version` in the terminal. 2. Reinstall the LSP — If the version is correct, reinstall the LSP: 1. Open the Command Palette: Command + Shift + P (macOS) or Ctrl + Shift + P (Windows/Linux). 2. Paste `dbt: Reinstall dbt LSP` and enter. This command downloads the LSP and re-activates the extension to resolve the error.  Unsupported dbt version If you see an error message indicating that your version of dbt is unsupported, then there is likely a problem with your environment. Check the dbt Path setting in your VS Code settings. If this path is set, ensure that it is pointing to a valid dbt v2 executable. If necessary, you can also install dbt v2 directly using these instructions: [Install the dbt v2 CLI](https://docs.getdbt.com/docs/local/install-dbt.md?version=2)  dbt v2 binary not found at the configured path If the extension reports that the dbt v2 binary can't be found at the configured path (for example, `dbt-fusion binary not found at [path]`), the `dbt.fusionPath` setting is pointing to a location that doesn't contain a valid binary. * Verify that [`dbt.fusionPath`](https://docs.getdbt.com/docs/configure-dbt-extension.md#dbt-extension-settings) points to a valid dbt v2 binary. * If you haven't installed dbt v2 manually, clear the setting and let the extension download and manage it for you. * To install manually, follow [Install the dbt v2 CLI](https://docs.getdbt.com/docs/local/install-dbt.md?version=2).  dbt v2 version is not compatible with this extension If the extension reports that the installed dbt v2 version isn't compatible with your dbt VS Code extension version, the two are outside the supported range. 1. Run `dbt --version` to check your installed dbt v2 version. 2. Compare it against the [version compatibility matrix](https://docs.getdbt.com/docs/dbt-versions/dbt-version-compatibility.md#compatibility-matrix) for your extension version. 3. Update dbt v2 or the extension so both fall within the supported range. Use the **Download compatible version** action in the notification if it appears.  dbt v2 crashes on startup If the extension reports that dbt v2 crashed on startup, confirm the binary runs on its own: 1. Run `dbt --version` in your terminal. If this fails, reinstall dbt v2 using [Install the dbt v2 CLI](https://docs.getdbt.com/docs/local/install-dbt.md?version=2). 2. Use the **Show Logs** action in the notification (or open the **Output** tab) to review the startup error.  A known-bad dbt v2 version is installed If the extension warns that your installed dbt v2 version has a known regression, dbt Labs has flagged that release as [known-bad](https://docs.getdbt.com/docs/dbt-versions/dbt-version-compatibility.md#known-bad-releases). Update to the version named in the notification. For standalone installations: ```shell dbt system update ``` The warning persists across restarts until you update. If you work in an air-gapped environment, refer to [known-bad releases](https://docs.getdbt.com/docs/dbt-versions/dbt-version-compatibility.md#known-bad-releases) for how to distribute the manifest locally.  Addressing the 'dbt language server is not running in this workspace' error To resolve the `dbt language server is not running in this workspace` error, you need to add your dbt project folder to a workspace: 1. In VS Code, click **File** in the toolbar then select **Add Folder to Workspace**. 2. Select the dbt project file you want to add to a workspace. 3. To save your workspace, click **File** then select **Save Workspace As**. 4. Navigate to the location you want to save your workspace. This should resolve the error and open your dbt project by opening the workspace it belongs to. For more information on workspaces, refer to [What is a VS Code workspace?](https://code.visualstudio.com/docs/editing/workspaces/workspaces).  Manifest cannot be downloaded from the dbt platform If the dbt VS Code extension cannot download the manifest from the dbt platform or you get `warning: dbt1200: Failed to download manifest` using dbt v2 locally, you are probably having DNS-related issues. To confirm this, do a DNS lookup for the host dbt v2 is trying to download from (for example, prodeu2.blob.core.windows.net) by using `dig` on Linux/Mac or `nslookup` on Windows. If this doesn't return an IP address, the likely reason is that your company uses the same cloud provider with private endpoints for cloud resources, and DNS requests for these are forwarded to private DNS zones. This situation can be remedied by setting up an internet fallback, which will then return a public IP to any cloud storage that does not have a private IP registered with the private DNS zone. For Azure refer to [Fallback to internet for Azure Private DNS zones](https://learn.microsoft.com/en-us/azure/dns/private-dns-fallback). #### More information about dbt v2 * [About the dbt extension](https://docs.getdbt.com/docs/about-dbt-extension.md) * [Supported features matrix](https://docs.getdbt.com/docs/dbt/supported-features.md) * [Install dbt](https://docs.getdbt.com/docs/local/install-dbt.md) * [Quickstart for dbt v2](https://docs.getdbt.com/guides/dbt.md?step=1) * [Upgrade guide](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md) * [dbt v2 license agreement](https://www.getdbt.com/dbt-fusion-engine-license-agreement) --- ### Simple metrics (Applies to dbt v1.12 and later) Simple metrics are metrics that directly reference a single column expression within a semantic model, without any additional columns involved. They are aggregations over a column in your data platform and can be filtered by one or multiple dimensions. The parameters, description, and type for simple metrics are: | Parameter | Description | Required | Type | | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------- | | `name` | The name of the metric. It must be unique within your project and can include lowercase letters, numbers, and underscores. Use this name to reference the metric from the Semantic Layer API. | Required | String | | `description` | The description of the metric. | Optional | String | | `type` | The type of the metric (cumulative, derived, ratio, or simple). | Required | String | | `label` | Defines the display value in downstream tools. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | String | | `agg` | The aggregation function to use. Use `sum`, `max`, `min`, `average`, `median`, `count`, `count_distinct`, `percentile`, and `sum_boolean`. | Required | String | | `expr` | The expression to use, like a column name. Defaults to the metric name. | Optional | String | | `percentile` | The percentile to use. Required if `agg` is `percentile`. | Optional | Integer | | `percentile_type` | The percentile type to use. Use `discrete` or `continuous`. Required for `percentile` metrics. | Optional | String | | `non_additive_dimension` | The non-additive dimension to use. | Optional | String | | `agg_time_dimension` | The time dimension to use. | Optional | String | | `join_to_timespine` | Indicates if the aggregated measure should be joined to the time spine table to fill in missing dates. Default `false`. | Optional | Boolean | | `fill_nulls_with` | Set the value in your metric definition instead of null (such as zero). | Optional | Integer | The following displays the complete specification for simple metrics, along with an example. (Applies to dbt v1.12 and later) Note that you must define simple metrics within a semantic model's YAML entry. ```yaml models: - name: my_model semantic_model: enabled: true ... metrics: - name: my_simple_metric # Required description: The metric description # Optional label: My simple metric label # Optional type: simple # Required agg: count_distinct # Required sum | max | min | average | median | count_distinct | percentile, and sum_boolean expr: case when is_a then 1 else 0 end # Optional for simple metric, defaults to name of metric join_to_timespine: true fill_nulls_with: 0 - name: my_simple_metric_that_uses_the_other_time_dimension description: The metric description label: My simple metric that uses the other time dimension label type: simple agg: count_distinct agg_time_dimension: my_other_time_dimension_column # Optional, if not using the default time dimension ``` For advanced data modeling, you can use `fill_nulls_with` and `join_to_timespine` to [set null metric values to zero](https://docs.getdbt.com/docs/build/fill-nulls-advanced.md), ensuring numeric values for every data row. #### Simple metrics example (Applies to dbt v1.12 and later) ```yaml metrics: - name: customers description: Count of customers type: simple label: Count of customers agg: count expr: customers fill_nulls_with: 0 join_to_timespine: true filter: "{{ Dimension('customer__customer_total') }} >= 20" # alias is not supported on simple metrics # use alias in input_metrics when referencing this metric in derived, ratio, or conversion metrics - name: large_orders description: "Order with order values over 20." type: simple label: Large orders agg: count expr: orders filter: "{{ Dimension('customer__order_total_dim') }} >= 20" ``` #### Related docs * [Fill null values for simple, derived, or ratio metrics](https://docs.getdbt.com/docs/build/fill-nulls-advanced.md) --- ### Snowflake and Apache Iceberg dbt supports materializing models in the Iceberg table format in two ways: * **Simplest:** The model config `table_format = 'iceberg'` instructs dbt to materialize this model as an Iceberg table in Snowflake Horizon (managed catalog). Whether dbt writes to Snowflake-managed storage by default depends on the version you're running — refer to [External volume defaults](#external-volume-defaults). * **Extensible:** Define an Iceberg catalog in `catalogs.yml` and configure this model with `catalog_name` #### Creating Iceberg tables dbt supports creating Iceberg tables for three of the Snowflake materializations: * [Table](https://docs.getdbt.com/docs/build/materializations.md#table) * [Incremental](https://docs.getdbt.com/docs/build/materializations.md#incremental) * [Dynamic Table](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#dynamic-tables) #### Iceberg catalogs Snowflake supports writing Iceberg tables to Snowflake Horizon (its managed catalog), and to external catalogs through [catalog-linked databases](https://docs.snowflake.com/en/user-guide/tables-iceberg-catalog-linked-database). Those external catalogs include Polaris (self-hosted), Open Catalog (Snowflake's managed Polaris), AWS Glue, GCP BigLake, Databricks Unity, and (in theory) any other catalog that implements Iceberg REST compatibility. ##### Snowflake Horizon (Snowflake-managed) ###### Simplest: Create a single Iceberg table models/MODEL\_NAME.sql ```sql {{ config( materialized = "table", table_format="iceberg", ) }} select * from {{ ref('raw_orders') }} ``` The following configurations are supported. For more information, check out the Snowflake reference for [`CREATE ICEBERG TABLE` (Snowflake as the catalog)](https://docs.snowflake.com/en/sql-reference/sql/create-iceberg-table-snowflake). | Parameter | Type | Required | Description | Sample input | Note | | ----------------------- | ------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `table_format` | String | Yes | Configures the objects table format. | `iceberg` | `iceberg` is the only accepted value. | | `external_volume` | String | No | Specifies the identifier (name) of the external volume where Snowflake writes the Iceberg table's metadata and data files. Set it to `SNOWFLAKE_MANAGED` to write to [Snowflake-managed storage](https://docs.snowflake.com/en/user-guide/tables-iceberg-internal-storage). | `my_s3_bucket`<br />`SNOWFLAKE_MANAGED` | Refer to [External volume defaults](#external-volume-defaults). | | `base_location_root` | String | No | If provided, the input overrides the default dbt `base_location` value of `_dbt` | `foo` | | | `base_location_subpath` | String | No | An optional suffix to add to the `base_location` path that dbt automatically specifies. | `jaffle_marketing_folder` | We recommend that you don't specify this. Modifying this parameter results in a new Iceberg table. Refer to [Base location](#base-location) for more information. | | `iceberg_version` | Integer | No | Specifies the Iceberg format version for the table. Defaults to `2`. Cannot be changed after table creation. | `3` | Set to `3` for improved `VARIANT` type support and better incremental/snapshot performance through deletion vectors. | ###### External volume defaults When a model sets `table_format: iceberg` without an `external_volume`, dbt's behavior depends on the version you're running. (Applies to dbt v2.0 and later) dbt emits `external_volume = 'SNOWFLAKE_MANAGED'` in the `CREATE ICEBERG TABLE` DDL and omits `base_location`, so the table uses Snowflake-managed storage. When you set `external_volume` to `SNOWFLAKE_MANAGED`, dbt omits `base_location` in every version. dbt resolves `external_volume` from the model config first and then from the catalog definition in `catalogs.yml`, so setting it in either place omits `base_location`. When `external_volume` names a user-defined volume, such as an Amazon S3 bucket, dbt emits both `external_volume` and `base_location`. ###### Extensible: Configure `horizon` catalog First, configure a catalog with `type: horizon` in `catalogs.yml`: ##### New spec (beta) catalogs.yml ```yaml catalogs: - name: my_horizon_catalog type: horizon table_format: iceberg # optional - default config: snowflake: # optional - specify additional Snowflake-specific configurations change_tracking: true iceberg_version: 3 # available in v1.12+ ``` ##### Old spec catalogs.yml ```yml catalogs: - name: my_horizon_catalog active_write_integration: snowflake_write_integration write_integrations: - name: snowflake_write_integration external_volume: dbt_external_volume table_format: iceberg catalog_type: built_in adapter_properties: change_tracking: True iceberg_version: 3 # available in v1.12+ ``` Next, configure a dbt model with the name of your Horizon catalog. models/my\_iceberg\_model.sql ```sql {{ config( materialized='table', catalog_name='my_horizon_catalog', iceberg_version=3, # available in v1.12+ ) }} select * from {{ ref('jaffle_shop_customers') }} ``` Finally, run the model: `dbt run -s my_iceberg_model`. Because dbt understands that `type: horizon` refers to Snowflake's managed catalog, dbt templates the appropriate Snowflake DDL/DML for creating and updating managed Iceberg tables. ##### External catalogs dbt can also template Snowflake DDL/DML for creating and updating Iceberg tables managed by external catalogs. First, you need to set up a catalog integration and (recommended) catalog-linked database within Snowflake. See Snowflake docs for how to [create a catalog integration](https://docs.snowflake.com/en/sql-reference/sql/create-catalog-integration) and [catalog-linked database](https://docs.snowflake.com/en/sql-reference/sql/create-database-catalog-linked). Caveats: * For some external catalogs (for example, AWS Glue), table and column identifiers must use only alphanumeric characters (letters and numbers), be lowercase, and surrounded by double quotes. * Starting in dbt v1.11, dbt-snowflake supports basic table materialization on Iceberg tables registered in a Glue catalog through a catalog-linked database. Note that incremental materializations aren't yet supported. After you create the external catalog integration, you can do two things: * **Query an externally managed table:** Snowflake can query Iceberg tables whose metadata lives in the external catalog. In this scenario, Snowflake is a "reader" of the external catalog. The table’s data remains in external cloud storage (AWS S3 or GCP Bucket) as defined in the catalog storage configuration. Snowflake uses the catalog integration to fetch metadata using the REST API. Snowflake then reads the data files from cloud storage. * **Write tables to the external catalog, using Snowflake compute:** You can materialize a dbt model as an Iceberg table using Snowflake's compute, and Snowflake registers and syncs that table to the external catalog (for example, AWS Glue or Databricks Unity). The dbt model appears in that catalog, and other query engines can read its data there. Now, we can configure that external catalog in `catalogs.yml`. Here is an example for an AWS Glue catalog: ##### New spec (beta) catalogs.yml ```yaml catalogs: - name: my_glue_catalog type: glue table_format: iceberg config: snowflake: catalog_database: catalog_linked_db_glue # name of catalog-linked database in Snowflake ``` ##### Old spec catalogs.yml ```yml catalogs: - name: my_glue_catalog active_write_integration: glue_rest write_integrations: - name: glue_rest catalog_type: iceberg_rest table_format: iceberg adapter_properties: catalog_linked_database: catalog_linked_db_glue # name of catalog-linked database in Snowflake catalog_linked_database_type: glue ``` #### Snowflake-specific configs for Iceberg catalogs These are the additional configurations, specific to Snowflake, that can be supplied and nested under `config.snowflake` (in the new catalog spec) or `adapter_properties` (in the old catalog spec). Available configurations are different when writing dbt models as Snowflake-managed Iceberg tables (Snowflake Horizon catalog) versus writing to external catalogs. ###### Snowflake-managed (Horizon) (Applies to dbt v1.12 and later) | Field | Required | Accepted values | | --------------------------------- | -------- | --------------------------------------------------------------------------------------- | | `change_tracking` | Optional | `True` or `False` | | `data_retention_time_in_days` | Optional | Standard Account: `1`, Enterprise or higher: `0` to `90`, default `1` | | `max_data_extension_time_in_days` | Optional | `0` to `90` with a default of `14` | | `storage_serialization_policy` | Optional | `COMPATIBLE` or `OPTIMIZED` | | `base_location_root` | Optional | Relative path segment (for example, `'subpath1/subpath2'`) | | `base_location_subpath` | Optional | Relative path segment (for example, `'subpath1/subpath2'`), only configurable per-model | | `iceberg_version` (v1.12+) | Optional | `2` (default) or `3` | ###### External catalogs (Applies to dbt v1.12 and later) | Field | Required | Accepted values | | --------------------------------- | ----------------------------------------- | ------------------------------------------------------------------------------- | | `auto_refresh` | Optional | `True` or `False` | | `catalog_linked_database` | Required for `catalog type: iceberg_rest` | Catalog-linked database name | | `catalog_linked_database_type` | Optional | Catalog-linked database type. For example, `glue` | | `max_data_extension_time_in_days` | Optional | `0` to `90` (default: `14`) | | `target_file_size` | Optional | Values like `'AUTO'`, `'16MB'`, `'32MB'`, `'64MB'`, `'128MB'`. Case-insensitive | | `iceberg_version` (v1.12+) | Optional | `2` (default) or `3` | * **storage\_serialization\_policy:** The serialization policy tells Snowflake what kind of encoding and compression to perform on the table data files. If not specified at table creation, the table inherits the value set at the schema, database, or account level. If the value isn’t specified at any level, the table uses the default value. You can’t change the value of this parameter after table creation. * **max\_data\_extension\_time\_in\_days:** The maximum number of days Snowflake can extend the data retention period for tables to prevent streams on the tables from becoming stale. The `MAX_DATA_EXTENSION_TIME_IN_DAYS` parameter enables you to limit this automatic extension period to control storage costs for data retention, or for compliance reasons. * **data\_retention\_time\_in\_days:** For managed Iceberg tables, you can set a retention period for Snowflake Time Travel and undropping the table over the default account values. For tables that use an external catalog, Snowflake uses the value of the DATA\_RETENTION\_TIME\_IN\_DAYS parameter to set a retention period for Snowflake Time Travel and undropping the table. When the retention period expires, Snowflake doesn't delete the Iceberg metadata or snapshots from your external cloud storage. * **change\_tracking:** Specifies whether to enable change tracking on the table. * **catalog\_linked\_database:** [Catalog-linked databases](https://docs.snowflake.com/en/user-guide/tables-iceberg-catalog-linked-database) (CLD) in Snowflake ensure that Snowflake can automatically sync metadata (including namespaces and Iceberg tables) from the external Iceberg catalog and registers them as remote tables in the catalog-linked database. The reason we require the usage of Catalog-linked databases for building Iceberg tables with external catalogs is that without it, dbt is unable to truly manage the table end-to-end. Snowflake doesn't support dropping the Iceberg table on non-CLDs in the external catalog; instead, it only allows unlinking the Snowflake table, which creates a discrepancy with how dbt expects to manage the materialized object. * **auto\_refresh:** Specifies whether Snowflake should automatically poll the external Iceberg catalog for metadata updates. If `REFRESH_INTERVAL_SECONDS` isn’t set on the catalog integration, the default refresh interval is 30 seconds. * **target\_file\_size:** Specifies a target Parquet file size. Default is `AUTO`. (Applies to dbt v1.12 and later) * **iceberg\_version:** Specifies the Iceberg format version for the table. Default value is `2`. Set to `3` for improved support for `VARIANT` data types and faster incremental operations. Version 3 uses [deletion vectors](https://docs.snowflake.com/en/user-guide/tables-iceberg-manage#tables-iceberg-deletion-vectors), which let Snowflake mark rows as deleted without rewriting the underlying data files, making incremental runs faster. Note that you can't change the Iceberg version after table creation. As an alternative, you can [configure the default Iceberg version](https://docs.snowflake.com/en/user-guide/tables-iceberg-v3-specification-support#configure-the-default-iceberg-version) at the account, database, or schema level in Snowflake. - **base\_location\_root:** Specifies the prefix of the [`BASE_LOCATION`](https://docs.snowflake.com/en/sql-reference/sql/create-iceberg-table-snowflake#optional-parameters), the write path for the Iceberg table. - **base\_location\_subpath:** Specifies the suffix of the [`BASE_LOCATION`](https://docs.snowflake.com/en/sql-reference/sql/create-iceberg-table-snowflake#optional-parameters), the write path for the Iceberg table. This property can only be set in model configurations, not in `catalogs.yml`. ##### Base location Snowflake's `CREATE ICEBERG TABLE` DDL requires a `base_location` when the table uses a custom external volume. dbt defines this parameter on your behalf to streamline usage and enforce basic isolation of table data within the `EXTERNAL VOLUME`. By default, dbt provides a `base_location` string of the form `_dbt/{SCHEMA_NAME}/{MODEL_NAME}`. Snowflake rejects `BASE_LOCATION` for Iceberg tables that use Snowflake-managed storage, so dbt omits it when you set `external_volume` to `SNOWFLAKE_MANAGED`. dbt also omits it when you leave `external_volume` unset, depending on the version you're running — refer to [External volume defaults](#external-volume-defaults). We recommend using the default behavior, but if you need to customize the resulting `base_location`, you can configure the `base_location` with the model configuration fields `base_location_root` and `base_location_subpath`. (Applies to dbt v2.0 and later) `base_location_subpath` is only accepted in model configurations. * If no inputs are provided, dbt outputs for base\_location `{{ external_volume }}/_dbt/{{ schema }}/{{ model_name }}` * If base\_location\_root = `foo`, dbt outputs `{{ external_volume }}/foo/{{ schema }}/{{ model_name }}` * If base\_location\_subpath = `bar`, dbt outputs `{{ external_volume }}/_dbt/{{ schema }}/{{ model_name }}/bar` * If base\_location\_root = `foo` and base\_location\_subpath = `bar`, dbt outputs `{{ external_volume }}/foo/{{ schema }}/{{ model_name }}/bar` While you can customize paths with `base_location_root` and `base_location_subpath`, we don't recommend you rely on these for environment isolation (such as separating development and production environments). These configuration values can be easily modified by anyone with repository access. For true environment isolation, use separate `EXTERNAL VOLUME`s with infrastructure-level access controls. An example model with a customized `base_location`: iceberg\_model.sql ```sql {{ config( materialized='table', catalog_name='my_horizon_catalog', base_location_root='foo', base_location_subpath='bar', ) }} select * from {{ ref('jaffle_shop_customers') }} ``` note While you can customize paths with `base_location_root` and `base_location_subpath`, we don't recommend relying on them for environment isolation (such as separating development and production environments). Anyone with repository access can easily modify these configuration values. For true environment isolation, use separate `external_volume` values with infrastructure-level access controls. ###### Rationale By default, dbt manages `base_location` on behalf of users to enforce best practices. With Snowflake-managed Iceberg format tables, the user owns and maintains the data storage of the tables in an external storage solution (the declared `external volume`). The `base_location` parameter declares where to write the data within the external volume. The Snowflake Iceberg catalog keeps track of your Iceberg table regardless of where the data lives within the `external volume` declared and the `base_location` provided. However, Snowflake permits passing anything into the `base_location` field, including an empty string, even reusing the same path across multiple tables. This behavior could result in future technical debt because it limits the ability to: * Navigate the underlying object store (S3/Azure blob) * Read Iceberg tables through an object-store integration * Grant schema-specific access to tables through object store * Use a crawler pointed at the tables within the external volume to build a new catalog with another tool To maintain best practices, dbt enforces an input and, by default, writes your tables within a `_dbt/{SCHEMA_NAME}/{TABLE_NAME}` prefix to ensure easier object-store observability and auditability. ##### Limitations * When you use Iceberg tables with dbt, dbt materializes your query in Iceberg. However, dbt often creates intermediary objects as temporary and transient tables for certain materializations, such as incremental ones. You can't configure these temporary objects to be Iceberg-formatted. You may see non-Iceberg tables created in the logs to support specific materializations, but they are dropped after usage. * You can't incrementally update a pre-existing incremental model to be an Iceberg table. To do so, you must fully rebuild the table with the `--full-refresh` flag. * As of Snowflake change bundle `2025-01`, the `SHOW TABLES` command doesn't include the `is_iceberg` column in its output. This forced dbt v1.9 to run a command similar to the following query for all the models in the dbt project (regardless of whether they're configured as `iceberg` models): ```sql select all_objects.*, is_iceberg from table(result_scan(last_query_id(-1))) all_objects left join INFORMATION_SCHEMA.tables as all_tables on all_tables.table_name = all_objects."name" and all_tables.table_schema = all_objects."schema_name" and all_tables.table_catalog = all_objects."database_name" ``` This query may be relatively inefficient and potentially expensive, depending on the size of your Snowflake warehouse. Thus, the ability to run Iceberg models is gated behind the `enable_iceberg_materializations` flag. --- ### Source freshness dbt platform dbt provides a helpful interface around dbt's [source data freshness](https://docs.getdbt.com/docs/build/sources.md#source-data-freshness) calculations. When a dbt job is configured to run source freshness checks, dbt will render a user interface showing you the state of the most recent check. This interface is intended to help you determine if your source data freshness is meeting the service level agreement (SLA) that you've defined for your organization. [![Data Sources in dbt](/img/docs/dbt-platform/using-dbt-platform/data-sources-next.png?v=2 "Data Sources in dbt")](#)Data Sources in dbt ##### Enabling source freshness checks [`dbt build`](https://docs.getdbt.com/reference/commands/build.md) does *not* include source freshness checks when building and testing resources in your DAG. Instead, you can use one of these common patterns for defining jobs: * Add `dbt build` to the run step to run models, tests, and so on. * Select the **Generate docs on run** checkbox to automatically [generate project docs](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md). * Select the **Run source freshness** checkbox to enable [source freshness](#checkbox) as the first step of the job. [![Selecting source freshness](/img/docs/dbt-platform/select-source-freshness.png?v=2 "Selecting source freshness")](#)Selecting source freshness To enable source freshness checks, first make sure to configure your sources with [source freshness information](https://docs.getdbt.com/docs/build/sources.md#source-data-freshness). You can add source freshness to the list of commands in the job run steps or enable the checkbox. However, you can expect different outcomes when you configure a job by selecting the **Run source freshness** checkbox compared to adding the command to the run steps. Review the following options and outcomes: | Options | Outcomes | | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Select checkbox[]()** | The **Run source freshness** checkbox in your **Execution Settings** will run `dbt source freshness` as the first step in your job and won't break subsequent steps if it fails. If you wanted your job dedicated *exclusively* to running freshness checks, you still need to include at least one placeholder step, such as `dbt compile`. | | **Add as a run step** | Add the `dbt source freshness` command to a job anywhere in your list of run steps. However, if your source data is out of date — this step will "fail", and subsequent steps will not run. dbt will trigger email notifications (if configured) based on the end state of this step.<br /><br />You can create a new job to check source freshness.<br /><br />If you *do not* want your models to run if your source data is out of date, then it could be a good idea to run `dbt source freshness` as the first step in your job. Otherwise, we recommend adding `dbt source freshness` as the last step in the job, or creating a separate job just for this task. | [![Adding a step to check source freshness](/img/docs/dbt-platform/using-dbt-platform/job-step-source-freshness.png?v=2 "Adding a step to check source freshness")](#)Adding a step to check source freshness ##### Source freshness check frequency It's important that your freshness jobs run frequently enough to measure data latency in accordance with your SLAs. You can imagine that if you have a 1 hour SLA on a particular dataset, checking the freshness of that table once daily would not be appropriate. As a good rule of thumb, you should run your source freshness jobs with at least double the frequency of your lowest SLA. Here's an example table of some reasonable check frequencies given typical SLAs: | SLA | Check frequency | | ------ | --------------- | | 1 hour | 30 mins | | 1 day | 12 hours | | 1 week | About daily | #### Further reading * Refer to [Artifacts](https://docs.getdbt.com/docs/deploy/artifacts.md) for more info on how to create dbt artifacts, share links to the latest documentation, and share source freshness reports with your team. * Source freshness for Snowflake is calculated using the `LAST_ALTERED` column. Read about the limitations in [Snowflake configs](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#source-freshness-known-limitation). --- ### SQL models #### Related reference docs * [Model configurations](https://docs.getdbt.com/reference/model-configs.md) * [Model properties](https://docs.getdbt.com/reference/model-properties.md) * [`run` command](https://docs.getdbt.com/reference/commands/run.md) * [`ref` function](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) #### Getting started Building your first models If you're new to dbt, we recommend that you read a [quickstart guide](https://docs.getdbt.com/guides.md) to build your first dbt project with models. dbt's Python capabilities are an extension of its capabilities with SQL models. If you're new to dbt, we recommend that you read this page first, before reading: ["Python Models"](https://docs.getdbt.com/docs/build/python-models.md) A SQL model is a `select` statement. Models are defined in `.sql` files (typically in your `models` directory): * Each `.sql` file contains one model / `select` statement * The model name is inherited from the filename and must match the *filename* of a model — including case sensitivity. Any mismatched casing can prevent dbt from applying configurations correctly and may affect metadata in [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md). * We strongly recommend using underscores for model names, not dots. For example, use `models/my_model.sql` instead of `models/my.model.sql`. * Models can be nested in subdirectories within the `models` directory. * Starting in dbt v1.12, you can use Jinja-style suffixes (`.j2`, `.jinja`, `.jinja2`) on `.sql` files (for example, `my_model.sql.j2`) by enabling the [`allow_jinja_file_extensions`](https://docs.getdbt.com/reference/global-configs/behavior-flags/allow_jinja_file_extensions.md) flag. Refer to [How we style our dbt models](https://docs.getdbt.com/best-practices/how-we-style/1-how-we-style-our-dbt-models.md) for details on how we recommend you name your models. When you execute the [`dbt run` command](https://docs.getdbt.com/reference/commands/run.md), dbt will build this model data warehouse by wrapping it in a `create view as` or `create table as` statement. For example, consider this `customers` model: models/customers.sql ```sql with customer_orders as ( select customer_id, min(order_date) as first_order_date, max(order_date) as most_recent_order_date, count(order_id) as number_of_orders from jaffle_shop.orders group by 1 ) select customers.customer_id, customers.first_name, customers.last_name, customer_orders.first_order_date, customer_orders.most_recent_order_date, coalesce(customer_orders.number_of_orders, 0) as number_of_orders from jaffle_shop.customers left join customer_orders using (customer_id) ``` When you execute `dbt run`, dbt will build this as a *view* named `customers` in your target schema: ```sql create view dbt_alice.customers as ( with customer_orders as ( select customer_id, min(order_date) as first_order_date, max(order_date) as most_recent_order_date, count(order_id) as number_of_orders from jaffle_shop.orders group by 1 ) select customers.customer_id, customers.first_name, customers.last_name, customer_orders.first_order_date, customer_orders.most_recent_order_date, coalesce(customer_orders.number_of_orders, 0) as number_of_orders from jaffle_shop.customers left join customer_orders using (customer_id) ) ``` Why a *view* named `dbt_alice.customers`? By default dbt will: * Create models as views * Build models in a target schema you define * Use your file name as the view or table name in the database You can use *configurations* to change any of these behaviors — more on that later. ##### FAQs How can I see the SQL that dbt is running? To check out the SQL that dbt is running, you can look in: * dbt: * Within the run output, click on a model name, and then select "Details" * dbt v1: * The `target/compiled/` directory for compiled `select` statements * The `target/run/` directory for compiled `create` statements * The `logs/dbt.log` file for verbose logging. Do I need to create my target schema before running dbt? Nope! dbt will check if the schema exists when it runs. If the schema does not exist, dbt will create it for you. If I rerun dbt, will there be any downtime as models are rebuilt? Nope! The SQL that dbt generates behind the scenes ensures that any relations are replaced atomically (i.e. your business users won't experience any downtime). The implementation of this varies on each warehouse, check out the [logs](https://docs.getdbt.com/faqs/Runs/checking-logs.md) to see the SQL dbt is executing. What happens if the SQL in my query is bad or I get a database error? If there's a mistake in your SQL, dbt will return the error that your database returns. ```shell $ dbt run --select customers Running with dbt=1.9.0 Found 3 models, 9 tests, 0 snapshots, 0 analyses, 133 macros, 0 operations, 0 seed files, 0 sources 14:04:12 | Concurrency: 1 threads (target='dev') 14:04:12 | 14:04:12 | 1 of 1 START view model dbt_alice.customers.......................... [RUN] 14:04:13 | 1 of 1 ERROR creating view model dbt_alice.customers................. [ERROR in 0.81s] 14:04:13 | 14:04:13 | Finished running 1 view model in 1.68s. Completed with 1 error and 0 warnings: Database Error in model customers (models/customers.sql) Syntax error: Expected ")" but got identifier `your-info-12345` at [13:15] compiled SQL at target/run/jaffle_shop/customers.sql Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1 ``` Any models downstream of this model will also be skipped. Use the error message and the [compiled SQL](https://docs.getdbt.com/faqs/Runs/checking-logs.md) to debug any errors. Which SQL dialect should I write my models in? Or which SQL dialect does dbt use? dbt can feel like magic, but it isn't actually magic. Under the hood, it's running SQL in your own warehouse — your data is not processed outside of your warehouse. As such, your models should just use the **SQL dialect of your own database**. Then, when dbt wraps your `select` statements in the appropriate DDL or DML, it will use the correct DML for your warehouse — all of this logic is written in to dbt. You can find more information about the databases, platforms, and query engines that dbt supports in the [Supported Data Platforms](https://docs.getdbt.com/docs/supported-data-platforms.md) docs. Want to go a little deeper on how this works? Consider a snippet of SQL that works on each warehouse: models/test\_model.sql ```sql select 1 as my_column ``` To replace an existing table, here's an *illustrative* example of the SQL dbt will run on different warehouses (the actual SQL can get much more complicated than this!) ##### Redshift ```sql -- you can't create or replace on redshift, so use a transaction to do this in an atomic way begin; create table "dbt_alice"."test_model__dbt_tmp" as ( select 1 as my_column ); alter table "dbt_alice"."test_model" rename to "test_model__dbt_backup"; alter table "dbt_alice"."test_model__dbt_tmp" rename to "test_model" commit; begin; drop table if exists "dbt_alice"."test_model__dbt_backup" cascade; commit; ``` ##### BigQuery ```sql -- Make an API call to create a dataset (no DDL interface for this)!!; create or replace table `dbt-dev-87681`.`dbt_alice`.`test_model` as ( select 1 as my_column ); ``` ##### Snowflake ```sql create schema if not exists analytics.dbt_alice; create or replace table analytics.dbt_alice.test_model as ( select 1 as my_column ); ``` #### Configuring models Configurations are "model settings" that you can set in your `dbt_project.yml` file, *and* in your model file using a `config` block. Some example configurations include: * Changing the materialization that a model uses — a [materialization](https://docs.getdbt.com/docs/build/materializations.md) determines the SQL that dbt uses to create the model in your warehouse. * Build models into separate [schemas](https://docs.getdbt.com/docs/build/custom-schemas.md). * Apply [tags](https://docs.getdbt.com/reference/resource-configs/tags.md) to a model. The following diagram shows an example directory structure of a models folder: ```text models ├── staging └── marts └── marketing ``` Here's an example of a model configuration: dbt\_project.yml ```yaml name: jaffle_shop config-version: 2 ... models: jaffle_shop: # this matches the `name:`` config +materialized: view # this applies to all models in the current project marts: +materialized: table # this applies to all models in the `marts/` directory marketing: +schema: marketing # this applies to all models in the `marts/marketing/`` directory ``` models/customers.sql ```sql {{ config( materialized="view", schema="marketing" ) }} with customer_orders as ... ``` It is important to note that configurations are applied hierarchically — a configuration applied to a subdirectory will override any general configurations. You can learn more about configurations in the [reference docs](https://docs.getdbt.com/reference/model-configs.md). ##### FAQs What materializations are available in dbt? dbt ships with five built-in materializations: `view`, `table`, `incremental`, `ephemeral`, and `materialized_view`. Check out the documentation on [materializations](https://docs.getdbt.com/docs/build/materializations.md) for more information on each of these options. You can also create your own [custom materializations](https://docs.getdbt.com/guides/create-new-materializations.md). This is an advanced feature of dbt. What model configurations exist? You can also configure: * [tags](https://docs.getdbt.com/reference/resource-configs/tags.md) to support easy categorization and graph selection * [custom schemas](https://docs.getdbt.com/reference/resource-properties/schema.md) to split your models across multiple schemas * [aliases](https://docs.getdbt.com/reference/resource-configs/alias.md) if your view/table name should differ from the filename * Snippets of SQL to run at the start or end of a model, known as [hooks](https://docs.getdbt.com/docs/build/hooks-operations.md) * Warehouse-specific configurations for performance (e.g. `sort` and `dist` keys on Redshift, `partitions` on BigQuery) Check out the docs on [model configurations](https://docs.getdbt.com/reference/model-configs.md) to learn more. #### Building dependencies between models You can build dependencies between models by using the [`ref` function](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) in place of table names in a query. Use the name of another model as the argument for `ref`. ##### Model models/customers.sql ```sql with customers as ( select * from {{ ref('stg_customers') }} ), orders as ( select * from {{ ref('stg_orders') }} ), ... ``` ##### Compiled code in dev ```sql create view dbt_alice.customers as ( with customers as ( select * from dbt_alice.stg_customers ), orders as ( select * from dbt_alice.stg_orders ), ... ) ... ``` ##### Compiled code in prod ```sql create view analytics.customers as ( with customers as ( select * from analytics.stg_customers ), orders as ( select * from analytics.stg_orders ), ... ) ... ``` dbt uses the `ref` function to: * Determine the order to run the models by creating a dependent acyclic graph (DAG). [![The DAG for our dbt project](/img/dbt-dag.png?v=2 "The DAG for our dbt project")](#)The DAG for our dbt project * Manage separate environments — dbt will replace the model specified in the `ref` function with the database name for the table (or view). Importantly, this is environment-aware — if you're running dbt with a target schema named `dbt_alice`, it will select from an upstream table in the same schema. Check out the tabs above to see this in action. Additionally, the `ref` function encourages you to write modular transformations, so that you can re-use models, and reduce repeated code. #### Testing and documenting models You can also document and test models — skip ahead to the section on [testing](https://docs.getdbt.com/docs/build/data-tests.md) and [documentation](https://docs.getdbt.com/docs/build/documentation.md) for more information. #### Additional FAQs Are there any example dbt models? Yes! * **Quickstart Tutorial:** You can build your own example dbt project in the [quickstart guide](https://docs.getdbt.com/docs/get-started-dbt.md) * **Jaffle Shop:** A demonstration project (closely related to the tutorial) for a fictional e-commerce store. Refer to [Clone the Jaffle Shop sample project](https://docs.getdbt.com/guides/clone-jaffle-shop.md) for clone instructions, or view the [main source code](https://github.com/dbt-labs/jaffle-shop) and [source code using DuckDB](https://github.com/dbt-labs/jaffle_shop_duckdb) on GitHub. * **GitLab:** Gitlab's internal dbt project is open source and is a great example of how to use dbt at scale ([source code](https://gitlab.com/gitlab-com/content-sites/handbook/blob/main/content/handbook/enterprise-data/platform/dbt-guide.md)) * **dummy-dbt:** A containerized dbt project that populates the Sakila database in Postgres and populates dbt seeds, models, snapshots, and tests. The project can be used for testing and experimentation purposes ([source code](https://github.com/gmyrianthous/dbt-dummy)) * **Google Analytics 4:** A demonstration project that transforms the Google Analytics 4 BigQuery exports to various models ([source code](https://github.com/stacktonic-com/stacktonic-dbt-example-project), [docs](https://stacktonic.com/article/google-analytics-big-query-and-dbt-a-dbt-example-project)) * **Make Open Data:** A production-grade ELT with tests, documentation, and CI/CD (GHA) about French open data (housing, demography, geography, etc). It can be used to learn with voluminous and ambiguous data. Contributions are welcome ([source code](https://github.com/make-open-data/make-open-data), [docs](https://make-open-data.fr/)) If you have an example project to add to this list, suggest an edit by clicking **Edit this page** below. Can I store my models in a directory other than the \`models\` directory in my project? By default, dbt expects the files defining your models to be located in the `models` subdirectory of your project. To change this, update the [model-paths](https://docs.getdbt.com/reference/project-configs/model-paths.md) configuration in your `dbt_project.yml` file, like so: dbt\_project.yml ```yml model-paths: ["transformations"] ``` Can I build my models in a schema other than my target schema or split my models across multiple schemas? Yes! Use the [schema](https://docs.getdbt.com/reference/resource-configs/schema.md) configuration in your `dbt_project.yml` file, or using a `config` block: dbt\_project.yml ```yml name: jaffle_shop ... models: jaffle_shop: marketing: +schema: marketing # models in the `models/marketing/` subdirectory will use the marketing schema ``` models/customers.sql ```sql {{ config( schema='core' ) }} ``` Do ref-able resource names need to be unique? Within one project: yes! To build dependencies between resources (such as models, seeds, and snapshots), you need to use the `ref` function, and pass in the resource name as an argument. dbt uses that resource name to uniquely resolve the `ref` to a specific resource. As a result, these resource names need to be unique, *even if they are in distinct folders*. A resource in one project can have the same name as a resource in another project (installed as a dependency). dbt uses the project name to uniquely identify each resource. We call this "namespacing." If you `ref` a resource with a duplicated name, it will resolve to the resource within the same namespace (package or project), or raise an error because of an ambiguous reference. Use [two-argument `ref`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models) to disambiguate references by specifying the namespace. Those resource will still need to land in distinct locations in the data warehouse. Read the docs on [custom aliases](https://docs.getdbt.com/docs/build/custom-aliases.md) and [custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md) for details on how to achieve this. How do I remove deleted models from my data warehouse? If you delete a model from your dbt project, dbt does not automatically drop the relation from your schema. This means that you can end up with extra objects in schemas that dbt creates, which can be confusing to other users. (This can also happen when you switch a model from being a view or table, to ephemeral) When you remove models from your dbt project, you should manually drop the related relations from your schema. As I create more models, how should I keep my project organized? What should I name my models? There's no one best way to structure a project! Every organization is unique. If you're just getting started, check out how we (dbt Labs) [structure our dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md). If models can only be \`select\` statements, how do I insert records? For those coming from an ETL (Extract Transform Load) paradigm, there's often a desire to write transformations as `insert` and `update` statements. In comparison, dbt will wrap your `select` query in a `create table as` statement, which can feel counter-productive. * If you wish to use `insert` statements for performance reasons (i.e. to reduce data that is processed), consider [incremental models](https://docs.getdbt.com/docs/build/incremental-models.md) * If you wish to use `insert` statements since your source data is constantly changing (e.g. to create "Type 2 Slowly Changing Dimensions"), consider [snapshotting your source data](https://docs.getdbt.com/docs/build/sources.md#source-data-freshness), and building models on top of your snaphots. Why can't I just write DML in my transformations? ###### `select` statements make transformations accessible More people know how to write `select` statements, than DML, making the transformation layer accessible to more people! ###### Writing good DML is hard If you write the DDL / DML yourself you can end up getting yourself tangled in problems like: * What happens if the table already exists? Or this table already exists as a view, but now I want it to be a table? * What if the schema already exists? Or, should I check if the schema already exists? * How do I replace a model atomically (such that there's no down-time for someone querying the table) * What if I want to parameterize my schema so I can run these transformations in a development environment? * What order do I need to run these statements in? If I run a `cascade` does it break other things? Each of these problems *can* be solved, but they are unlikely to be the best use of your time. ###### dbt does more than generate SQL You can test your models, generate documentation, create snapshots, and more! ###### You reduce your vendor lock in SQL dialects tend to diverge the most in DML and DDL (rather than in `select` statements) — check out the example [here](https://docs.getdbt.com/faqs/Models/sql-dialect.md). By writing less SQL, it can make a migration to a new database technology easier. If you do need to write custom DML, there are ways to do this in dbt using [custom materializations](https://docs.getdbt.com/guides/create-new-materializations.md). How do I specify column types? Simply cast the column to the correct type in your model: ```sql select id, created::timestamp as created from some_other_table ``` You might have this question if you're used to running statements like this: ```sql create table dbt_alice.my_table id integer, created timestamp; insert into dbt_alice.my_table ( select id, created from some_other_table ) ``` In comparison, dbt would build this table using a `create table as` statement: ```sql create table dbt_alice.my_table as ( select id, created from some_other_table ) ``` So long as your model queries return the correct column type, the table you create will also have the correct column type. To define additional column options: * Rather than enforcing uniqueness and not-null constraints on your column, use dbt's [data testing](https://docs.getdbt.com/docs/build/data-tests.md) functionality to check that your assertions about your model hold true. * Rather than creating default values for a column, use SQL to express defaults (e.g. `coalesce(updated_at, current_timestamp()) as updated_at`) * In edge-cases where you *do* need to alter a column (e.g. column-level encoding on Redshift), consider implementing this via a [post-hook](https://docs.getdbt.com/reference/resource-configs/pre-hook-post-hook.md). --- ### Supported data platforms dbt connects to and runs SQL against your database, warehouse, lake, or query engine. These SQL-speaking platforms are collectively referred to as *data platforms*. dbt connects with data platforms by using a dedicated adapter plugin for each. Plugins are built as Python modules that dbt v1 discovers if they are installed on your system. Refer to the [Build, test, document, and promote adapters](https://docs.getdbt.com/guides/adapter-creation.md) guide for details. (Applies to dbt v2.0 and later) #### Adapter lifecycle dbt v2 is available across adapters (data warehouse connectors). Track status by adapter using the following table: | Adapter | Lifecycle | | ----------------------- | --------- | | Snowflake | Preview | | BigQuery | Preview | | Databricks | Preview | | Redshift | Preview | | Apache Spark (CLI only) | Beta | | DuckDB (CLI only) | Beta | *Note that adapter lifecycle may differ between the dbt platform and local development. An adapter can reach GA in the dbt platform before it reaches GA for local use.* #### Types of Adapters There are two types of adapters available today: * **Trusted** — [Trusted adapters](https://docs.getdbt.com/docs/trusted-adapters.md) are those where the adapter maintainers have decided to participate in the Trusted Adapter Program and have made a commitment to meeting those requirements. For adapters supported in dbt, maintainers have undergone an additional rigorous process that covers contractual requirements for development, documentation, user experience, and maintenance. * **Community** — [Community adapters](https://docs.getdbt.com/docs/community-adapters.md) are open-source and maintained by community members. These adapters are not part of the Trusted Adapter Program and could have usage inconsistencies. Considerations for depending on an open-source project 1. Does it work? 2. Does anyone "own" the code, or is anyone liable for ensuring it works? 3. Do bugs get fixed quickly? 4. Does it stay up-to-date with new dbt v1 features? 5. Is the usage substantial enough to self-sustain? 6. Do other known projects depend on this library? --- ### Tableau dbt platform | Starter, Enterprise, Enterprise+ The Tableau integration allows you to use worksheets to query the Semantic Layer directly and produce your dashboards with trusted data. It provides a live connection to the Semantic Layer through Tableau Desktop or Tableau Server. #### Prerequisites * You have [configured the Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md) and are using dbt v1.6 or higher. * You must have [Tableau Desktop](https://www.tableau.com/en-gb/products/desktop) version 2021.1 and greater, Tableau Server, or [Tableau Cloud](https://www.tableau.com/products/cloud-bi). * Log in to Tableau Desktop (with Cloud or Server credentials) or Tableau Cloud. You can also use a licensed Tableau Server deployment. * You need your [dbt host](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md#3-view-connection-detail), [Environment ID](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md#set-up-dbt-semantic-layer), and a [service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) or a [personal access token](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) to log in. This account should be set up with the Semantic Layer. * You must have a dbt Starter or Enterprise-tier [account](https://www.getdbt.com/pricing). Suitable for both Multi-tenant and Single-tenant deployment. 📹 Learn about the dbt Semantic Layer with on-demand video courses! Explore our [dbt Semantic Layer on-demand course](https://learn.getdbt.com/courses/semantic-layer) to learn how to define and query metrics in your dbt project. Additionally, dive into mini-courses for querying the dbt Semantic Layer in your favorite tools: [Tableau](https://courses.getdbt.com/courses/tableau-querying-the-semantic-layer), [Excel](https://learn.getdbt.com/courses/querying-the-semantic-layer-with-excel), [Hex](https://courses.getdbt.com/courses/hex-querying-the-semantic-layer), and [Mode](https://courses.getdbt.com/courses/mode-querying-the-semantic-layer). #### Installing the connector The Semantic Layer Tableau connector is available to download directly on [Tableau Exchange](https://exchange.tableau.com/products/1020). The connector is supported in Tableau Desktop, Tableau Server, and Tableau Cloud. Alternatively, you can follow these steps to install the connector. Note that these steps only apply to Tableau Desktop and Tableau Server. The connector for Tableau Cloud is managed by Tableau. 1. Download the GitHub [connector file](https://github.com/dbt-labs/semantic-layer-tableau-connector/releases/latest/download/dbt_semantic_layer.taco) locally and add it to your default folder: | Operating system | Tableau Desktop | Tableau Server | | ---------------- | --------------------------------------------------------------------- | ------------------------------------- | | Windows | `C:\Users\\[Windows User]\Documents\My Tableau Repository\Connectors` | `C:\Program Files\Tableau\Connectors` | | Mac | `/Users/[user]/Documents/My Tableau Repository/Connectors` | Not applicable | | Linux | `/opt/tableau/connectors` | `/opt/tableau/connectors` | 2. Install the [JDBC driver](https://docs.getdbt.com/docs/dbt-apis/sl-jdbc.md) to the folder based on your operating system: * Windows: `C:\Program Files\Tableau\Drivers` * Mac: `~/Library/Tableau/Drivers` or `/Library/JDBC` or `~/Library/JDBC` * Linux: `/opt/tableau/tableau_driver/jdbc` 3. Open Tableau Desktop or Tableau Server and find the **Semantic Layer by dbt Labs** connector on the left-hand side. You may need to restart these applications for the connector to be available. 4. Connect with your Host, Environment ID, and service or personal token information dbt provides during the [Semantic Layer configuration](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md). * In Tableau Server, the authentication screen may show "User" & "Password" instead, in which case the User is the Environment ID and the password is the Service Token. #### Using the integration 1. **Authentication** — Once you authenticate, the system will direct you to the data source page. 2. **Access all Semantic Layer Objects** — Use the "ALL" data source to access all the metrics, dimensions, and entities configured in your Semantic Layer. Note that the "METRICS\_AND\_DIMENSIONS" data source has been deprecated and replaced by "ALL". Be sure to use a live connection since extracts are not supported at this time. 3. **Access saved queries** — You can optionally access individual [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md) that you've defined. These will also show up as unique data sources when you log in. 4. **Access worksheet** — From your data source selection, go directly to a worksheet in the bottom left-hand corner. 5. **Query metrics and dimensions** — Then, you'll find all the metrics, dimensions, and entities that are available to query on the left side of your window based on your selection. Visit the [Tableau documentation](https://help.tableau.com/current/pro/desktop/en-us/gettingstarted_overview.htm) to learn more about how to use Tableau worksheets and dashboards. ##### Publish from Tableau Desktop to Tableau Server * **From Desktop to Server** — Like any Tableau workflow, you can publish your workbook from Tableau Desktop to Tableau Server. For step-by-step instructions, visit Tableau's [publishing guide](https://help.tableau.com/current/pro/desktop/en-us/publish_workbooks_share.htm). ###### Modifying time granularity When you select time dimensions in the **Group By** menu, you'll see a list of available time granularities. The lowest granularity is selected by default. Metric time is the default time dimension for grouping your metrics. info Note: [Custom time granularities](https://docs.getdbt.com/docs/build/metricflow-time-spine.md#add-custom-granularities) (like fiscal year) aren't currently supported or accessible in this integration. Only [standard granularities](https://docs.getdbt.com/docs/build/dimensions.md?dimension=time_gran#time) (like day, week, month, and so on) are available. If you'd like to access custom granularities, consider using the [Semantic Layer APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md). #### Things to note **Aggregation**<br /> * All metrics are shown as using the "SUM" aggregation type in Tableau's UI, and this cannot be altered using Tableau's interface. * The Semantic Layer controls the aggregation type in code and it is intentionally fixed. Keep in mind that the underlying aggregation in the Semantic Layer might not be "SUM" ("SUM" is Tableau's default). **Data sources and display**<br /> * In the "ALL" data source, Tableau surfaces all metrics and dimensions from the Semantic Layer on the left-hand side. Note, that not all metrics and dimensions can be combined. You will receive an error message if a particular dimension cannot be sliced with a metric (or vice versa). You can use saved queries for smaller pieces of data that you want to combine. * To display available metrics and dimensions, Semantic Layer returns metadata for a fake table with the dimensions and metrics as 'columns' on this table. Because of this, you can't actually query this table for previews or extracts. **Calculations and querying**<br /> * Certain Table calculations like "Totals" and "Percent Of" may not be accurate when using metrics aggregated in a non-additive way (such as count distinct) * In any of our Semantic Layer interfaces (not only Tableau), you must include a [time dimension](https://docs.getdbt.com/docs/build/cumulative.md#limitations) when working with any cumulative metric that has a time window or granularity. * We can support calculated fields for creating parameter filters or dynamically selecting metrics and dimensions. However, other uses of calculated fields are not supported. * *Note: For calculated field use cases that are not currently covered, please reach out to [dbt Support](<mailto:support@getdbt.com?subject=dbt Semantic Layer feedback>) and share them so we can further understand.* * When using saved queries that include filters, we will automatically apply any filters that the query has. #### Unsupported functionality The following Tableau features aren't supported at this time, however, the Semantic Layer may support some of this functionality in a future release: * Updating the data source page * Using "Extract" mode to view your data * Unioning Tables * Writing Custom SQL / Initial SQL * Table Extensions * Cross-Database Joins * Some functions in Analysis --> Create Calculated Field * Filtering on a Date Part time dimension for a Cumulative metric type * Changing your date dimension to use "Week Number" * Performing joins between tables that the Semantic Layer creates. It handles joins for you, so there's no need to join components in the Semantic Layer. Note, that you *can* join tables from the Semantic Layer to ones outside your data platform. * The Tableau integration doesn't currently display descriptive labels defined in your `metrics` configuration, meaning custom labels won't be visible when those metrics are imported/queried into Tableau. #### FAQs I'm receiving an \`Failed ALPN\` error when trying to connect to the dbt Semantic Layer. If you're receiving a `Failed ALPN` error when trying to connect the dbt Semantic Layer with the various [data integration tools](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) (such as Tableau, DBeaver, Datagrip, ADBC, or JDBC), it typically happens when connecting from a computer behind a corporate VPN or Proxy (like Zscaler or Check Point). The root cause is typically the proxy interfering with the TLS handshake as the Semantic Layer uses gRPC/HTTP2 for connectivity. To resolve this: * If your proxy supports gRPC/HTTP2 but isn't configured to allow ALPN, adjust its settings accordingly to allow ALPN. Or create an exception for the dbt domain. * If your proxy does not support gRPC/HTTP2, add an SSL interception exception for the dbt domain in your proxy settings This should help in successfully establishing the connection without the Failed ALPN error. --- ### Trial and billing Start a dbt Wizard trial, set a spend limit, and manage paid access for both the dbt platform and local CLI. This page covers how to get dbt Wizard access and pay for it. For the models dbt Wizard can use and how tokens and credits work, refer to [Models and pricing](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/overview.md). Billing and spend controls are shared across the dbt platform and local CLI. Both surfaces draw from the same account-level balance. #### Prerequisites * Account admin or billing admin permissions to start a trial or change a spend limit. * A dbt account to manage usage, billing, spend limits, and more. * If you don't have one, you can create one during setup. No paid dbt platform plan required. This is generally useful for users on self-hosted dbt running the CLI. * A business email address. Personal domains such as Gmail aren't eligible for a trial. #### What you get by plan Every new account gets free dbt Wizard usage credits to start. What you get, and how you keep going, depends on your plan. | Plan | What you get | How it renews | When it runs out | | -------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------- | ------------------------- | ------------------------------------------------------------------------------------ | | Developer and Starter, or self-hosted dbt with a free dbt account | 30-day trial with $100 in usage credits per account | One-time | Add a credit card and set a monthly spend limit or contact your account team. | | Enterprise, including [legacy Enterprise](https://docs.getdbt.com/docs/platform/billing/plans-and-billing.md#legacy-plans) | $100/month in usage credits per account | Resets each billing month | [Contact your account team](https://www.getdbt.com/contact) to add committed spend | | Enterprise+ | $200/month in usage credits per account | Resets each billing month | [Contact your account team](https://www.getdbt.com/contact) to add committed spend | | [Legacy Team](https://docs.getdbt.com/docs/platform/billing/plans-and-billing.md#legacy-plans) | No access to dbt Wizard | — | Move to a [Starter, Enterprise, or Enterprise+ plan](https://www.getdbt.com/pricing) | * On Developer and Starter plans, the trial is opt-in, so it won't start automatically. It ends when you use up the $100 in credits or after 30 days, whichever comes first, and unused credits don't carry over. * Legacy Enterprise plans get the same dbt Wizard access and monthly usage credits as current Enterprise. Only legacy Team has no access. * Enterprise and Enterprise+ usage credits are granted automatically — there's no trial to start and no credit card required. Credits don't roll over or get prorated, and if you downgrade out of Enterprise or Enterprise+, any unused credits are removed at the plan change. * Enterprise and Enterprise+ accounts should add a committed spend amount to their contract to keep using dbt Wizard. You may lose access to dbt Wizard without this commit in place. If you've set an optional monthly dbt Wizard spend limit, that still applies and pauses usage once reached. Running dbt Wizard from the CLI against a self-hosted dbt project? Run `dbt login` (or `wizard login`) to get the same 30-day trial. The command creates your free dbt account and provisions the trial together, and that account is where you manage usage and spend limits. Everything on this page describes dbt managed billing — usage that dbt Labs bills through your dbt account. If you bring your own key, your AI provider bills you directly and none of this applies. Refer to [BYOK for dbt platform](https://docs.getdbt.com/docs/platform/wizard-byok-platform.md) or [BYOK for the CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) instead. #### Start your trial Start your trial from anywhere in the dbt platform or from the dbt Wizard CLI. Note, Enterprise-tiered plans automatically have a [spend limit set](https://docs.getdbt.com/docs/dbt-ai/wizard-billing-faqs.md#what-does-the-dbt-wizard-enterprise-plan-monthly-usage-credits-include) ##### dbt platform Start from **Billing & Usage**, or from the dbt Wizard prompt in Studio IDE or the home tab as they all take you to the same flow. 1. Click your account name, then select **Account settings**. (Or click on any button that says "Start trial" to start your Wizard trial.) 2. Under **Settings**, click **Billing & Usage**. 3. On the **Overview** tab, find the **dbt Wizard** card and click **Start trial**. No credit card is required. Your 30-day trial with $100 in usage credits starts right away. The $100 is per account and shared by everyone on it, and you can track how much you've used anytime in **Billing & Usage**. [![The Billing & Usage Overview page, showing dbt State and dbt Wizard cards with Start trial buttons, plus a usage-by-month chart](/img/docs/dbt-platform/wizard-billing-overview.png?v=2 "The Billing & Usage Overview page, showing dbt State and dbt Wizard cards with Start trial buttons, plus a usage-by-month chart")](#)The Billing & Usage Overview page, showing dbt State and dbt Wizard cards with Start trial buttons, plus a usage-by-month chart ##### Wizard CLI There's no **Start trial** button in the CLI. Logging in is what starts your trial — one command creates your free dbt account, if you don't have one, and provisions the 30-day trial with $100 in usage credits for your account at the same time. The $100 is per account, not per user, so it's shared with anyone else on your account. 1. [Install dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md). 2. Run `dbt login` and complete the browser sign-in, or create a new account to manage your dbt Wizard spend limits: ```shell dbt login ``` 3. Run `wizard` in your project and choose **dbt-managed** when onboarding asks how AI usage is billed. You don't need an AI provider key. You don't need a paid dbt platform plan, so this path is the same whether you're on a dbt platform plan or running against a self-hosted dbt project. Refer to [Use dbt Wizard locally](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) for the full onboarding walkthrough. #### Set up paid usage What you do next depends on your plan. ##### Developer, Starter, or self hosted 1. Go to **Billing & Usage > Usage-based features**. 2. Click **Set up billing**. 3. Add a credit card and fill in your payment details. Click **Save card**. 4. Complete your setup to choose a preset monthly spend limit, or set your own with the **Custom** option. 5. Optionally, turn on automatic increases for when you're close to your limit. 6. Click **Continue**. 7. Review and confirm your spend limit. Click **Activate dbt Wizard** to start your billing. You won't be charged today and will be billed monthly for actual usage, up to the limit you set. Usage then pauses if you reach that limit. [![The Add billing page, showing a credit card form and a complete billing setup flow](/img/docs/dbt-platform/wizard-add-billing.png?v=2 "The Add billing page, showing a credit card form and a complete billing setup flow")](#)The Add billing page, showing a credit card form and a complete billing setup flow [![The Set your dbt Wizard spend limit page, showing pre-set monthly options, a Custom option, and an auto-raise toggle](/img/docs/dbt-platform/wizard-manage-spend.png?v=2 "The Set your dbt Wizard spend limit page, showing pre-set monthly options, a Custom option, and an auto-raise toggle")](#)The Set your dbt Wizard spend limit page, showing pre-set monthly options, a Custom option, and an auto-raise toggle [![The Activate dbt Wizard page where you can review and confirm your spend limit and a button to activate dbt Wizard](/img/docs/dbt-platform/wizard-activate.png?v=2 "The Activate dbt Wizard page where you can review and confirm your spend limit and a button to activate dbt Wizard")](#)The Activate dbt Wizard page where you can review and confirm your spend limit and a button to activate dbt Wizard ##### Enterprise and Enterprise+ plans There's no trial to start and no self-serve credit card flow. Your monthly usage credits — $100/month on Enterprise and $200/month on Enterprise+ — are granted automatically per account, not per user, and shared by everyone on the account. [Contact your account team](https://www.getdbt.com/contact) to set up or adjust committed spend. Enterprise and Enterprise+ accounts should add a committed spend amount to their contract to keep using dbt Wizard. You may lose access to dbt Wizard without this commit in place. If you've set an optional monthly dbt Wizard spend limit, that still applies and pauses usage once reached. #### Manage your spend limit Your spend limit caps how much dbt managed dbt Wizard usage your account can consume in a billing period, across both the dbt platform and local development. * You only pay for actual usage, up to the limit you choose. The limit is a cap, not a prepaid charge. * If you reach your limit, dbt Wizard usage pauses until you raise it or the next billing cycle starts. This applies on every plan, including Enterprise-tiered plans that set the optional limit. Enterprise-tiered accounts that don't have a committed spend amount will be prompted to connect with their account rep about adding a spend amount. * Limits are set separately for dbt Wizard and [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md), but both draw from your account's overall usage-based spend. To view or update your limit, go to **Billing & Usage > Usage-based features > Wizard**. Enterprise-tiered plans can [contact their account team](https://www.getdbt.com/contact) to adjust their limit. [![The Manage your dbt Wizard spend limit page, showing pre-set monthly options and a Custom option.](/img/docs/dbt-platform/wizard-manage-spend.png?v=2 "The Manage your dbt Wizard spend limit page, showing pre-set monthly options and a Custom option.")](#)The Manage your dbt Wizard spend limit page, showing pre-set monthly options and a Custom option. #### View your usage and costs To see what you've spent, go to **Account settings > Billing & Usage > Usage-based features** and open the **Wizard** tab. From there you can check: * How much of your included monthly usage you've used, what's left, and when it resets. * Your current dbt Wizard spend limit, with an **Edit** button to change it. * **dbt Wizard usage by model**, which breaks down your usage (in UTC) so you can see which models are driving your costs. #### How usage is measured dbt Wizard usage is measured in tokens, then converted into dollar-based usage based on the model and the token type, such as input, cached read, cache write, or output. Refer to [Key terms](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/overview.md#key-terms) for what a token is, and the [Model Provider Rate Table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) for current rates. #### Related docs * [dbt Wizard billing FAQs](https://docs.getdbt.com/docs/dbt-ai/wizard-billing-faqs.md) for common billing questions * [Models and pricing](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/overview.md) for model options and token pricing * [BYOK for dbt platform](https://docs.getdbt.com/docs/platform/wizard-byok-platform.md) or [BYOK for the CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) for bring-your-own-key setup * [Billing](https://docs.getdbt.com/docs/platform/billing.md) for general dbt platform billing --- ### Trusted adapters Trusted adapters take part in the Trusted Adapter Program, including a commitment to meet the program's requirements. They are maintained by dbt Labs, partners, and community members. Trusted adapters in dbt undergo an additional rigorous process that covers development, documentation, user experience, and maintenance requirements. We strongly recommend using them in production environments. For further details, refer to [What it means to be trusted](https://docs.getdbt.com/guides/adapter-creation.md?step=8#what-it-means-to-be-trusted). Free and open-source tools for the data professional are increasingly abundant. This is by-and-large a *good thing*, however it requires due diligence that wasn't required in a paid-license, closed-source software world. As a user, there are important questions to answer before taking a dependency on an open-source project. The trusted adapter designation is meant to streamline this process for end users. ##### Trusted adapter specifications Refer to the [Build, test, document, and promote adapters](https://docs.getdbt.com/guides/adapter-creation.md) guide for more information, particularly if you are an adapter maintainer considering having your adapter be added to the trusted list. ##### Trusted adapters ![](/img/icons/alloydb.svg) ###### AlloyDB * [Set up in the dbt platform](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-postgresql-alloydb.md) <br /> * [Install with dbt v1](https://docs.getdbt.com/docs/local/connect-data-platform/alloydb-setup.md) <br /> [![](https://badge.fury.io/py/dbt-postgres.svg/)](https://badge.fury.io/py/dbt-postgres) dbt platformdbt v1 ![](/img/icons/apache-spark.svg) ###### Apache Spark * [Set up in the dbt platform](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-apache-spark.md) <br /> * [Install with dbt v1](https://docs.getdbt.com/docs/local/connect-data-platform/spark-setup.md) * [Install with dbt v2](https://docs.getdbt.com/docs/local/connect-data-platform/spark-setup.md) <br /> [![](https://badge.fury.io/py/dbt-spark.svg/)](https://badge.fury.io/py/dbt-spark) ![](/img/icons/athena.svg) ###### Athena * [Set up in the dbt platform](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-amazon-athena.md) <br /> * [Install with dbt v1](https://docs.getdbt.com/docs/local/connect-data-platform/athena-setup.md) <br /> <br /> [![](https://badge.fury.io/py/dbt-athena.svg/)](https://badge.fury.io/py/dbt-athena) dbt platformdbt v1 ![](/img/icons/azure-synapse-analytics.svg) ###### Azure Synapse * [Set up in the dbt platform](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-azure-synapse-analytics.md) <br /> * [Install with dbt v1](https://docs.getdbt.com/docs/local/connect-data-platform/azuresynapse-setup.md) <br /> [![](https://badge.fury.io/py/dbt-synapse.svg/)](https://badge.fury.io/py/dbt-synapse) dbt platformdbt v1 ![](/img/icons/bigquery.svg) ###### BigQuery * [Set up in the dbt platform](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-bigquery.md) <br /> * [Install with dbt v1](https://docs.getdbt.com/docs/local/connect-data-platform/bigquery-setup.md) <br /> * [Install with dbt v2](https://docs.getdbt.com/docs/local/connect-data-platform/bigquery-setup.md) <br /> [![](https://badge.fury.io/py/dbt-bigquery.svg/)](https://badge.fury.io/py/dbt-bigquery) ![](/img/icons/clickhouse.svg) ###### ClickHouse * [Install with dbt v1](https://docs.getdbt.com/docs/local/connect-data-platform/clickhouse-setup.md) <br /> [![](https://badge.fury.io/py/dbt-clickhouse.svg/)](https://badge.fury.io/py/dbt-clickhouse) dbt v1 ![](/img/icons/databricks.svg) ###### Databricks * [Set up in the dbt platform](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-databricks.md) <br /> * [Install with dbt v1](https://docs.getdbt.com/docs/local/connect-data-platform/databricks-setup.md) <br /> * [Install with dbt v2](https://docs.getdbt.com/docs/local/connect-data-platform/databricks-setup.md) <br /> [![](https://badge.fury.io/py/dbt-databricks.svg/)](https://badge.fury.io/py/dbt-databricks) ![](/img/icons/duckdb-seeklogo.svg) ###### DuckDB * [Install with dbt v1](https://docs.getdbt.com/docs/local/connect-data-platform/duckdb-setup.md?version=1) <br /> * [Install with dbt v2](https://docs.getdbt.com/docs/local/connect-data-platform/duckdb-setup.md?version=2) <br /> [![](https://badge.fury.io/py/dbt-duckdb.svg/)](https://badge.fury.io/py/dbt-duckdb) ![](/img/icons/dremio.svg) ###### Dremio * [Install with dbt v1](https://docs.getdbt.com/docs/local/connect-data-platform/dremio-setup.md) <br /> <br /> [![](https://badge.fury.io/py/dbt-dremio.svg/)](https://badge.fury.io/py/dbt-dremio) dbt v1 ![](/img/icons/glue.svg) ###### Glue * [Install with dbt v1](https://docs.getdbt.com/docs/local/connect-data-platform/glue-setup.md) <br /> <br /> [![](https://badge.fury.io/py/dbt-glue.svg/)](https://badge.fury.io/py/dbt-glue) dbt v1 ![](/img/icons/exasol.svg) ###### Exasol * [Install with dbt v1](https://docs.getdbt.com/docs/local/connect-data-platform/exasol-setup.md) <br /> <br /> [![](https://badge.fury.io/py/dbt-exasol.svg/)](https://badge.fury.io/py/dbt-exasol) dbt v1 ![](/img/icons/dbt-hive.svg) ###### Cloudera Hive * [Install with dbt v1](https://docs.getdbt.com/docs/local/connect-data-platform/hive-setup.md) <br /> <br /> [![](https://badge.fury.io/py/dbt-hive.svg/)](https://badge.fury.io/py/dbt-hive) dbt v1 ![](/img/icons/dbt-ibm-db2.svg) ###### IBM Db2 * [Install with dbt v1](https://docs.getdbt.com/docs/local/connect-data-platform/ibm-db2-setup.md) <br /> <br /> [![](https://badge.fury.io/py/ibm-dbt-db2.svg/)](https://badge.fury.io/py/ibm-dbt-db2) dbt v1 ![](/img/icons/dbt-ibm-netezza.svg) ###### IBM Netezza * [Install with dbt v1](https://docs.getdbt.com/docs/local/connect-data-platform/ibmnetezza-setup.md) <br /> <br /> [![](https://badge.fury.io/py/dbt-ibm-netezza.svg/)](https://badge.fury.io/py/dbt-ibm-netezza) dbt v1 ![](/img/icons/dbt-impala.svg) ###### Cloudera Impala * [Install with dbt v1](https://docs.getdbt.com/docs/local/connect-data-platform/impala-setup.md) <br /> <br /> [![](https://badge.fury.io/py/dbt-impala.svg/)](https://badge.fury.io/py/dbt-impala) dbt v1 ![](/img/icons/lakebase.svg) ###### Databricks Lakebase * [Install with dbt v1](https://docs.getdbt.com/docs/local/connect-data-platform/lakebase-setup.md) <br /> <br /> [![](https://badge.fury.io/py/dbt-postgres.svg/)](https://badge.fury.io/py/dbt-postgres) dbt platformdbt v1 ![](/img/icons/materialize.svg) ###### Materialize * [Install with dbt v1](https://docs.getdbt.com/docs/local/connect-data-platform/materialize-setup.md) <br /> <br /> [![](https://badge.fury.io/py/dbt-materialize.svg/)](https://badge.fury.io/py/dbt-materialize) dbt v1 ![](/img/icons/fabric_warehouse.svg) ###### Microsoft Fabric Warehouse * [Set up in the dbt platform](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-microsoft-fabric.md) <br /> * [Install with dbt v1](https://docs.getdbt.com/docs/local/connect-data-platform/fabric-setup.md) <br /> [![](https://badge.fury.io/py/dbt-fabric.svg/)](https://badge.fury.io/py/dbt-fabric) dbt platformdbt v1 ![](/img/icons/fabric_lakehouse.svg) ###### Microsoft Fabric Lakehouse * [Install with dbt v1](https://docs.getdbt.com/docs/local/connect-data-platform/fabricspark-setup.md) <br /> [![](https://badge.fury.io/py/dbt-fabricspark.svg/)](https://badge.fury.io/py/dbt-fabricspark) dbt v1 ![](/img/icons/oracle.svg) ###### Oracle Autonomous Database * [Install with dbt v1](https://docs.getdbt.com/docs/local/connect-data-platform/oracle-setup.md) <br /> [![](https://badge.fury.io/py/dbt-oracle.svg/)](https://badge.fury.io/py/dbt-oracle) dbt v1 ![](/img/icons/postgres.svg) ###### Postgres * [Set up in the dbt platform](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-postgresql-alloydb.md) <br /> * [Install with dbt v1](https://docs.getdbt.com/docs/local/connect-data-platform/postgres-setup.md) <br /> [![](https://badge.fury.io/py/dbt-postgres.svg/)](https://badge.fury.io/py/dbt-postgres) dbt platformdbt v1 ![](/img/icons/redshift.svg) ###### Redshift * [Set up in the dbt platform](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-redshift.md) <br /> * [Install with dbt v1](https://docs.getdbt.com/docs/local/connect-data-platform/redshift-setup.md) <br /> * [Install with dbt v2](https://docs.getdbt.com/docs/local/connect-data-platform/redshift-setup.md) <br /> [![](https://badge.fury.io/py/dbt-redshift.svg/)](https://badge.fury.io/py/dbt-redshift) ![](/img/icons/risingwave.svg) ###### RisingWave * [Install with dbt v1](https://docs.getdbt.com/docs/local/connect-data-platform/risingwave-setup.md) <br /> <br /> [![](https://badge.fury.io/py/dbt-risingwave.svg/)](https://badge.fury.io/py/dbt-risingwave) dbt v1 ![](/img/icons/singlestore.svg) ###### SingleStore * [Install with dbt v1](https://docs.getdbt.com/docs/local/connect-data-platform/singlestore-setup.md) <br /> <br /> [![](https://badge.fury.io/py/dbt-singlestore.svg/)](https://badge.fury.io/py/dbt-singlestore) dbt v1 ![](/img/icons/snowflake.svg) ###### Snowflake * [Set up in the dbt platform](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-snowflake.md) <br /> * [Install with dbt v1](https://docs.getdbt.com/docs/local/connect-data-platform/snowflake-setup.md) <br /> * [Install with dbt v2](https://docs.getdbt.com/docs/local/connect-data-platform/snowflake-setup.md) <br /> [![](https://badge.fury.io/py/dbt-snowflake.svg/)](https://badge.fury.io/py/dbt-snowflake) ![](/img/icons/starburst.svg) ###### Starburst/Trino * [Set up in the dbt platform](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-starburst-trino.md) <br /> * [Install with dbt v1](https://docs.getdbt.com/docs/local/connect-data-platform/trino-setup.md) <br /> [![](https://badge.fury.io/py/dbt-trino.svg/)](https://badge.fury.io/py/dbt-trino) dbt platformdbt v1 ![](/img/icons/teradata.svg) ###### Teradata * [Set up in the dbt platform](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-teradata.md) <br /> * [Install with dbt v1](https://docs.getdbt.com/docs/local/connect-data-platform/teradata-setup.md) <br /> [![](https://badge.fury.io/py/dbt-teradata.svg/)](https://badge.fury.io/py/dbt-teradata) dbt platformdbt v1 --- ### Unit tests 💡Did you know... Available from dbt v1.8 or with the [dbt "v1 Latest" release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). Historically, dbt's test coverage was confined to [“data” tests](https://docs.getdbt.com/docs/build/data-tests.md), assessing the quality of input data or resulting datasets' structure. However, these tests could only be executed *after* building a model. There is an additional type of test in dbt: unit tests. In software programming, unit tests validate small portions of your functional code, and they work much the same way here. Unit tests allow you to validate your SQL modeling logic on a small set of static inputs *before* you materialize your full model in production. Unit tests enable test-driven development, benefiting developer efficiency and code reliability. #### Prerequisites * We currently only support unit testing SQL models. * We currently only support adding unit tests to models in your *current* project. * We currently *don't* support unit testing models that use the [`materialized view`](https://docs.getdbt.com/docs/build/materializations.md#materialized-view) materialization. * We currently *don't* support unit testing models that use recursive SQL. * We currently *don't* support unit testing models that use introspective queries. * If your model has multiple versions, by default the unit test will run on *all* versions of your model. Read [unit testing versioned models](https://docs.getdbt.com/reference/resource-properties/unit-testing-versions.md) for more information. * Unit tests must be defined in a YML file in your [`models/` directory](https://docs.getdbt.com/reference/project-configs/model-paths.md). * Table names must be aliased in order to unit test `join` logic. * Include all [`ref`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) or [`source`](https://docs.getdbt.com/reference/dbt-jinja-functions/source.md) model references in the unit test configuration as `input`s to avoid "node not found" errors during compilation. Unit tests are discovered from `model-paths` (by default, the `models/` directory), so define them alongside your models in a `.yml` file under your `model-paths`. Don't define unit test YAML in the `tests/` directory, which is reserved for [data tests](https://docs.getdbt.com/docs/build/data-tests.md). ###### Adapter-specific caveats * You must specify all fields in a BigQuery `STRUCT` in a unit test. You cannot use only a subset of fields in a `STRUCT`. * Redshift customers need to be aware of a [limitation when building unit tests](https://docs.getdbt.com/reference/resource-configs/redshift-configs.md#unit-test-limitations) that requires a workaround. * Redshift sources need to be in the same database as the models. tip Check out our [Unit tests on-demand course](https://learn.getdbt.com/learn/course/unit-testing/welcome-to-unit-testing-5min/introduction-to-unit-testing) to learn how to add unit tests and more! Read the [reference doc](https://docs.getdbt.com/reference/resource-properties/unit-tests.md) for more details about formatting your unit tests. ##### When to add a unit test to your model You should unit test a model: * When your SQL contains complex logic: * Regex * Date math * Window functions * `case when` statements when there are many `when`s * Truncation * When you're writing custom logic to process input data, similar to creating a function. * We don't recommend conducting unit testing for functions like `min()` since these functions are tested extensively by the warehouse. If an unexpected issue arises, it's more likely a result of issues in the underlying data rather than the function itself. Therefore, fixture data in the unit test won't provide valuable information. * Logic for which you had bugs reported before. * Edge cases not yet seen in your actual data that you want to handle. * Prior to refactoring the transformation logic (especially if the refactor is significant). * Models with high "criticality" (public, contracted models or models directly upstream of an exposure). ##### When to run unit tests dbt Labs strongly recommends only running unit tests in development or CI environments. Since the inputs of the unit tests are static, there's no need to use additional compute cycles running them in production. Use them in development for a test-driven approach and CI to ensure changes don't break them. Use the [resource type](https://docs.getdbt.com/reference/global-configs/resource-type.md) flag `--exclude-resource-type` or the (Applies to dbt v1.11 and later) `DBT_ENGINE_EXCLUDE_RESOURCE_TYPES` environment variable to exclude unit tests from your production builds and save compute. To run only unit tests on demand, use the `test_type` selector — this works across all engines (dbt v1 and dbt v2): ```bash dbt test --select "test_type:unit" ``` (Applies to dbt v2.0 and later) #### Run unit tests locally [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") You can run unit tests locally when you're working through tricky SQL and you want to know right away whether your logic works. By default, each unit test sends a query to your data platform and waits for the result. This can slow down testing and use warehouse compute. Because unit tests use static fixtures instead of real data, they don’t need to necessarily run on your data platform. Use the [`compute: local` config](https://docs.getdbt.com/reference/resource-configs/compute.md) to run them locally with DuckDB for faster feedback *without* the warehouse compute cost. That gives you: * A test loop that keeps up with you. Change your SQL, rerun the test, and repeat without waiting on a warehouse queue. * Room to test as you go, so you catch broken logic while you're still writing it instead of finding it in CI (or worse, after it reaches production) * Queries that execute with the same functions and runtime semantics as the remote warehouse * Your data platform's compute left for building models. ##### Prerequisites * Snowflake or BigQuery. Local execution isn't available for other data platforms, and it only applies to unit tests. * The direct upstream models of the model you're testing already exist in your data platform. dbt fetches their schemas to translate your SQL, so if they don't exist, the test fails with an error about fetching the upstream relation schema. * `static_analysis` isn't set to `off` on the test. Local execution needs static analysis to translate your SQL, so `compute: local` promotes [static analysis](https://docs.getdbt.com/reference/resource-configs/static-analysis.md) to `strict` for that test. If you set `static_analysis: off`, the test can't run locally and fails with `ExecutorFailed (dbt1401)`. ##### What to know before you use it * dbt fetches upstream schemas the first time you run the test, then caches them. Later runs reuse the cache, so they don't re-fetch. * Local execution only works if dbt can compile your model's SQL and translate it to DuckDB. Complex SQL and functions specific to your data platform might have no DuckDB equivalent so Snowflake's `AI_CLASSIFY` and `haversine` are two examples. So for example, a model that calls `haversine` fails with `failed in db_runner: Internal: Catalog Error: Scalar Function with name haversine does not exist!`. * A translation failure is a test failure. `local` doesn't fall back to your data platform, so the test fails and dbt exits with a non-zero code. ##### How to configure You can configure it on a single unit test: models/schema.yml ```yaml unit_tests: - name: test_is_valid_email_address model: dim_customers config: compute: local ``` Or on every unit test in your project: dbt\_project.yml ```yaml unit_tests: my_project: +compute: local ``` [`compute`](https://docs.getdbt.com/reference/resource-configs/compute.md) accepts two values: | Value | What it does | | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `remote` | Sends the test to your data platform to run, using warehouse compute like any other query. This is the default, so you only need to set it explicitly to opt a test out of a project-level `+compute: local`. | | `local` | Runs the test with DuckDB, wherever dbt itself is running. Nothing is sent to your data platform, so the test returns quickly and uses no warehouse compute. | You might also see `sidecar` in error messages but it means the same thing as `local`. #### Unit testing a model This example creates a new `dim_customers` model with a field `is_valid_email_address` that calculates whether or not the customer’s email is valid: ```sql with customers as ( select * from {{ ref('stg_customers') }} ), accepted_email_domains as ( select * from {{ ref('top_level_email_domains') }} ), check_valid_emails as ( select customers.customer_id, customers.first_name, customers.last_name, customers.email, coalesce (regexp_like( customers.email, '^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}$' ) = true and accepted_email_domains.tld is not null, false) as is_valid_email_address from customers left join accepted_email_domains on customers.email_top_level_domain = lower(accepted_email_domains.tld) ) select * from check_valid_emails ``` The logic posed in this example can be challenging to validate. You can add a unit test to this model to ensure the `is_valid_email_address` logic captures all known edge cases: emails without `.`, emails without `@`, and emails from invalid domains. ```yaml unit_tests: - name: test_is_valid_email_address description: "Check my is_valid_email_address logic captures all known edge cases - emails without ., emails without @, and emails from invalid domains." model: dim_customers given: - input: ref('stg_customers') format: dict rows: - {email: cool@example.com, email_top_level_domain: example.com} - {email: cool@unknown.com, email_top_level_domain: unknown.com} - {email: badgmail.com, email_top_level_domain: gmail.com} - {email: missingdot@gmailcom, email_top_level_domain: gmail.com} - input: ref('top_level_email_domains') format: dict rows: - {tld: example.com} - {tld: gmail.com} expect: format: dict rows: - {email: cool@example.com, is_valid_email_address: true} - {email: cool@unknown.com, is_valid_email_address: false} - {email: badgmail.com, is_valid_email_address: false} - {email: missingdot@gmailcom, is_valid_email_address: false} ``` The previous example defines the mock data using the inline `dict` format, but you can also use `csv` or `sql` either inline or in a separate fixture file. Store your fixture files in a `fixtures` subdirectory in any of your [test paths](https://docs.getdbt.com/reference/project-configs/test-paths.md). For example, `tests/fixtures/my_unit_test_fixture.sql`. The following examples show how to define mock data and expected output using `csv` and `sql`. models/schema.yml ```yaml unit_tests: - name: test_is_valid_email_address__csv model: dim_customers given: - input: ref('stg_customers') format: dict rows: - {email: cool@example.com, email_top_level_domain: example.com} - {email: cool@unknown.com, email_top_level_domain: unknown.com} - {email: badgmail.com, email_top_level_domain: gmail.com} - {email: missingdot@gmailcom, email_top_level_domain: gmail.com} - input: ref('top_level_email_domains') format: csv rows: | tld example.com gmail.com expect: format: csv fixture: valid_email_address_fixture_output ``` models/schema.yml ```yaml unit_tests: - name: test_is_valid_email_address__sql model: dim_customers given: - input: ref('stg_customers') format: dict rows: - {email: cool@example.com, email_top_level_domain: example.com} - {email: cool@unknown.com, email_top_level_domain: unknown.com} - {email: badgmail.com, email_top_level_domain: gmail.com} - {email: missingdot@gmailcom, email_top_level_domain: gmail.com} - input: ref('top_level_email_domains') format: sql rows: | select 'example.com' as tld union all select 'gmail.com' as tld expect: format: sql fixture: valid_email_address_fixture_output ``` When using the `dict` or `csv` format, you only have to define the mock data for the columns relevant to you. This enables you to write succinct and *specific* unit tests. note The direct parents of the model that you’re unit testing (in this example, `stg_customers` and `top_level_email_domains`) need to exist in the warehouse before you can execute the unit test. Use the [`--empty`](https://docs.getdbt.com/reference/commands/build.md#the---empty-flag) flag to build an empty version of the models to save warehouse spend. ```bash dbt run --select "stg_customers top_level_email_domains" --empty ``` Alternatively, use `dbt build` to, in lineage order: * Run the unit tests on your model. * Materialize your model in the warehouse. * Run the data tests on your model. Now you’re ready to run this unit test. You have a couple of options for commands depending on how specific you want to be: * `dbt test --select dim_customers` runs *all* of the tests on `dim_customers`. * `dbt test --select "dim_customers,test_type:unit"` runs all of the *unit* tests on `dim_customers`. * `dbt test --select test_is_valid_email_address` runs the test named `test_is_valid_email_address`. ```shell dbt test --select test_is_valid_email_address 16:03:49 Running with dbt=1.8.0-a1 16:03:49 Registered adapter: postgres=1.8.0-a1 16:03:50 Found 6 models, 5 seeds, 4 data tests, 0 sources, 0 exposures, 0 metrics, 410 macros, 0 groups, 0 semantic models, 1 unit test 16:03:50 16:03:50 Concurrency: 5 threads (target='postgres') 16:03:50 16:03:50 1 of 1 START unit_test dim_customers::test_is_valid_email_address ................... [RUN] 16:03:51 1 of 1 FAIL 1 dim_customers::test_is_valid_email_address ............................ [FAIL 1 in 0.26s] 16:03:51 16:03:51 Finished running 1 unit_test in 0 hours 0 minutes and 0.67 seconds (0.67s). 16:03:51 16:03:51 Completed with 1 error and 0 warnings: 16:03:51 16:03:51 Failure in unit_test test_is_valid_email_address (models/marts/unit_tests.yml) 16:03:51 actual differs from expected: @@ ,email ,is_valid_email_address → ,cool@example.com,True→False ,cool@unknown.com,False ...,... ,... 16:03:51 16:03:51 compiled Code at models/marts/unit_tests.yml 16:03:51 16:03:51 Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1 ``` The clever regex statement wasn’t as clever as initially thought, as the model incorrectly flagged `cool@example.com` as an invalid email address. Updating the regex logic to `'^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}$'` (those pesky escape characters) and rerunning the unit test solves the problem: ```shell dbt test --select test_is_valid_email_address 16:09:11 Running with dbt=1.8.0-a1 16:09:12 Registered adapter: postgres=1.8.0-a1 16:09:12 Found 6 models, 5 seeds, 4 data tests, 0 sources, 0 exposures, 0 metrics, 410 macros, 0 groups, 0 semantic models, 1 unit test 16:09:12 16:09:13 Concurrency: 5 threads (target='postgres') 16:09:13 16:09:13 1 of 1 START unit_test dim_customers::test_is_valid_email_address ................... [RUN] 16:09:13 1 of 1 PASS dim_customers::test_is_valid_email_address .............................. [PASS in 0.26s] 16:09:13 16:09:13 Finished running 1 unit_test in 0 hours 0 minutes and 0.75 seconds (0.75s). 16:09:13 16:09:13 Completed successfully 16:09:13 16:09:13 Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1 ``` Your model is now ready for production! Adding this unit test helped catch an issue with the SQL logic *before* you materialized `dim_customers` in your warehouse and will better ensure the reliability of this model in the future. #### Unit testing incremental models When configuring your unit test, you can override the output of macros, vars, or environment variables. This enables you to unit test your incremental models in "full refresh" and "incremental" modes. note Incremental models need to exist in the database before running unit tests or doing a `dbt build`. Use the [`--empty` flag](https://docs.getdbt.com/reference/commands/build.md#the---empty-flag) to build an empty version of the models to save warehouse spend. You can also optionally select only your incremental models using the [`--select` flag](https://docs.getdbt.com/reference/node-selection/syntax.md#shorthand). ```shell dbt run --select "config.materialized:incremental" --empty ``` After running the command, you can then perform a regular `dbt build` for that model and then run your unit test. When testing an incremental model, the expected output is the **result of the materialization** (what will be merged/inserted), not the resulting model itself (what the final table will look like after the merge/insert). For example, say you have an incremental model in your project: my\_incremental\_model.sql ```sql {{ config( materialized='incremental' ) }} select * from {{ ref('events') }} {% if is_incremental() %} where event_time > (select max(event_time) from {{ this }}) {% endif %} ``` You can define unit tests on `my_incremental_model` to ensure your incremental logic is working as expected: ```yaml unit_tests: - name: my_incremental_model_full_refresh_mode model: my_incremental_model overrides: macros: # unit test this model in "full refresh" mode is_incremental: false given: - input: ref('events') rows: - {event_id: 1, event_time: 2020-01-01} expect: rows: - {event_id: 1, event_time: 2020-01-01} - name: my_incremental_model_incremental_mode model: my_incremental_model overrides: macros: # unit test this model in "incremental" mode is_incremental: true given: - input: ref('events') rows: - {event_id: 1, event_time: 2020-01-01} - {event_id: 2, event_time: 2020-01-02} - {event_id: 3, event_time: 2020-01-03} - input: this # contents of current my_incremental_model rows: - {event_id: 1, event_time: 2020-01-01} expect: # what will be inserted/merged into my_incremental_model rows: - {event_id: 2, event_time: 2020-01-02} - {event_id: 3, event_time: 2020-01-03} ``` There is currently no way to unit test whether the dbt framework inserted/merged the records into your existing model correctly, but [we're investigating support for this in the future](https://github.com/dbt-labs/dbt/issues/8664). #### Unit testing a model that depends on ephemeral model(s) If you want to unit test a model that depends on an ephemeral model, you must use `format: sql` for that input. ```yaml unit_tests: - name: my_unit_test model: dim_customers given: - input: ref('ephemeral_model') format: sql rows: | select 1 as id, 'emily' as first_name expect: rows: - {id: 1, first_name: emily} ``` #### Unit test exit codes Unit test successes and failures are represented by two exit codes: * Pass (0) * Fail (1) Exit codes differ from data test success and failure outputs because they don't directly reflect failing data tests. Data tests are queries designed to check specific conditions in your data, and they return one row per failed test case (for example, the number of values with duplicates for the `unique` test). dbt reports the number of failing records as failures. Whereas, each unit test represents one 'test case', so results are always 0 (pass) or 1 (fail) regardless of how many records failed within that test case. Learn about [exit codes](https://docs.getdbt.com/reference/exit-codes.md) for more information. #### Additional resources * [Unit testing reference page](https://docs.getdbt.com/reference/resource-properties/unit-tests.md) * [Supported data formats for mock data](https://docs.getdbt.com/reference/resource-properties/data-formats.md) * [Unit testing versioned models](https://docs.getdbt.com/reference/resource-properties/unit-testing-versions.md) * [Unit test inputs](https://docs.getdbt.com/reference/resource-properties/unit-test-input.md) * [Unit test overrides](https://docs.getdbt.com/reference/resource-properties/unit-test-overrides.md) * [Platform-specific data types](https://docs.getdbt.com/reference/resource-properties/data-types.md) --- ### Upgrade to dbt v2 Preview ### Upgrade to dbt v2 [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Local development After [installing the dbt VS Code extension](https://docs.getdbt.com/docs/install-dbt-extension.md), use the **Get started** panel to upgrade your project to dbt v2 if you haven't already done so. note If you are already running dbt v2, you must be on version `2.0.0-beta.66` or higher to use the upgrade tool. The dbt extension provides two ways to upgrade your project to dbt v2 from the **Get started** panel: * [**Agentic migration**:](#agentic-migration) Runs the dbt v1-to-dbt v2 migration with or AI agents. Select this option in the **Check dbt v2 compatibility** step. Requires or AI agents. * [**Manual CLI onboarding**:](#manual-cli-onboarding) Walks you through the upgrade in your terminal. Use this if you prefer the CLI or don't use or AI agents. #### Agentic migration 1. Select **Start with an agent** from the **Check dbt v2 compatibility** step. 2. The extension installs the **Migrate dbt v1 to dbt v2** agent skill to your editor's skill folder, then opens your AI chat with a migration prompt. 3. Click **Install & open chat** to continue. The chat opens with the migration prompt already loaded, so you can run it directly from the editor. No CLI commands are required. #### Manual CLI onboarding The **Get started** panel has an upgrade assistant that guides you through the upgrade process in your terminal. [![The dbt extension Get started panel and upgrade assistant.](/img/docs/extension/vsce-manual-upgrade.png?v=2 "The dbt extension Get started panel and upgrade assistant.")](#)The dbt extension Get started panel and upgrade assistant. You can start manual onboarding in either of the following ways: ##### From the Get started panel * From the **Get started** panel, select **Start manually in CLI** in the **Check dbt v2 compatibility** step. * Follow the prompts in the upgrade assistant to complete the upgrade. * Run `dbtf compile` to verify your project is ready for dbt v2. [![The message received when you have completed upgrading your project to v2.](/img/docs/extension/fusion-onboarding-complete.png?v=2 "The message received when you have completed upgrading your project to v2.")](#)The message received when you have completed upgrading your project to v2. Once the upgrade is completed, you're ready to dive into all the features that dbt v2 has to offer! Next, [sign in or register](https://docs.getdbt.com/docs/sign-in-dbt-extension.md) for a dbt platform account to keep using advanced features after the 14-day trial. ##### From your terminal * From your terminal, run: ```shell dbt init --fusion-upgrade ``` The upgrade tool guides you through a series of prompts:  Do you have an existing dbt platform account? If you answer `Y`, the tool provides instructions for downloading your dbt platform profile or authenticating. If you answer `N`, the flow continues to local project setup.  Ready to run dbt init? If no `profiles.yml` file is present, the tool guides you through creating one and connecting to your data warehouse.  Ready to run dbt debug? If a `profiles.yml` file exists, the tool validates that your project is configured correctly and can connect to your data warehouse.  Ready to run dbt parse? The tool parses your dbt project to check compatibility with dbt v2. * If parsing fails, the tool can run [dbt-autofix](https://github.com/dbt-labs/dbt-autofix?tab=readme-ov-file#installation) to help resolve errors. * If you skip dbt-autofix, you can run it later or manually fix errors. The upgrade tool can't continue until parsing errors are resolved.  Ready to run dbt compile with static analysis off? After parsing succeeds, the tool compiles your project without static analysis. This mimics dbt v1 behavior by rendering Jinja into SQL while temporarily disabling dbt v2's advanced SQL comprehension. AI Agents There are cases where dbt-autofix may not resolve all errors and requires manual intervention. For those cases, the dbt-autofix tool provides an [AI Agents.md](https://github.com/dbt-labs/dbt-autofix/blob/main/AGENTS.md) file to help AI agents continue the migration work after dbt-autofix has completed its part.  Ready to run dbt compile? The tool compiles your project with full dbt v2 static analysis. This checks that your SQL code is valid in the context of your warehouse's tables and columns. [![The message received when you have completed upgrading your project to v2.](/img/docs/extension/fusion-onboarding-complete.png?v=2 "The message received when you have completed upgrading your project to v2.")](#)The message received when you have completed upgrading your project to v2. Once the upgrade is completed, you're ready to dive into all the features that dbt v2 has to offer! Next, [sign in or register](https://docs.getdbt.com/docs/sign-in-dbt-extension.md) for a dbt platform account to keep using advanced features after the 14-day trial. #### Next steps Once you've upgraded your project to dbt v2, you can: * [Sign in or register](https://docs.getdbt.com/docs/sign-in-dbt-extension.md) for a dbt platform account to keep using advanced features after the 14-day trial. * [Configure your local environment](https://docs.getdbt.com/docs/configure-dbt-extension.md) to mirror your dbt platform environment and [set environment variables](https://docs.getdbt.com/docs/configure-dbt-extension.md#configure-environment-variables) required by your project. * [Compare changes locally](https://docs.getdbt.com/docs/dbt/vs-compare-changes.md) to preview data changes caused by your local edits. * [Optimize static analysis for development and deployment](https://docs.getdbt.com/best-practices/optimize-static-analysis-for-development-and-deployment.md) for stronger local validation without slowing deployment jobs. * Review the [limitations and unsupported features](https://docs.getdbt.com/docs/dbt/supported-features.md#limitations). --- ### Upgrade versions in dbt platform dbt platform In dbt, both [jobs](https://docs.getdbt.com/docs/deploy/jobs.md) and [environments](https://docs.getdbt.com/docs/dbt-platform-environments.md) are configured to use a specific version of dbt. The version can be upgraded at any time. #### Environments Navigate to the settings page of an environment, then click **Edit**. Click the **dbt version** dropdown bar and make your selection. You can select a [release track](#release-tracks) to receive ongoing updates (recommended), or a legacy version of dbt v1. Be sure to save your changes before navigating away. [![Example environment settings in dbt](/img/docs/dbt-platform/platform-configuring-dbt-platform/choosing-dbt-version/example-environment-settings.png?v=2 "Example environment settings in dbt")](#)Example environment settings in dbt ##### Release Tracks Starting in 2024, your project gets upgraded automatically on a cadence that you choose: The **v1 Latest** track ensures you have up-to-date dbt functionality, and early access to new features of the dbt framework. The **v1 Compatible** and **v1 Extended** tracks are designed for customers who need a less-frequent release cadence, the ability to test new dbt releases before they go live in production, and/or ongoing compatibility with the latest open source releases of dbt v1. As a best practice, dbt Labs recommends that you test the upgrade in development first; use the [Override dbt version](#override-dbt-version) setting to test *your* project on the latest dbt version before upgrading your deployment environments and the default development environment for all your colleagues. To upgrade an environment in the [dbt Admin API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md) or [Terraform](https://registry.terraform.io/providers/dbt-labs/dbtcloud/latest), set `dbt_version` to the name of your release track: * `fusion-nightly` * `fusion-stable` (formerly `latest-fusion`) * `fusion-extended` * `fusion-fallback` * `latest` (default) * `compatible` (available to Starter, Enterprise, Enterprise+ plans) * `extended` (available to all Enterprise plans) ##### Override dbt version Configure your project to use a different dbt version than what's configured in your [development environment](https://docs.getdbt.com/docs/dbt-platform-environments.md#types-of-environments). This *override* only affects your user account, no one else's. Use this to safely test new dbt features before upgrading the dbt version for your projects. 1. Click your account name from the left side panel and select **Account settings**. 2. Choose **Credentials** from the sidebar and select a project. This opens a side panel. 3. In the side panel, click **Edit** and scroll to the **User development settings** section. 4. Choose a version from the **dbt version** dropdown and click **Save**. An example of overriding the configured version to [**v1 Latest** release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) for the selected project: [![Example of overriding the dbt version on your user account](/img/docs/dbt-platform/platform-configuring-dbt-platform/choosing-dbt-version/example-override-version.png?v=2 "Example of overriding the dbt version on your user account")](#)Example of overriding the dbt version on your user account 5. (Optional) Verify that dbt will use your override setting to build the project by invoking a `dbt build` command in the Studio IDE's command bar. Expand the **System Logs** section and find the output's first line. It should begin with `Running with dbt=` and list the version dbt is using. <br /> <br /> For users on Release tracks, the output will display `Running dbt...` instead of a specific version, reflecting the flexibility and continuous automatic updates provided by the release track functionality. #### dbt v2 dbt Labs has introduced the new [dbt v2](https://docs.getdbt.com/docs/introduction.md), a ground-up rebuild of dbt. This is currently generally available for Snowflake projects and in preview for other supported adapters on the dbt platform. Eligible customers can update environments to dbt v2 using the same workflows as v1.x, but remember: * If you don't see the **v2 Stable** release track as an option, you should check with your dbt Labs account team about eligibility. * To increase the compatibility of your project, update all jobs and environments to the **v1 Latest** release track and read more about the changes in our [upgrade guide](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md). * Make sure you're using a [supported adapter](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md?version=2.0) and authentication method:  BigQuery[Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") * Service Account / User Token * Native OAuth * External OAuth * [Workload Identity Federation](https://docs.getdbt.com/docs/platform/manage-access/set-up-bigquery-oauth.md#set-up-bigquery-workload-identity-federation) (Microsoft Entra) * [Required permissions](https://docs.getdbt.com/docs/local/connect-data-platform/bigquery-setup.md#required-permissions)  Databricks[Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") * Service Account / User Token * Native OAuth  Redshift[Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") * Username / Password * IAM profile  Snowflake * Username / Password * Native OAuth * External OAuth * Key pair using a modern PKCS#8 method * MFA *Adapter lifecycle can differ between the dbt platform and local development — an adapter can reach GA in the dbt platform before it reaches GA for local use.* <br /> * Once you upgrade your development environment(s) to `v2 Stable`, every user will have to restart the IDE. [![Upgrade to v2 in your environment settings.](/img/docs/dbt-platform/platform-configuring-dbt-platform/platform-upgrading-dbt-versions/upgrade-fusion.png?v=2 "Upgrade to v2 in your environment settings.")](#)Upgrade to v2 in your environment settings. ##### Upgrading environments to dbt v2 When you're ready to upgrade your project(s) to dbt v2, there are some tools available to you in the dbt platform UI to help you get started. The dbt v2 upgrade assistant will step you through the process of preparing and upgrading your projects. [![The v2 upgrade assistant.](/img/docs/dbt-platform/platform-configuring-dbt-platform/choosing-dbt-version/fusion-upgrade-gui.png?v=2 "The v2 upgrade assistant.")](#)The v2 upgrade assistant. ###### Prerequisites To take advantage of the upgrade assistant and other upgrade tools, you'll need to meet the following prerequisites: * Your dbt project must be updated to use the **v1 Latest** release track. * You must have a `developer` license. * You must have the proper [permissions set](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) to execute individual upgrade tasks. Migrating to dbt v2 is a multi-step process and some of these steps may be repeated across projects by different users: | Upgrade task | Required permission(s) | Supported permission sets | | ----------------------------------------------- | ------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------- | | Enable dbt v2 access (triggers migration flows) | dbt v2 readiness: `write`<br />Projects: `write` | Admin, Account Admin, and anyone assigned the v2 Migration Admin set, provided their base role also has `write` access to projects | | View dbt v2 readiness and job eligibility | dbt v2 readiness: `read` | Developer, Admin, Member, Account Admin, v2 Migration Admin | | **Run once on dbt v2** job action | Runs: `write` | Job Admin, Job Runner, Admin, Member, Project Creator, Account Admin | | Enable **v2 Stable** for an environment | Environments: `write` | Job Admin, Admin, Member, Project Creator, Account Admin | ###### Assign access to upgrade The dbt v2 readiness & upgrade flow are controlled by two account-level settings that an admin must configure. Step 1: Enable dbt v2 readiness features (required) The upgrade assistant and readiness panel only appear after enabling this setting. From your **Account settings**: 1. Navigate to the **Account** screen. 2. Click **Edit** and scroll to the **Settings** section. 3. Click the box next to **Enable dbt v2 readiness & upgrade features**. 4. Click **Save**. Once enabled, all admins and developers can see each project's v2 readiness status and which jobs are eligible or ineligible for dbt v2. Admins can also initiate the dbt v2 upgrade from development environments, environment settings, and job settings (subject to existing user permissions). Developer-licensed users will have access to debug tools to help make projects dbt v2 eligible in both development and production environments. Step 2: Restrict upgrade access (optional, Enterprise/Enterprise+ only) By default, all admins and developer-licensed users can access the dbt v2 readiness & upgrade flow. To restrict upgrade execution to users with the `v2 Migration Admin` permission set, enable this additional setting. From your **Account settings**: 1. Navigate to the **Account** screen. 2. Click **Edit** and scroll to the **Settings** section. 3. Click the box next to **Enable restricted dbt v2 upgrade permissions**. 4. Click **Save**. This hides the dbt v2 upgrade workflows from users who don't have the [`v2 Migration Admin`](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#v2-migration-admin) permission set. To grant access to the upgrade workflows to specific projects and/or specific users: 1. Navigate to an existing group in your **Account settings** and click **Edit**, or click [**Create group**](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#create-new-groups) to create a new one. 2. Scroll to the **Access and permissions** section and click **Add permission**. 3. Select the **v2 Migration Admin** permission set from the dropdown, then select the project(s) you want the users to access. 4. Click **Save**. The dbt v2 upgrade workflows helps identify areas of the project that need to be updated and provides tools for manually resolving and autofixing any errors. ###### Upgrade your development environment To begin the process of upgrading to dbt v2 with the assistant: 1. From the project homepage or sidebar menu, click the **Start dbt v2 upgrade** or **Get started** button. You will be redirected to the Studio IDE. [![Start the v2 upgrade.](/img/docs/dbt-platform/platform-configuring-dbt-platform/choosing-dbt-version/start-upgrade.png?v=2 "Start the v2 upgrade.")](#)Start the v2 upgrade. 2. At the top of the Studio IDE click **Check deprecation warnings**. [![Begin the process of parsing for deprecation warnings.](/img/docs/dbt-platform/platform-configuring-dbt-platform/choosing-dbt-version/check-deprecations.png?v=2 "Begin the process of parsing for deprecation warnings.")](#)Begin the process of parsing for deprecation warnings. 3. dbt parses your project for the deprecations and presents a list of all deprecation warnings along with the option to **Autofix warnings**. Autofixing attempts to correct all syntax errors automatically. See [Fix deprecation warnings](https://docs.getdbt.com/docs/platform/studio-ide/autofix-deprecations.md) for more information. [![Begin the process of parsing for deprecation warnings.](/img/docs/dbt-platform/platform-configuring-dbt-platform/choosing-dbt-version/check-deprecations.png?v=2 "Begin the process of parsing for deprecation warnings.")](#)Begin the process of parsing for deprecation warnings. 4. Once the deprecation warnings have been resolved, click the **Enable dbt v2** button. This upgrades your development environment to dbt v2! [![You're now ready to upgrade to v2 in your development environment!](/img/docs/dbt-platform/platform-configuring-dbt-platform/choosing-dbt-version/autofix-success.png?v=2 "You're now ready to upgrade to v2 in your development environment!")](#)You're now ready to upgrade to v2 in your development environment! Now that you've upgraded your development environment to dbt v2, you're ready to start the process of upgrading your Production, Staging, and General environments. Follow your organization's standard procedures and use the [release tracks](#release-tracks) to upgrade. Enable the dbt v2 readiness panel The dbt v2 readiness panel shows each project's eligibility status and blockers in the dbt platform. It's rolling out in phases — if it's not enabled for your account yet, an [account admin](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#account-admin) can turn it on in **Account settings** → **Account**. Refer to [Enable dbt v2 readiness features](https://docs.getdbt.com/guides/prepare-v2-upgrade.md?step=2) for setup steps. If you have access to dbt Wizard, use the [dbt Wizard's dbt v2 migration workflow](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md#fusion-migration-workflow) to help you fix compatibility errors directly from the Studio IDE using dbt Wizard — no manual log investigation needed! ###### Upgrade considerations Keep in mind the following considerations during the upgrade process: * **Manifest compatibility** — dbt v2 produces a `v12` [manifest](https://docs.getdbt.com/reference/artifacts/manifest-json.md) that's compatible with dbt v1. The only differences are optional dbt v2-specific fields that only dbt v2 writes, which dbt v1 safely ignores. As a result, you can run dbt v2 and dbt v1 side by side. State-dependent features such as `state:modified`, `--defer`, and cross-environment `dbt docs generate` work across mixed dbt v2 and dbt v1 environments, so you can migrate to dbt v2 incrementally without breaking existing dbt v1 jobs. State-aware orchestration is now dbt State [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) works with all engines and environments: dbt v1, the dbt platform, and dbt v2. If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). #### Jobs Each job in dbt can be configured to inherit parameters from the environment it belongs to. [![Settings of a dbt job](/img/docs/dbt-platform/platform-configuring-dbt-platform/choosing-dbt-version/job-settings.png?v=2 "Settings of a dbt job")](#)Settings of a dbt job The example job seen in the screenshot above belongs to the environment "Prod". It inherits the dbt version of its environment as shown by the **Inherited from ENVIRONMENT\_NAME (DBT\_VERSION)** selection. You may also manually override the dbt version of a specific job to be any of the current dbt v1 releases supported by Cloud by selecting another option from the dropdown. #### Supported versions dbt Labs has always encouraged users to upgrade dbt v1 versions whenever a new minor version is released. We released our first major version of dbt - `dbt 1.0` - in December 2021. Alongside this release, we updated our policy on which versions of dbt v1 we will support in the dbt platform. > **Starting with v1.0, all subsequent minor versions are available in dbt. Versions are actively supported, with patches and bug fixes, for 1 year after their initial release. At the end of the 1-year window, we encourage all users to upgrade to a newer version for better ongoing maintenance and support.** We provide different support levels for different versions, which may include new features, bug fixes, or security patches: * **[Active](https://docs.getdbt.com/docs/dbt-versions.md#current-version-support)**: In the first few months after a minor version's initial release, we patch it with bugfix releases. These include fixes for regressions, new bugs, and older bugs / quality-of-life improvements. We implement these changes when we have high confidence that they're narrowly scoped and won't cause unintended side effects. * **[Critical](https://docs.getdbt.com/docs/dbt-versions.md#current-version-support)**: When a newer minor version ships, the previous one transitions to "Critical Support" for the remainder of its one-year window. Patches during this period are limited to critical security and installation fixes. After the one-year window ends, the version reaches end of life. * **[End of Life](https://docs.getdbt.com/docs/dbt-versions.md#end-of-life-versions)**: Minor versions that have reached EOL no longer receive new patch releases. * **Deprecated**: dbt v1 versions that are no longer maintained by dbt Labs, nor supported in the dbt platform. We'll continue to update the following release table so that users know when we plan to stop supporting different versions of dbt v1 in dbt. ##### Latest releases | dbt v1 | Initial release | Support level and end date | | ------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ | ------------------------------------------------------ | | [**v2.0**](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md) | Currently in [beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles.md) | TBD | | [**v1.12**](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.12.md) | Jul 16, 2026 | **Active support — July 15, 2027** | | [**v1.11**](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.11.md) | Dec 19, 2025 | **Critical support — Dec 18, 2026** | | [**v1.10**](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.10.md) | Jun 16, 2025 | Deprecated ⛔️ | | [**v1.9**](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.9.md) | Dec 9, 2024 | Deprecated ⛔️ | | [**v1.8**](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.8.md) | May 9, 2024 | Deprecated ⛔️ | | [**v1.7**](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.7.md) | Nov 2, 2023 | End of Life ⚠️<br />Deprecation date: January 31, 2027 | | [**v1.6**](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.6.md>) | Jul 31, 2023 | End of Life ⚠️<br />Deprecation date: January 31, 2027 | | [**v1.5**](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.5.md>) | Apr 27, 2023 | End of Life ⚠️<br />Deprecation date: January 31, 2027 | | [**v1.4**](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.4.md>) | Jan 25, 2023 | End of Life ⚠️<br />Deprecation date: January 31, 2027 | | [**v1.3**](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.3.md>) | Oct 12, 2022 | End of Life ⚠️<br />Deprecation date: January 31, 2027 | | [**v1.2**](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.2.md>) | Jul 26, 2022 | Deprecated ⛔️ | | [**v1.1**](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.1.md>) | Apr 28, 2022 | Deprecated ⛔️ | | [**v1.0**](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.0.md>) | Dec 3, 2021 | Deprecated ⛔️ | | **v0.X** ⛔️ | (Various dates) | Deprecated ⛔️ | All functionality in v1 since the v1.7 release is available in [dbt release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md), which provide automated upgrades at a cadence appropriate for your team. 1 Release tracks are required for the Developer and Starter plans on dbt. Accounts using older dbt versions will be migrated to the **v1 Latest** release track. For customers of dbt: dbt Labs strongly recommends migrating environments on older and unsupported versions to [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) or a supported version. On January 31, 2027, dbt Labs will deprecate v1 versions v1.3 through v1.7, removing them from availability in dbt platform. Starting with v1.0, dbt will ensure that you're always using the latest compatible patch release of `dbt-core` and plugins, including all the latest fixes. You may also choose to try prereleases of those patch releases before they are generally available. For more on version support and future releases, see [Understanding dbt v1 versions](https://docs.getdbt.com/docs/dbt-versions.md). ##### Need help upgrading? If you want more advice on how to upgrade your dbt projects, check out our [migration guides](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade.md) and our [upgrading Q\&A page](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md#upgrading-legacy-versions-under-10). ##### Testing your changes before upgrading Once you know what code changes you'll need to make, you can start implementing them. We recommend you: * Create a separate dbt project, "Upgrade project", to test your changes before making them live in your main dbt project. * In your "Upgrade project", connect to the same repository you use for your production project. * Set the development environment [settings](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) to run the latest version of dbt v1. * Check out a branch `dbt-version-upgrade`, make the appropriate updates to your project, and verify your dbt project compiles and runs with the new version in the Studio IDE. * If upgrading directly to the latest version results in too many issues, try testing your project iteratively on successive minor versions. There are years of development and a few breaking changes between distant versions of dbt v1 (for example, 1.0 --> 1.10). The likelihood of experiencing problems upgrading between successive minor versions is much lower, which is why upgrading regularly is recommended. * Once you have your project compiling and running on the latest version of dbt in the development environment for your `dbt-version-upgrade` branch, try replicating one of your production jobs to run off your branch's code. * You can do this by creating a new deployment environment for testing, setting the custom branch to 'ON' and referencing your `dbt-version-upgrade` branch. You'll also need to set the dbt version in this environment to the latest dbt v1 version. [![Setting your testing environment](/img/docs/dbt-platform/platform-configuring-dbt-platform/platform-upgrading-dbt-versions/upgrade-environment.png?v=2 "Setting your testing environment")](#)Setting your testing environment * Then add a job to the new testing environment that replicates one of the production jobs your team relies on. * If that job runs smoothly, you should be all set to merge your branch into main. * Then change your development and deployment environments in your main dbt project to run off the newest version of dbt v1. --- ### Upgrading to dbt utils v1.0 For the first time, [dbt utils](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/) is crossing the major version boundary. From [last month’s blog post](https://www.getdbt.com/blog/announcing-dbt-v1.3-and-utils/): > It’s time to formalize what was already unofficial policy: you can rely on dbt utils in the same way as you do dbt v1, with stable interfaces and consistent and intuitive naming. Just like the switch to dbt 1.0 last year, there are some breaking changes as we standardized and prepared for the future. Most changes can be handled with find-and-replace. If you need help, post on the [Community Forum](https://discourse.getdbt.com) or in [#package-ecosystem](https://getdbt.slack.com/archives/CU4MRJ7QB) channel on Slack. #### New features * `get_single_value()` — An easy way to pull a single value from a SQL query, instead of having to access the `[0][0]`th element of a `run_query` result. * `safe_divide()` — Returns null when the denominator is 0, instead of throwing a divide-by-zero error. * New `not_empty_string` test — An easier wrapper than using `expression_is_true` to check the length of a column. #### Enhancements * Many tests are more meaningful when you run them against subgroups of a table. For example, you may need to validate that recent data exists for every turnstile instead of a single data source being sufficient. Add the new `group_by_columns` argument to your tests to do so. Review [this article](https://www.emilyriederer.com/post/grouping-data-quality/) by the test's author for more information. * With the addition of an on-by-default `quote_identifiers` argument in the `star()` macro, you can now disable quoting if necessary. * The `recency` test now has an optional `ignore_time_component` argument which can be used when testing against a date column. This prevents the time of day the test runs from causing false negatives/positives. #### Fixes * `union()` now includes/excludes columns case-insensitively * `slugify()` prefixes an underscore when the first char is a digit * The `expression_is_true` test doesn’t output `*` unless storing failures, a cost improvement for BigQuery. #### Breaking Changes ##### Changes to `surrogate_key()`: * `surrogate_key()` has been replaced by `generate_surrogate_key()`. The original treated null values and blank strings the same, which could lead to duplicate keys being created. `generate_surrogate_key()` does not have this flaw. Compare the [surrogate keys calculated for these columns](https://docs.google.com/spreadsheets/d/1qWfdbieUOSgkzdY0kmJ9iCgdqyWccA0R-6EW0EgaMQc/edit#gid=0): ![A table comparing the behavior of surrogate\_key and generate\_surrogate\_key](/assets/images/surrogate_key_behaviour-2248a1a7c8bfa9df30140fadc6021a99.png) Changing the calculation method for surrogate keys, even for the better, could have significant consequences in downstream uses (such as snapshots and incremental models which use this column as their `unique_key`). As a result, it's possible to opt into the legacy behavior by setting the following variable in your dbt project: ```yaml #dbt_project.yml vars: surrogate_key_treat_nulls_as_empty_strings: true #turn on legacy behavior ``` By creating a new macro instead of updating the behavior of the old one, we are requiring all projects who use this macro to make an explicit decision about which approach is better for their context. **Our recommendation is that existing users should opt into the legacy behavior** unless you are confident that either: * your surrogate keys never contained nulls, or * your surrogate keys are not used for incremental models, snapshots or other stateful artifacts and so can be regenerated with new values without issue. Warning to package maintainers You can not assume one behavior or the other, as each project can customize its behavior. ##### Functionality now native to dbt v1: * The `expression_is_true` test no longer has a dedicated `condition` argument. Instead, use `where` which is [now available natively to all tests](https://docs.getdbt.com/reference/resource-configs/where.md): ```yaml models: - name: old_syntax tests: - dbt_utils.expression_is_true: expression: "col_a + col_b = total" #replace this... condition: "created_at > '2018-12-31'" - name: new_syntax tests: - dbt_utils.expression_is_true: expression: "col_a + col_b = total" # ...with this... where: "created_at > '2018-12-31'" ``` **Note** — This may cause some tests to get the same autogenerated names. To resolve this, you can [define a custom name for a test](https://docs.getdbt.com/reference/resource-properties/data-tests.md#define-a-custom-name-for-one-test). * The deprecated `unique_where` and `not_null_where` tests have been removed, because [where is now available natively to all tests](https://docs.getdbt.com/reference/resource-configs/where.md). To migrate, find and replace `dbt_utils.unique_where` with `unique` and `dbt_utils.not_null_where` with `not_null`. * `dbt_utils.current_timestamp()` is replaced by `dbt.current_timestamp()`. * Note that Postgres and Snowflake’s implementation of `dbt.current_timestamp()` differs from the old `dbt_utils` one ([full details here](https://github.com/dbt-labs/dbt-utils/pull/597#issuecomment-1231074577)). If you use Postgres or Snowflake and need identical backwards-compatible behavior, use `dbt.current_timestamp_backcompat()`. This discrepancy will hopefully be reconciled in a future version of dbt. * All other cross-db macros have moved to the dbt namespace, with no changes necessary other than replacing `dbt_utils.` with `dbt.`. Review the [cross database macros documentation](https://docs.getdbt.com/reference/dbt-jinja-functions/cross-database-macros.md) for the full list. * In your code editor, you can do a global find and replace with regex: `\{\{\s*dbt_utils\.(any_value|bool_or|cast_bool_to_text|concat|dateadd|datediff|date_trunc|escape_single_quotes|except|hash|intersect|last_day|length|listagg|position|replace|right|safe_cast|split_part|string_literal|type_bigint|type_float|type_int|type_numeric|type_string|type_timestamp|type_bigint|type_float|type_int|type_numeric|type_string|type_timestamp|except|intersect|concat|hash|length|position|replace|right|split_part|escape_single_quotes|string_literal|any_value|bool_or|listagg|cast_bool_to_text|safe_cast|dateadd|datediff|date_trunc|last_day)` → `{{ dbt.$1` ##### Removal of `insert_by_period` materialization * The `insert_by_period` materialization has been moved to the [experimental-features repo](https://github.com/dbt-labs/dbt-labs-experimental-features/tree/main/insert_by_period). To continue to use it, add the below to your packages.yml file: ```yaml packages: - git: https://github.com/dbt-labs/dbt-labs-experimental-features subdirectory: insert_by_period revision: XXXX #optional but highly recommended. Provide a full git sha hash, e.g. 1c0bfacc49551b2e67d8579cf8ed459d68546e00. If not provided, uses the current HEAD. ``` ##### Removal of deprecated legacy behavior: * `safe_add()` only works with a list of arguments; use `{{ dbt_utils.safe_add(['column_1', 'column_2']) }}` instead of varargs `{{ dbt_utils.safe_add('column_1', 'column_2') }}`. * Several long-promised deprecations to `deduplicate()` have been applied: * The `group_by` argument is replaced by `partition_by`. * `relation_alias` is removed. If you need an alias, you can pass it directly to the `relation` argument. * `order_by` is now mandatory. Pass a static value like `1` if you don’t care how they are deduplicated. * The deprecated `table` argument has been removed from `unpivot()`. Use `relation` instead. #### Resolving error messages After upgrading, these are common error messages you may encounter, along with their resolutions. `dict object has no attribute MACRO_NAME` **Cause**: No macro called `MACRO_NAME` exists. This is most likely because the macro has moved to the `dbt` namespace (see above). It could also be because you haven't run dbt deps or have misspelled a macro's name. **Resolution**: For [cross-database macros](https://docs.getdbt.com/reference/dbt-jinja-functions/cross-database-macros.md), change `dbt_utils.MACRO_NAME()` to `dbt.MACRO_NAME()`. `macro 'dbt_macro__generate_surrogate_key' takes not more than 1 argument(s)` **Cause**: `generate_surrogate_key()` requires a single argument containing a list of columns, not a set of varargs. **Resolution**: Change to `dbt_utils.generate_surrogate_key(['column_1', 'column_2'])` - note the square brackets. `The dbt_utils.surrogate_key has been replaced by dbt_utils.generate_surrogate_key` **Cause**: `surrogate_key()` has been replaced. **Resolution**: 1. Decide whether you need to enable backwards compatibility [as detailed above](#changes-to-surrogate_key). 2. Find and replace `dbt_utils.surrogate_key` with `dbt_utils.generate_surrogate_key`. `macro dbt_macro__test_expression_is_true takes no keyword argument condition` **Cause**: `condition` has been removed from the `expression_is_true` test, now that `where` is available on all tests automatically. **Resolution**: Replace `condition` with `where`. `No materialization insert_by_period was found for adapter` **Cause**: `insert_by_period` has moved to the experimental features repo (see above). **Resolution**: Install the package as [described above](#removal-of-insert_by_period-materialization). `dbt found two tests with the name "XXX".` **Cause**: Changing from `condition` to `where` in the `expression_is_true` test, as configs are not part of a test's unique name. **Resolution**: Define a [custom name for your test](https://docs.getdbt.com/reference/resource-properties/tests#define-a-custom-name-for-one-test). --- ### Upgrading to v1.0 Available in v1 ##### Resources * [Discourse](https://discourse.getdbt.com/t/3180) * [Changelog](https://github.com/dbt-labs/dbt/blob/1.0.latest/CHANGELOG.md) * [dbt v1 CLI Installation guide](https://docs.getdbt.com/docs/local/install-dbt.md) * [Cloud upgrade guide](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) #### What to know before upgrading dbt v1 major version 1.0 includes a number of breaking changes! Wherever possible, we have offered backwards compatibility for old behavior, and (where necessary) made migration simple. ##### Renamed fields in `dbt_project.yml` **These affect everyone:** * [model-paths](https://docs.getdbt.com/reference/project-configs/model-paths.md) have replaced `source-paths` in `dbt-project.yml`. * [seed-paths](https://docs.getdbt.com/reference/project-configs/seed-paths.md) have replaced `data-paths` in `dbt-project.yml` with a default value of `seeds`. * The [packages-install-path](https://docs.getdbt.com/reference/project-configs/packages-install-path.md) was updated from `modules-path`. Additionally the default value is now `dbt_packages` instead of `dbt_modules`. You may need to update this value in [`clean-targets`](https://docs.getdbt.com/reference/project-configs/clean-targets.md). * Default for `quote_columns` is now `True` for all adapters other than Snowflake. **These probably don't:** * The default value of [test-paths](https://docs.getdbt.com/reference/project-configs/test-paths.md) has been updated to be the plural `tests`. * The default value of [analysis-paths](https://docs.getdbt.com/reference/project-configs/analysis-paths.md) has been updated to be the plural `analyses`. ##### Tests The two **test types** are now "singular" and "generic" (instead of "data" and "schema", respectively). The `test_type:` selection method accepts `test_type:singular` and `test_type:generic`. (It will also accept `test_type:schema` and `test_type:data` for backwards compatibility.) **Not backwards compatible:** The `--data` and `--schema` flags to dbt test are no longer supported, and tests no longer have the tags `'data'` and `'schema'` automatically applied. Updated docs: [data tests](https://docs.getdbt.com/docs/build/data-tests.md), [test selection](https://docs.getdbt.com/reference/node-selection/test-selection-examples.md), [selection methods](https://docs.getdbt.com/reference/node-selection/methods.md). The `greedy` flag/property has been renamed to **`indirect_selection`**, which is now eager by default. **Note:** This reverts test selection to its pre-v0.20 behavior by default. `dbt test -s my_model` *will* select multi-parent tests, such as `relationships`, that depend on unselected resources. To achieve the behavior change in v0.20 + v0.21, set `--indirect-selection=cautious` on the CLI or `indirect_selection: cautious` in YAML selectors. Updated docs: [test selection examples](https://docs.getdbt.com/reference/node-selection/test-selection-examples.md), [yaml selectors](https://docs.getdbt.com/reference/node-selection/yaml-selectors.md). ##### Global macros Global project macros have been reorganized, and some old unused macros have been removed: `column_list`, `column_list_for_create_table`, `incremental_upsert`. This is unlikely to affect your project. ##### Installation * [Installation docs](https://docs.getdbt.com/docs/supported-data-platforms.md) reflects adapter-specific installations * `python -m pip install dbt` is no longer supported, and will raise an explicit error. Install the specific adapter plugin you need as `python -m pip install dbt-<adapter>`. * `brew install dbt` is no longer supported. Install the specific adapter plugin you need (among Postgres, Redshift, Snowflake, or BigQuery) as `brew install dbt-<adapter>`. * Removed official support for python 3.6, which is reaching end of life on December 23, 2021 ##### For users of adapter plugins * **BigQuery:** Support for ingestion-time-partitioned tables has been officially deprecated in favor of modern approaches. Use `partition_by` and incremental modeling strategies instead. For more information, refer to [Incremental models](https://docs.getdbt.com/docs/build/incremental-models.md). ##### For maintainers of plugins + other integrations We've introduced a new [**structured event interface**](https://docs.getdbt.com/reference/events-logging.md), and we've transitioned all dbt logging to use this new system. **This includes a breaking change for adapter plugins**, requiring a very simple migration. For more details, see the [`events` module README](https://github.com/dbt-labs/dbt/blob/HEAD/core/dbt/events/README.md#adapter-maintainers). If you maintain a different kind of plugin that *needs* legacy logging, for the time being, you can re-enable it with an env var (`DBT_ENABLE_LEGACY_LOGGER=True`); be advised that we will remove this capability in a future version of dbt. The [**dbt RPC Server**](https://docs.getdbt.com/reference/commands/rpc.md) has been split out from `dbt-core` and is now packaged separately. Its functionality will be fully deprecated by the end of 2022, in favor of a new dbt Server. Instead of `dbt rpc`, use `dbt-rpc serve`. **Artifacts:** New schemas (manifest v4, run results v4, sources v3). Notable changes: add `metrics` nodes; schema test + data test nodes are renamed to generic test + singular test nodes; freshness threshold default values look slightly different. ##### Deprecations from long ago Several under-the-hood changes from past minor versions, tagged with deprecation warnings, have now been fully deprecated. * The `packages` argument of [dispatch](https://docs.getdbt.com/reference/dbt-jinja-functions/dispatch.md) has been deprecated and will raise an exception when used. * The "adapter\_macro" macro has been deprecated. Instead, use the [dispatch](https://docs.getdbt.com/reference/dbt-jinja-functions/dispatch.md) method to find a macro and call the result. * The `release` arg has been removed from the `execute_macro` method. #### New features and changed documentation * Add [metrics](https://docs.getdbt.com/docs/build/build-metrics-intro.md), a new node type * [Generic tests](https://docs.getdbt.com/best-practices/writing-custom-generic-tests.md) can be defined in `tests/generic` (new), in addition to `macros/` (as before) * [Parsing](https://docs.getdbt.com/reference/parsing.md): partial parsing and static parsing have been turned on by default. * [Global configs](https://docs.getdbt.com/reference/global-configs/about-global-configs.md) have been standardized. Related updates to [global CLI flags](https://docs.getdbt.com/reference/global-configs/about-global-configs.md) and [`profiles.yml`](https://docs.getdbt.com/docs/local/profiles.yml.md). * [The `init` command](https://docs.getdbt.com/reference/commands/init.md) has a whole new look and feel. It's no longer just for first-time users. * Add `result:<status>` subselectors for smarter reruns when dbt models have errors and tests fail. See examples: [Pro-tips for Workflows](https://docs.getdbt.com/best-practices/best-practice-workflows.md#pro-tips-for-workflows) * Secret-prefixed [env vars](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md) are now allowed only in `profiles.yml` + `packages.yml` --- ### Upgrading to v1.1 Available in v1 ##### Resources * [Changelog](https://github.com/dbt-labs/dbt/blob/1.1.latest/CHANGELOG.md) * [dbt v1 CLI Installation guide](https://docs.getdbt.com/docs/local/install-dbt.md) * [Cloud upgrade guide](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) #### What to know before upgrading There are no breaking changes for code in dbt projects and packages. We are committed to providing backwards compatibility for all versions 1.x. If you encounter an error upon upgrading, please let us know by [opening an issue](https://github.com/dbt-labs/dbt/issues/new). ##### For maintainers of adapter plugins We have reworked the testing suite for adapter plugin functionality. For details on the new testing suite, refer to the "Test your adapter" step in the [Build, test, document, and promote adapters](https://docs.getdbt.com/guides/adapter-creation.md) guide. The abstract methods `get_response` and `execute` now only return `connection.AdapterReponse` in type hints. Previously, they could return a string. We encourage you to update your methods to return an object of class `AdapterResponse`, or implement a subclass specific to your adapter. This also gives you the opportunity to add fields specific to your adapter's query execution, such as `rows_affected` or `bytes_processed`. ##### For consumers of dbt artifacts (metadata) The manifest schema version will be updated to v5. The only change is to the default value of `config` for parsed nodes. For users of [state-based functionality](https://docs.getdbt.com/reference/node-selection/syntax.md#about-node-selection), such as the `state:modified` selector, recall that: > The `--state` artifacts must be of schema versions that are compatible with the currently running dbt version. If you have two jobs, whereby one job compares or defers to artifacts produced by the other, you'll need to upgrade both at the same time. If there's a mismatch, dbt will alert you with this error message: ```text Expected a schema version of "https://schemas.getdbt.com/dbt/manifest/v5.json" in <state-path>/manifest.json, but found "https://schemas.getdbt.com/dbt/manifest/v4.json". Are you running with a different version of dbt? ``` #### New and changed documentation [**Incremental models**](https://docs.getdbt.com/docs/build/incremental-models.md) can now accept a list of multiple columns as their `unique_key`, for models that need a combination of columns to uniquely identify each row. This is supported by the most common data warehouses, for incremental strategies that make use of the `unique_key` config (`merge` and `delete+insert`). [**Generic tests**](https://docs.getdbt.com/reference/resource-properties/data-tests.md) can define custom names. This is useful to "prettify" the synthetic name that dbt applies automatically. It's needed to disambiguate the case when the same generic test is defined multiple times with different configurations. [**Sources**](https://docs.getdbt.com/reference/source-properties.md) can define configuration inline with other `.yml` properties, just like other resource types. The only supported config is `enabled`; you can use this to dynamically enable/disable sources based on environment or package variables. ##### Advanced and experimental functionality **Fresh Rebuilds.** There's a new *experimental* selection method in town: [`source_status:fresher`](https://docs.getdbt.com/reference/node-selection/methods.md#source_status). Much like the `state:` and `result` methods, the goal is to use dbt metadata to run your DAG more efficiently. If dbt has access to previous and current results of `dbt source freshness` (the `sources.json` artifact), dbt can compare them to determine which sources have loaded new data, and select only resources downstream of "fresher" sources. Read more in [Understanding State](https://docs.getdbt.com/reference/node-selection/syntax.md#about-node-selection) and [CI/CD in dbt](https://docs.getdbt.com/docs/deploy/continuous-integration.md). [**dbt-Jinja functions**](https://docs.getdbt.com/reference/dbt-jinja-functions-context-variables.md) have a new landing page, and two new members: * [`print`](https://docs.getdbt.com/reference/dbt-jinja-functions/print.md) exposes the Python `print()` function. It can be used as an alternative to `log()`, and together with the `QUIET` config, for advanced macro-driven workflows. * [`selected_resources`](https://docs.getdbt.com/reference/dbt-jinja-functions/selected_resources.md) exposes, at runtime, the list of DAG nodes selected by the current task. [**Global configs**](https://docs.getdbt.com/reference/global-configs/about-global-configs.md) include some new additions: * `QUIET` and `NO_PRINT`, to control which log messages dbt prints to terminal output. For use in advanced macro-driven workflows, such as [codegen](https://hub.getdbt.com/dbt-labs/codegen/latest/). * `CACHE_SELECTED_ONLY` is an *experimental* config that can significantly speed up dbt's start-of-run preparations, in cases where you're running only a few models from a large project that manages many schemas. ##### For users of specific adapters **dbt-bigquery** added Support for finer-grained configuration of query timeout and retry when defining your [connection profile](https://docs.getdbt.com/docs/local/connect-data-platform/bigquery-setup.md). **dbt-spark** added support for a [`session` connection method](https://docs.getdbt.com/docs/local/connect-data-platform/spark-setup.md#session), for use with a pySpark session, to support rapid iteration when developing advanced or experimental functionality. This connection method is not recommended for new users, and it is not supported in dbt. ##### Dependencies [Python compatibility](https://docs.getdbt.com/faqs/Core/install-python-compatibility.md): dbt v1 officially supports Python 3.10 --- ### Upgrading to v1.10 Available in v1 #### Resources * dbt [v1.10 changelog](https://github.com/dbt-labs/dbt/blob/1.10.latest/CHANGELOG.md) * [dbt v1 CLI Installation guide](https://docs.getdbt.com/docs/local/install-dbt.md) * [dbt platform upgrade guide](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md#release-tracks) #### What to know before upgrading dbt Labs is committed to providing backward compatibility for all versions 1.x. Any behavior changes will be accompanied by a [behavior change flag](https://docs.getdbt.com/reference/global-configs/behavior-changes.md#behavior-change-flags) to provide a migration window for existing projects. If you encounter an error upon upgrading, please let us know by [opening an issue](https://github.com/dbt-labs/dbt/issues/new). Starting in 2024, dbt provides the functionality from new versions of dbt v1 via [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) with automatic upgrades. If you have selected the **v1 Latest** release track in dbt, you already have access to all the features, fixes, and other functionality that is included in dbt v1.10! If you have selected the **v1 Compatible** release track, you will have access in the next monthly **v1 Compatible** release after the dbt v1.10 final release. #### New and changed features and functionality New features and functionality available in dbt v1.10 ##### The `--sample` flag Large data sets can slow down dbt build times, making it harder for developers to test new code efficiently. The [`--sample` flag](https://docs.getdbt.com/docs/build/sample-flag.md), available for the `run` and `build` commands, helps reduce build times and warehouse costs by running dbt in sample mode. It generates filtered refs and sources using time-based sampling, allowing developers to validate outputs without building entire models. ##### Move standalone anchors under `anchors:` key As part of the ongoing process of making the dbt authoring language more precise, dbt v1.10 raises a warning when it sees an unexpected top-level key in a properties YAML file. A common use case behind these unexpected keys is standalone anchor definitions at the top level of a properties YAML file. You can use the new top-level `anchors:` key as a container for these reusable configuration blocks. For example, rather than using this configuration: models/\_models.yml ```yml id_column: &id_column_alias name: id description: This is a unique identifier. data_type: int data_tests: - not_null - unique models: - name: my_first_model columns: - *id_column_alias - name: unrelated_column_a description: This column is not repeated in other models. - name: my_second_model columns: - *id_column_alias ``` Move the anchor under the `anchors:` key instead: models/\_models.yml ```yml anchors: - &id_column_alias name: id description: This is a unique identifier. data_type: int data_tests: - not_null - unique models: - name: my_first_model columns: - *id_column_alias - name: unrelated_column_a description: This column is not repeated in other models - name: my_second_model columns: - *id_column_alias ``` This move is only necessary for fragments defined outside of the main YAML structure. For more information about this new key, see [anchors](https://docs.getdbt.com/reference/resource-properties/anchors.md). ##### Parsing `catalogs.yml` dbt v1 can now parse the `catalogs.yml` file. This is an important milestone in the journey to supporting external catalogs for Iceberg tables, as it enables write integrations. You'll be able to provide a config specifying a catalog integration for your producer model: For example: ```yml catalogs: - name: catalog_dave # materializing the data to an external location, and metadata to that data catalog write_integrations: - name: databricks_glue_write_integration external_volume: databricks_external_volume_prod table_format: iceberg catalog_type: unity ``` The implementation for the model would look like this: models/schemas.yml ```yaml models: - name: my_second_public_model config: catalog_name: catalog_dave ``` Check out our [docs on external catalog support](https://docs.getdbt.com/docs/build/iceberg/about-catalogs.md) today! We'll have more information about this in the coming weeks, but this is an exciting step in journey to cross-platform support. ##### Integrating dbt v1 artifacts with dbt projects With [hybrid projects](https://docs.getdbt.com/docs/deploy/hybrid-projects.md), dbt v1 users working in the command line interface (CLI) can execute runs that seamlessly upload [artifacts](https://docs.getdbt.com/reference/artifacts/dbt-artifacts.md) into dbt. This enhances hybrid dbt v1/dbt deployments by: * Fostering collaboration between dbt + dbt v1 users by enabling them to visualize and perform [cross-project references](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#how-to-write-cross-project-ref) to models defined in dbt v1 projects. This feature unifies dbt + dbt v1 workflows for a more connected dbt experience. * Giving dbt and dbt v1 users insights into their models and assets in [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md). To view Catalog, you must have have a [developer or read-only license](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md). * (Coming soon) Enabling users working in the [Canvas](https://docs.getdbt.com/docs/platform/canvas.md) to build off of models already created by a central data team in dbt v1 rather than having to start from scratch. Hybrid projects are available as a private beta to [dbt Enterprise accounts](https://www.getdbt.com/pricing). Contact your account representative to register your interest in the beta. ##### Managing changes to legacy behaviors dbt v1.10 introduces new flags for [managing changes to legacy behaviors](https://docs.getdbt.com/reference/global-configs/behavior-changes.md). You may opt into recently introduced changes (disabled by default), or opt out of mature changes (enabled by default), by setting `true` / `false` values, respectively, for `flags` in `dbt_project.yml`. You can read more about each of these behavior changes in the following links: * (Introduced, disabled by default) [`validate_macro_args`](https://docs.getdbt.com/reference/global-configs/behavior-flags/validate_macro_args.md). If set to `true`, dbt raises a warning if the argument names or types in your macro YAML don't match the macro's Jinja definition or use unsupported type values. Note that projects using `--warn-error` may experience build failures at parse time if macro argument declarations in YAML are out of sync with the macro's Jinja signature. * (Introduced, disabled by default) [`require_all_warnings_handled_by_warn_error`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_all_warnings_handled_by_warn_error.md). If this flag is set to `true`, all warnings raised during a run will be routed through the `--warn-error` / `--warn-error-options` handler. This ensures consistent behavior when promoting warnings to errors or silencing them. When the flag is `false` (which is the current default), only some warnings are processed by the handler — others may bypass it. Turning it on for projects that use `--warn-error` (or `--warn-error-options='{"error":"all"}'`) may cause build failures on warnings that were previously ignored to fail so we recommend enabling it gradually, one a project at a time. ##### Deprecation warnings Starting in `v1.10`, you will receive deprecation warnings for dbt code that will become invalid in the future, including: * Custom inputs (for example, unrecognized resource properties, configurations, and top-level keys) * Duplicate YAML keys in the same file * Unexpected Jinja blocks (for example, `{% endmacro %}` tags without a corresponding `{% macro %}` tag) * Some `properties` are moving to `configs` * And more dbt will start raising these warnings in version `1.10`, but making these changes will not be a prerequisite for using it. We at dbt Labs understand that it will take existing users time to migrate their projects, and it is not our goal to disrupt anyone with this update. The goal is to enable you to work with more safety, feedback, and confidence going forward. What does this mean for you? 1. If your project (or dbt package) encounters a new deprecation warning in `v1.10`, plan to update your invalid code soon. Although it’s just a warning for now, in a future version, dbt will enforce stricter validation of the inputs in your project. Check out the [`dbt-autofix` tool](https://github.com/dbt-labs/dbt-autofix) to autofix many of these! 2. In the future, the [`meta` config](https://docs.getdbt.com/reference/resource-configs/meta.md) will be the only place to put custom user-defined attributes. Everything else will be strongly typed and strictly validated. If you have an extra attribute you want to include in your project, or a model config you want to access in a custom materialization, you must nest it under `meta` moving forward. 3. If you are using the [`—-warn-error` flag](https://docs.getdbt.com/reference/global-configs/warnings.md) (or `--warn-error-options '{"error": "all"}'`) to promote all warnings to errors, this will include new deprecation warnings coming to dbt v1. If you don’t want these to be promoted to errors, the `--warn-error-options` flag gives you more granular control over exactly which types of warnings are treated as errors. You can set `"warn": ["Deprecations"]` (new as of `v1.10`) to continue treating the deprecation warnings as warnings. 4. The `--models` / `--model` / `-m` flag was renamed to `--select` / `--s` way back in dbt v0.21 (Oct 2021). Silently skipping this flag means ignoring your command's selection criteria, which could mean building your entire DAG when you only meant to select a small subset. For this reason, the `--models` / `--model` / `-m` flag **will raise a warning** in dbt v1.10, and an error in dbt v2. Please update your job definitions accordingly. ###### Custom inputs Historically, dbt has allowed you to configure inputs largely unconstrained. A common example of this is setting custom YAML properties: ```yml models: - name: my_model description: A model in my project. dbt_is_awesome: true # a custom property ``` dbt detects the unrecognized custom property (`dbt_is_awesome`) and silently continues. Without a set of strictly defined inputs, it becomes challenging to validate your project's configuration. This creates unintended issues such as: * Silently ignoring misspelled properties and configurations (for example, `desciption:` instead of `description:`). * Unintended collisions with user code when dbt introduces a new “reserved” property or configuration. If you have an unrecognized custom property, you will receive a warning, and in a future version, dbt will cease to support custom properties. Moving forward, these should be nested under the [`meta` config](https://docs.getdbt.com/reference/resource-configs/meta.md), which will be the only place to put custom user-defined attributes: ```yml models: - name: my_model description: A model in my project. config: meta: dbt_is_awesome: true ``` ###### Custom keys not nested under meta Previously, when you could define any additional fields directly under `config`, it could lead to collisions between pre-existing user-defined configurations and official configurations of the dbt framework. In the future, the `meta` config will be the sole location for custom user-defined attributes. Everything else will be strongly typed and strictly validated. If you have an extra attribute you want to include in your project, or a model config you want to access in a custom materialization, you must nest it under `meta` moving forward: ```yaml models: - name: my_model config: meta: custom_config_key: value columns: - name: my_column config: meta: some_key: some_value ``` ###### Duplicate keys in the same yaml file If two identical keys exist in the same properties YAML file, you will get a warning, and in a future version, dbt will stop supporting duplicate keys. Previously, if identical keys existed in the same properties YAML file, dbt silently overwrite, using the last configuration listed in the file. profiles.yml ```yml my_profile: target: my_target outputs: ... my_profile: # dbt would use only this profile key target: my_other_target outputs: ... ``` Moving forward, you should delete unused keys or move them to a separate properties YAML file. ###### Unexpected Jinja blocks If you have an orphaned Jinja block, you will receive a warning, and in a future version, dbt will stop supporting unexpected Jinja blocks. Previously, these orphaned Jinja blocks were silently ignored. macros/my\_macro.sql ```sql {% endmacro %} # orphaned endmacro jinja block {% macro hello() %} hello! {% endmacro %} ``` Moving forward, you should delete these orphaned Jinja blocks. ###### Properties moving to configs Some historical properties are moving entirely to configs. This will include: `freshness`, `meta`, `tags`, `docs`, `group`, and `access` If you previously set one of the impacted properties, such as `freshness`: ```yaml sources: - name: ecom schema: raw description: E-commerce data for the Jaffle Shop freshness: warn_after: count: 24 period: hour ``` You should now set it under `config`: ```yaml sources: - name: ecom schema: raw description: E-commerce data for the Jaffle Shop config: freshness: warn_after: count: 24 period: hour ``` ###### Custom output path for source freshness The ability to override the default path for `sources.json` via the `--output` or `-o` flags has been deprecated. You can still set the path for all artifacts in the step with `--target-path`, but will receive a warning if trying to set the path for just source freshness. ###### Warn error options The `warn_error_option` options for `include` and `exclude` have been deprecated and replaced with `error` and `warn`, respectively. ```yaml ... flags: warn_error_options: error: # Previously called "include" warn: # Previously called "exclude" silence: # To silence or ignore warnings - NoNodesForSelectionCriteria ``` #### Adapter-specific features and functionalities Snowflake column size change [Snowflake plans to increase](https://docs.snowflake.com/en/release-notes/bcr-bundles/un-bundled/bcr-2118) the default column size for string and binary data types in September 2026. `dbt-snowflake` versions below v1.10.6 may fail to build certain incremental models when this change is deployed.  Assess impact and required actions If you're using a `dbt-snowflake` version below v1.10.6 or have not yet migrated to a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) in the dbt platform, your adapter version is incompatible with this change and may fail to build incremental models that meet *both* of the following conditions: * Contain string columns with collation defined * Use the `on_schema_change='sync_all_columns'` config To check whether this change affects your project, run the following [list](https://docs.getdbt.com/reference/commands/list.md) command: ```bash dbt ls -s config.materialized:incremental,config.on_schema_change:sync_all_columns --resource-type model ``` * If the command returns `No nodes selected!`, no action is required. * If the command returns one or more models (for example, `Found 1000 models, 644 macros`), you may be impacted if those models have string columns that don't specify a width. In that case, upgrade to a version that includes the fix: * **dbt v1**: `dbt-snowflake` v1.10.6 or later. For upgrade instructions, refer to [Upgrade adapters](https://docs.getdbt.com/docs/local/install-dbt.md) in the dbt v1 installation instructions. * **dbt platform**: Any release track (**v1 Latest**, **v1 Compatible**, **v1 Extended**, or **v1 Fallback**). * **dbt v2**: v2.0.0. This ensures your incremental models can safely handle schema changes while maintaining required collation settings. ##### Snowflake * You can use the `platform_detection_timeout_seconds` parameter to control how long the Snowflake connector waits when detecting the cloud platform where the connection is being made. For more information, see [Snowflake setup](https://docs.getdbt.com/docs/local/connect-data-platform/snowflake-setup.md#platform_detection_timeout_seconds). ##### BigQuery * `dbt-bigquery` cancels BigQuery jobs that exceed their configured timeout by sending a cancellation request. If the request succeeds, dbt stops the job. If the request fails, the BigQuery job may keep running in the background until it finishes or you cancel it manually. For more information, see [Timeout and retries](https://docs.getdbt.com/docs/local/connect-data-platform/bigquery-setup.md#timeouts-and-retries). #### Quick hits * Use the [`--use-fast-test-edges`](https://docs.getdbt.com/reference/global-configs/fast-test-edges.md) flag with `dbt build` to reduce the number of test edges dbt adds to the execution graph. In large projects, this can significantly reduce run times and memory usage. * Provide the [`loaded_at_query`](https://docs.getdbt.com/reference/resource-properties/freshness.md#loaded_at_query) property for source freshness to specify custom SQL to generate the `maxLoadedAt` time stamp on the source (versus the [built-in query](https://github.com/dbt-labs/dbt-adapters/blob/6c41bedf27063eda64375845db6ce5f7535ef6aa/dbt/include/global_project/macros/adapters/freshness.sql#L4-L16), which uses the `loaded_at_field`). You cannot define `loaded_at_query` if the `loaded_at_field` config is also provided. * Provide validation for macro arguments using the [`validate_macro_args`](https://docs.getdbt.com/reference/global-configs/behavior-flags/validate_macro_args.md) flag, which is disabled by default. When enabled, this flag checks that documented macro argument names match those in the macro definition and validates their types against a supported format. Previously, dbt did not enforce standard argument types, treating the type field as documentation-only. If no arguments are documented, dbt infers them from the macro and includes them in the manifest.json file. Learn more about [supported types](https://docs.getdbt.com/reference/resource-properties/arguments.md#supported-types). * You can use the [`config.meta_get()`](https://docs.getdbt.com/reference/dbt-jinja-functions/config.md#configmeta_get) and [`config.meta_require()`](https://docs.getdbt.com/reference/dbt-jinja-functions/config.md#configmeta_require) functions to access custom configurations stored under `meta`. --- ### Upgrading to v1.11 Available in v1 #### Resources * [dbt v1.11 changelog](https://github.com/dbt-labs/dbt/blob/1.11.latest/CHANGELOG.md) * [dbt v1 CLI Installation guide](https://docs.getdbt.com/docs/local/install-dbt.md) * [dbt platform upgrade guide](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md#release-tracks) #### What to know before upgrading dbt Labs is committed to providing backward compatibility for all versions 1.x. Any behavior changes will be accompanied by a [behavior change flag](https://docs.getdbt.com/reference/global-configs/behavior-changes.md#behavior-change-flags) to provide a migration window for existing projects. If you encounter an error upon upgrading, please let us know by [opening an issue](https://github.com/dbt-labs/dbt/issues/new). Starting in 2024, dbt provides the functionality from new versions of dbt v1 via [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) with automatic upgrades. If you have selected the **v1 Latest** release track in dbt, you already have access to all the features, fixes, and other functionality included in the latest dbt v1 version! If you have selected the **v1 Compatible** release track, you will have access in the next monthly **v1 Compatible** release after the dbt v1.11 final release. #### New and changed features and functionality New features and functionality available in dbt v1.11 ##### User-defined functions (UDFs) dbt v1.11 introduces support for user-defined functions (UDFs), which enable you to define and register custom functions in your warehouse. Like macros, UDFs promote code reuse, but they are objects in the warehouse so you can reuse the same logic in tools outside dbt. Key features include: * **Define UDFs as first-class dbt resources**: Create UDF files in a `functions/` directory with corresponding YAML configuration. * **Execution**: Create, update, and rename UDFs as part of DAG execution using `dbt build --select "resource_type:function"` * **DAG integration**: When executing `dbt build`, UDFs are built before models that reference them, ensuring proper dependency management. * **New `function()` macro**: Reference UDFs in your models using the `{{ function('function_name') }}` Jinja macro. * **Deferral**: When you run a dbt command with `--defer` and `--state`, `function()` calls resolve to the UDF in the state manifest, so you can run models that depend on UDFs without building those UDFs first. Read more about UDFs, including prerequisites and how to define and use them in the [UDF documentation](https://docs.getdbt.com/docs/build/udfs.md). ##### `DBT_ENGINE_` prefix for environment variables Engine configuration environment variables use the `DBT_ENGINE_` prefix. For example, `DBT_STATE` becomes `DBT_ENGINE_STATE`, `DBT_PROJECT_DIR` becomes `DBT_ENGINE_PROJECT_DIR`, and so on. Refer to [About flags (global configs)](https://docs.getdbt.com/reference/global-configs/about-global-configs.md) for the full mapping. ##### Managing changes to legacy behaviors dbt v1.11 introduces new flags for [managing changes to legacy behaviors](https://docs.getdbt.com/reference/global-configs/behavior-changes.md). You may opt into recently introduced changes (disabled by default), or opt out of mature changes (enabled by default), by setting `true` / `false` values, respectively, for `flags` in `dbt_project.yml`. You can read more about each of these behavior changes in the following links: * (Introduced, disabled by default) [`require_unique_project_resource_names`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_unique_project_resource_names.md). This flag is set to `false` by default. With this setting, if two unversioned resources in the same package share the same name, dbt continues to run and raises a [`DuplicateNameDistinctNodeTypesDeprecation`](https://docs.getdbt.com/reference/deprecations.md#duplicatenamedistinctnodetypesdeprecation) warning. When set to `true`, dbt raises a `DuplicateResourceNameError` error. * (Introduced, disabled by default) [`require_ref_searches_node_package_before_root`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_ref_searches_node_package_before_root.md). This flag is set to `false` by default. With this setting, when dbt resolves a `ref()` in a package model, it searches for the referenced model in the root project *first*, then in the package where the model is defined. When set to `true`, dbt searches the package where the model is defined *before* searching the root project. ##### Deprecation warnings enabled by default Deprecation warnings from JSON schema validation are now enabled by default when validating your YAML configuration files (such as `schema.yml` and `dbt_project.yml`) for projects running using the Snowflake, Databricks, BigQuery, and Redshift adapters. These warnings help you proactively identify and update deprecated configurations (such as misspelled config keys, deprecated properties, or incorrect data types). You'll see the following deprecation warnings by default: * [CustomKeyInConfigDeprecation](https://docs.getdbt.com/reference/deprecations.md#customkeyinconfigdeprecation) * [CustomKeyInObjectDeprecation](https://docs.getdbt.com/reference/deprecations.md#customkeyinobjectdeprecation) * [CustomTopLevelKeyDeprecation](https://docs.getdbt.com/reference/deprecations.md#customtoplevelkeydeprecation) * [MissingPlusPrefixDeprecation](https://docs.getdbt.com/reference/deprecations.md#missingplusprefixdeprecation) * [SourceOverrideDeprecation](https://docs.getdbt.com/reference/deprecations.md#sourceoverridedeprecation) Each deprecation type can be silenced using the [warn-error-options](https://docs.getdbt.com/reference/global-configs/warnings.md#configuration) project configuration. For example, to silence all of the above deprecations within `dbt_project.yml`: dbt\_project.yml ```yml flags: warn_error_options: silence: - CustomTopLevelKeyDeprecation - CustomKeyInConfigDeprecation - CustomKeyInObjectDeprecation - MissingPlusPrefixDeprecation - SourceOverrideDeprecation ``` Alternatively, the `--warn-error-options` flag can be used to silence specific deprecations from the command line: ```sh dbt parse --warn-error-options '{"silence": ["CustomTopLevelKeyDeprecation", "CustomKeyInConfigDeprecation", "CustomKeyInObjectDeprecation", "MissingPlusPrefixDeprecation", "SourceOverrideDeprecation"]}' ``` To silence *all* deprecation warnings within `dbt_project.yml`: dbt\_project.yml ```yml flags: warn_error_options: silence: - Deprecations ``` Similarly, all deprecation warnings can be silenced via the `--warn-error-options` command line flag: ```sh dbt parse --warn-error-options '{"silence": ["Deprecations"]}' ``` #### Adapter-specific features and functionalities ##### Snowflake * The Snowflake adapter supports basic table materialization on Iceberg tables registered in a Glue catalog through a [catalog-linked database](https://docs.snowflake.com/en/user-guide/tables-iceberg-catalog-linked-database#label-catalog-linked-db-create). For more information, see [Glue Data Catalog](https://docs.getdbt.com/docs/build/iceberg/adapters/snowflake-iceberg-support.md#external-catalogs). * The `cluster_by` configuration is supported in dynamic tables. For more information, see [Dynamic table clustering](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#dynamic-table-clustering). * The `immutable_where` configuration is supported in dynamic tables. For more information, see [Snowflake configurations](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#immutable-where). * You can set [`copy_grants: true`](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#copy-grants-dynamic-tables) on a dynamic table to preserve existing object-level privileges when the table is recreated during a `--full-refresh`. When set to `false` (default), all previously granted permissions are dropped on recreation, requiring manual re-grants. * Set the [`refresh_warehouse`](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#refresh-warehouse) parameter to choose which Snowflake warehouse runs a dynamic table's automatic refreshes. This is separate from `snowflake_warehouse`, which is used for DDL execution. For example, you might use a smaller warehouse for refreshes and a larger one for DDL. If `refresh_warehouse` is not set, `snowflake_warehouse` is used for both DDL execution and automatic refreshes. ##### BigQuery * To improve performance, dbt can issue a single batch query when calculating source freshness through metadata, instead of executing one query per source. To enable this feature, set [bigquery\_use\_batch\_source\_freshness](https://docs.getdbt.com/reference/global-configs/bigquery-changes.md#the-bigquery_use_batch_source_freshness-flag) to `true`. ##### Redshift * The new `datasharing` profile credential enables `dbt-redshift` to use Redshift-native metadata commands (`SHOW` commands such as `SHOW TABLES` and `SHOW COLUMNS`) instead of PostgreSQL catalog tables such as `pg_*` and `information_schema`. This supports cross-database and cross-cluster access with [Redshift Datasharing](https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html). For configuration details, refer to [Redshift setup](https://docs.getdbt.com/docs/local/connect-data-platform/redshift-setup.md#datasharing).[Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") * The `drop_without_cascade` profile credential emits `DROP TABLE/VIEW/MATERIALIZED VIEW` statements without `CASCADE`. Redshift resolves the `CASCADE` dependency graph on every `DROP`, which adds overhead on large clusters. If your project has no downstream dependents (for example, it uses only unbound views) you can set `drop_without_cascade: true` to skip that cost. When enabled and a dependent object exists, Redshift raises an error. For configuration details, refer to [Redshift setup](https://docs.getdbt.com/docs/local/connect-data-platform/redshift-setup.md). ##### Spark * New profile configurations have been added to enhance [retry handling for PyHive connections](https://docs.getdbt.com/reference/resource-configs/spark-configs.md#retry-handling-for-pyhive-connections): * `poll_interval`: Controls how frequently the adapter polls the Thrift server to check if an async query has completed. * `query_timeout`: Adds an overall timeout (in seconds) for query execution. If a query exceeds the set duration during polling, it raises a `DbtRuntimeError`. This helps prevent indefinitely hanging queries. * `query_retries`: Handles connection loss during query polling by automatically retrying. #### Quick hits You will find these quick hits in dbt v1.11: * The [`--sqlparse`](https://docs.getdbt.com/reference/global-configs/sqlparse.md) flag sets [`sqlparse`](https://sqlparse.readthedocs.io/en/latest/api.html#security-and-performance-considerations) `MAX_GROUPING_DEPTH` and `MAX_GROUPING_TOKENS` when dbt parses SQL during compilation. * The `dbt ls` command can now write out nested keys. This makes it easier to debug and troubleshoot your project. Example: `dbt ls --output json --output-keys config.materialized` * Manifest metadata now includes `run_started_at`, providing better tracking of when dbt runs were initiated. * When a model is disabled, unit tests for that model are automatically disabled as well. * You can use the new [`config.meta_get()`](https://docs.getdbt.com/reference/dbt-jinja-functions/config.md#configmeta_get) and [`config.meta_require()`](https://docs.getdbt.com/reference/dbt-jinja-functions/config.md#configmeta_require) functions to access custom configurations stored under `meta`. These functions have been backported to dbt v1.10. --- ### Upgrading to v1.12 Available in v1 #### Resources * [dbt v1.12 changelog](https://github.com/dbt-labs/dbt/blob/1.12.latest/CHANGELOG.md) * [dbt v1 CLI Installation guide](https://docs.getdbt.com/docs/local/install-dbt.md) * [dbt platform upgrade guide](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md#release-tracks) #### What to know before upgrading dbt Labs is committed to providing backward compatibility for all versions 1.x. Any behavior changes will be accompanied by a [behavior change flag](https://docs.getdbt.com/reference/global-configs/behavior-changes.md#behavior-change-flags) to provide a migration window for existing projects. If you encounter an error upon upgrading, please let us know by [opening an issue](https://github.com/dbt-labs/dbt/issues/new). dbt provides the functionality from new versions of dbt v1 via [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) with automatic upgrades. If you have selected the **v1 Latest** release track in dbt, you already have access to all the features, fixes, and other functionality included in the latest dbt v1 version! If you have selected the **v1 Compatible** release track, you will have access to the next monthly **v1 Compatible** release after the dbt v1.12 final release. #### New and changed features and functionality ##### Opt-in v2 parser dbt v1.12 introduces the `--use-v2-parser` flag that delegates parsing to the new v2 Rust parser instead of the v1 Python parser. The Rust parser is significantly faster than the Python parser — especially on larger projects, where it can be 5–10× quicker. If you're looking to speed up your development workflow or cut down on job startup times. Using the Rust parser is a natural first step toward v2 compatibility, so you can catch and fix any project issues gradually rather than all at once. This is an opt-in flag that changes no behavior unless explicitly set, making it a low-risk way to test v2 parser compatibility from within dbt v1.12. note The Rust parser is beta. Its output manifest may differ from the Python parser's in edge cases, which can affect downstream behavior. Fall back by removing `--use-v2-parser`, and [report issues](https://github.com/dbt-labs/dbt/issues) to help us close the gap. For more information on how to enable the flag, related behaviors, and parser error types, refer to [Opt-in v2 parser](https://docs.getdbt.com/reference/global-configs/parsing.md#opt-in-v2-parser). ##### New Iceberg `catalogs` spec We reworked the [`catalogs.yml` spec](https://docs.getdbt.com/docs/build/iceberg/catalogs-yml.md) to make it simpler to use, and to bring it up to speed with latest developments in Iceberg support among ecosystem vendors. The new spec can also power [cross-platform dbt Mesh](https://docs.getdbt.com/docs/mesh/cross-platform-mesh.md). ##### Native private packages dbt v1 now supports [native private packages](https://docs.getdbt.com/docs/build/packages.md#native-private-packages) in `packages.yml` and `dependencies.yml`. You can install packages from private GitHub, GitLab, or Azure DevOps repos using the `private` key without configuring a token or full Git URL. dbt uses your system's SSH configuration for authentication. Use the [`provider` key](https://docs.getdbt.com/docs/build/packages.md#using-the-provider-key) to specify your Git provider and tell dbt which SSH URL format to construct. For example: ```yaml packages: - private: your-org/your-internal-repo provider: "github" # Supported values: "github", "gitlab", "ado" ``` ##### Extensions to UDFs * dbt v1.12 adds support for JavaScript user-defined functions (UDFs) on Snowflake and BigQuery. Define a JavaScript UDF by creating a `.js` file in your `functions/` directory and a corresponding YAML file with the function's arguments and return type. For more information, refer to [User-defined functions](https://docs.getdbt.com/docs/build/udfs.md). * You can define multiple argument signatures for the same user-defined function (UDF) using the `overloads` property. This lets you call the same function name with different input types, without creating separate UDFs for each variant. This is supported for SQL UDFs in Snowflake and Postgres, and Python UDFs in Snowflake. Each overload references a separate file using `defined_in`, with optional `arguments` and `returns`. All overloads are grouped into one DAG node, so they're built and selected together. On retry, dbt skips overloads that succeeded and reruns only those that failed. For more information, refer to [Defining overloaded UDFs](https://docs.getdbt.com/docs/build/udfs.md#defining-udfs-in-dbt#defining-overloaded-UDFs) and [`overloads`](https://docs.getdbt.com/reference/resource-properties/overloads.md). * You can specify public third-party PyPI packages for your Python UDF with the optional `packages` config. The warehouse installs these packages when it creates the UDF, which lets your UDF use functionality from external Python libraries. For more information, refer to [Defining UDFs in dbt](https://docs.getdbt.com/docs/build/udfs.md#defining-udfs-in-dbt) and the [packages](https://docs.getdbt.com/reference/resource-configs/packages.md) config reference. * Python UDFs are now supported in Databricks, in addition to Snowflake and BigQuery. For Databricks, Python UDFs require [Unity Catalog](https://docs.databricks.com/aws/en/data-governance/unity-catalog/), and the `runtime_version` and `entry_point` configs are ignored. For more information, refer to [User-defined functions](https://docs.getdbt.com/docs/build/udfs.md). ##### `latest_version_pointer` for versioned models For versioned models, you can configure dbt to automatically create a pointer view named after a model's base name (for example, `dim_customers`) once the latest version materializes successfully. This lets you query the current version without maintaining a view manually. Enable this feature in your project with the [`latest_version_pointer_enabled_by_default: true`](https://docs.getdbt.com/reference/global-configs/behavior-flags/latest_version_pointer_enabled_by_default.md) flag in `dbt_project.yml`, or per model using the [`latest_version_pointer.enabled`](https://docs.getdbt.com/reference/resource-configs/latest_version_pointer.md) config. You can customize the pointer name per model with `latest_version_pointer.alias`, or globally by overriding the [`generate_latest_version_pointer_alias`](https://docs.getdbt.com/docs/build/custom-aliases.md#generate_latest_version_pointer_alias) macro. For more information, refer to [Model versions](https://docs.getdbt.com/docs/mesh/govern/model-versions.md#pointing-to-the-latest-version). ##### `--sql` flag for `dbt run-operation` You can now use the `--sql` flag with `dbt run-operation` to execute ad hoc database statements directly against your warehouse, without defining a macro. This is useful for one-off operations like dropping or altering a table, applying grants, or running a data fix. The statement runs through dbt's full Jinja compilation pipeline, so you have access to `ref()`, `source()`, `var()`, `target`, and all other context variables. For more information, refer to [About dbt run-operation](https://docs.getdbt.com/reference/commands/run-operation.md). ##### `on_error` model config You can configure whether downstream models run when an upstream model fails using the [`on_error`](https://docs.getdbt.com/reference/resource-configs/on_error.md) config. Set `on_error: continue` on a model to allow its downstream models to still attempt to run even when it fails. By default (`skip_children`), dbt skips all downstream models on failure. Note that [`--fail-fast`](https://docs.getdbt.com/reference/global-configs/failing-fast.md) takes precedence — runs with `--fail-fast` stop at the first failure, even if a model is configured with `on_error: continue`. ##### Apache Ossie semantic layer support * dbt v1.12 supports the [Apache Ossie](https://github.com/apache/ossie) standard for defining semantic models and metrics. You can place Ossie-format `.json` files in an `osi/` directory at the root of your project, and dbt parses them into the manifest alongside any native dbt semantic models. To use a different directory, configure [`osi-paths`](https://docs.getdbt.com/reference/project-configs/osi-paths.md) in `dbt_project.yml`. Ossie versions `0.1.0` and `0.1.1` are supported; any other version raises a parse error. For more information, refer to [Ossie semantic layer documents](https://docs.getdbt.com/docs/build/ossie-semantic-models.md). * dbt writes an `osi_document.json` file to your `target/` directory alongside `semantic_manifest.json` at parse time. This artifact provides an Ossie representation of your project's Semantic Layer. For more information, refer to [Semantic manifest](https://docs.getdbt.com/reference/artifacts/sl-manifest.md#apache-ossie-document). This is separate from the new Semantic Layer YAML spec below, which changes how you define semantic models and metrics in native dbt YAML. ##### New Semantic Layer YAML spec dbt v1.12 adds support for the latest Semantic Layer YAML specification, which simplifies how you define metrics and dimensions by embedding semantic annotations directly alongside each model. Key changes in the new spec: * `semantic_model` is nested directly under each model instead of being a standalone top-level key. * Entities and dimensions are defined at the column level. * Measures are replaced with `type: simple` metrics defined within the model. * `type_params` is deprecated; its parameters are now top-level keys within each metric definition. For migration guidance and a comparison between the latest spec and the legacy spec, refer to [Migrate to the latest YAML spec](https://docs.getdbt.com/docs/build/latest-metrics-spec.md). For the semantic model reference, refer to [Semantic models](https://docs.getdbt.com/docs/build/semantic-models.md). ##### `selector` method for named YAML selectors You can reference a named selector from `selectors.yml` inside `--select` or `--exclude` using the [`selector` method](https://docs.getdbt.com/reference/node-selection/methods.md#selector) (for example, `selector:my_selector`). This makes it easier to compose reusable YAML selectors with other [selection methods](https://docs.getdbt.com/reference/node-selection/methods.md), [graph operators](https://docs.getdbt.com/reference/node-selection/graph-operators.md), and [set operators](https://docs.getdbt.com/reference/node-selection/set-operators.md) on the command line without duplicating logic. When you use the legacy `--selector` flag together with `--select` or `--exclude`, dbt only uses `--selector` for node selection and ignores `--select` and `--exclude`. Starting in dbt v1.12, dbt also raises a warning when these flags are combined. If you want to combine a selector with `--select` or `--exclude`, use the new `selector:` method instead. ##### Support for `vars.yml` You can use the [`vars.yml`](https://docs.getdbt.com/docs/build/project-variables.md#defining-variables-in-varsyml) file, located at the project root, to define project variables. This keeps variable definitions in one place and helps simplify `dbt_project.yml`. Variables defined in `vars.yml` are parsed *before* `dbt_project.yml`, so you can reference them in `dbt_project.yml` using `{{ var('...') }}`. You can continue to define variables in `dbt_project.yml` as before, but you cannot define variables in both files. For details and precedence, refer to [Project variables](https://docs.getdbt.com/docs/build/project-variables.md). ##### Improved exception handling and error messages dbt v1.12 improves exception handling so error messages are clearer and stack traces are easier to interpret. Previously, some internal failures surfaced as Python errors (for example, `AttributeError`, `KeyError`, `IndexError`, `RuntimeError`), which could be difficult to understand. In dbt v1.12, these are replaced with dbt errors (such as `CompilationError` and `ParsingError`) that include a clear error message. When you need the full Python error output for debugging, use `--debug` or check the logs. Key improvements: * **Cleaner default output**: Built-in Python exceptions (`Exception`, `ValueError`, `RuntimeError`) are replaced with dbt errors, so dbt no longer treats them as internal errors or displays unnecessary stack traces. * **Parsing and config validation**: Invalid field values raise a `ParsingError` instead of a raw `InvalidFieldValue` exception when applying `dbt_project.yml` configs to resources. In a generic data test, a `config` value that is a string or a number instead of a set of key-value pairs raises a `TestConfigNotDictError`. * **Snapshot validation**: When snapshot validation fails, dbt shows the relevant error message and omits the long Python error output. * **`dbt run-operation`**: When a `run-operation` call fails, the exception message is included in `run_results.json`, which makes failures easier to inspect. * **Cycle detection**: Dependency graph cycles raise a `CompilationError` instead of the built-in `RuntimeError`. * **Semantic model dependencies**: When a semantic model references a disabled or missing model, dbt raises a `CompilationError` instead of an `IndexError`.  More scenarios with exception handling improvements * A string concatenation in a `doc()` argument (such as `doc('foo' ~ 'bar')`) is skipped during doc block resolution instead of crashing with an `AttributeError`. * A Jinja variable (such as `doc(my_variable)`) raises a `DocTargetNotFoundError`. * When a `meta` value in `schema.yml` references an undefined Jinja variable, dbt converts it to `None` instead of raising a `TypeError` during partial parse. * When `sources`, `tables`, `exposure` tags, or `packages` are set to `null`, dbt treats them as an empty list instead of raising a `TypeError`. * When a model with custom contract constraints is evaluated during `state:modified` selection, dbt returns `None` for unknown constraint types instead of raising a `KeyError`. ##### Managing changes to legacy behaviors dbt v1.12 introduces new flags for [managing changes to legacy behaviors](https://docs.getdbt.com/reference/global-configs/behavior-changes.md). You may opt into recently introduced changes (disabled by default), or opt out of mature changes (enabled by default), by setting `true` / `false` values, respectively, for `flags` in `dbt_project.yml`. You can read more about each of these behavior changes in the following links: * (Mature, enabled by default) [`require_all_warnings_handled_by_warn_error`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_all_warnings_handled_by_warn_error.md). All warnings raised during a run are now routed through the `--warn-error` / `--warn-error-options` handler. This only affects projects using `warn_error: true` or `--warn-error`. * (Mature, enabled by default) [`require_batched_execution_for_custom_microbatch_strategy`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_batched_execution_for_custom_microbatch_strategy.md). dbt now executes custom microbatch strategies in batches. If you have a custom microbatch macro and set the flag to `false`, dbt issues a deprecation warning. * (Mature, enabled by default) [`require_nested_cumulative_type_params`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_nested_cumulative_type_params.md). dbt now raises an error (instead of a warning) if cumulative metrics use the un-nested syntax. Cumulative-type metrics must be nested under the `cumulative_type_params` field, as required since dbt v1.9. * (Mature, enabled by default) [`require_yaml_configuration_for_mf_time_spines`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_yaml_configuration_for_mf_time_spines.md). dbt suppresses the `MFTimespineWithoutYamlConfigurationDeprecation` warning. The legacy SQL file configuration continues to work. This flag has no functional impact. * (Mature, enabled by default) [`skip_nodes_if_on_run_start_fails`](https://docs.getdbt.com/reference/global-configs/behavior-flags/skip_nodes_if_on_run_start_fails.md). dbt skips all selected resources if there is a failure on an `on-run-start` hook. * (Mature, enabled by default) [`state_modified_compare_more_unrendered_values`](https://docs.getdbt.com/reference/global-configs/behavior-flags/state_modified_compare_more_unrendered_values.md). dbt uses unrendered values in `state:modified` comparisons, reducing false positives when configs differ by target environment (such as `prod` vs. `dev`). * (Mature, enabled by default) [`validate_macro_args`](https://docs.getdbt.com/reference/global-configs/behavior-flags/validate_macro_args.md). dbt validates macro arguments by default. * (Introduced, disabled by default) [`require_valid_schema_from_generate_schema_name`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_valid_schema_from_generate_schema_name.md). This flag is set to `false` by default. With this setting, dbt raises the [`GenerateSchemaNameNullValueDeprecation`](https://docs.getdbt.com/reference/deprecations.md#generateschemanamenullvaluedeprecation) warning when a custom `generate_schema_name` macro returns a `null` value. When set to `true`, dbt enforces stricter validation and raises a parsing error instead of a warning. * (Introduced, disabled by default) [`require_sql_header_in_test_configs`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_sql_header_in_test_configs.md). When set to `true`, you can set [`sql_header`](https://docs.getdbt.com/reference/resource-configs/sql_header.md) in the `config` of a generic data test at the model or column level in your `properties.yml` file. For more information, refer to [Data test configurations](https://docs.getdbt.com/reference/data-test-configs.md). * (Introduced, disabled by default) [`require_corrected_analysis_fqns`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_corrected_analysis_fqns.md). When set to `true`, dbt applies project-level analysis configuration from `dbt_project.yml`. Previously, dbt silently ignored this configuration. This flag also corrects fully qualified names (FQNs) of analyses by removing the extra path segment, making them consistent with other resource types (for example, `your_project.my_analysis` instead of `your_project.analyses.my_analysis`). For more information, refer to [Analyses](https://docs.getdbt.com/docs/build/analyses.md). * (Introduced, disabled by default) [`require_source_and_semantic_model_names_without_spaces`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_source_and_semantic_model_names_without_spaces.md). By default, dbt raises a [`ResourceNamesWithSpacesDeprecation`](https://docs.getdbt.com/reference/deprecations.md#resourcenameswithspacesdeprecation) warning if it detects a space in a source name or semantic model name. When the flag is set to `true`, dbt raises an error. * (Introduced, disabled by default) [`allow_jinja_file_extensions`](https://docs.getdbt.com/reference/global-configs/behavior-flags/allow_jinja_file_extensions.md). When set to `True`, dbt recognizes Jinja-style extension suffixes (`.j2`, `.jinja`, `.jinja2`) on `.sql` and `.md` files. This enables Jinja-aware syntax highlighting in IDEs that associate these suffixes with Jinja templating. * (Introduced, disabled by default) [`latest_version_pointer_enabled_by_default`](https://docs.getdbt.com/reference/global-configs/behavior-flags/latest_version_pointer_enabled_by_default.md). When set to `true`, dbt automatically creates a latest version pointer view for every versioned model in your project, without requiring per-model configuration. #### Adapter-specific features and functionalities ##### Snowflake * You can use the [`snowflake.quote_args`](https://docs.getdbt.com/reference/resource-configs/quote_args.md) config on JavaScript UDFs to control whether argument names are quoted in the generated `CREATE FUNCTION` statement. When `true` (default), Snowflake quotes argument names, preserving their exact casing, so you reference arguments using the same case as defined in the YAML inside the function body. When `false`, argument names are unquoted and Snowflake uppercases them, so you must reference them in uppercase inside the function body. * You can set the [`iceberg_version`](https://docs.getdbt.com/docs/build/iceberg/adapters/snowflake-iceberg-support.md) config on Snowflake Iceberg tables to control which Iceberg format version Snowflake uses. Set it to `3` to use Iceberg V3, which improves `VARIANT` type support and makes row-level changes more efficient by tracking deletions separately instead of rewriting data. The default value is `2`. Note that you cannot change the value of `iceberg_version` after table creation. * You can configure the [`scheduler`](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#scheduler) parameter on Snowflake dynamic tables to control how refreshes are managed. Setting it to `ENABLE` lets Snowflake automatically refresh the dynamic table, while `DISABLE` means dbt manages refreshes during model execution. When `scheduler` is set to `ENABLE`, you must also specify [`target_lag`](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#target-lag). By default, dbt sets `scheduler` to `DISABLE` (unlike Snowflake’s native default of `ENABLE`), so dbt controls the refresh schedule unless you explicitly opt in to Snowflake’s scheduler. If you specify `target_lag` without setting `scheduler`, dbt automatically sets `scheduler` to `ENABLE`. * You can use the [`snowflake_initialization_warehouse`](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#initialization-warehouse) parameter to specify a warehouse for the initial build and reinitialization of a dynamic table, separate from `snowflake_warehouse` which is used for regular incremental refreshes. * You can create Snowflake dynamic tables as transient (no [Fail-safe period](https://docs.snowflake.com/en/user-guide/data-failsafe)) by setting the [`transient`](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#transient-dynamic-tables) config on models. When `transient` is not set on a model, the [`snowflake_default_transient_dynamic_tables`](https://docs.getdbt.com/reference/global-configs/snowflake-changes.md#the-snowflake_default_transient_dynamic_tables-flag) flag controls the default. Set this flag to `true` to make all dynamic tables transient by default. ##### BigQuery * BigQuery now supports [parallel microbatch execution](https://docs.getdbt.com/docs/build/parallel-batch-execution.md), in addition to Snowflake. * Added the [`bigquery_use_standard_sql_for_partitions`](https://docs.getdbt.com/reference/global-configs/bigquery-changes.md#the-bigquery_use_standard_sql_for_partitions-flag) flag, which controls whether `get_partitions_metadata()` uses standard SQL (`INFORMATION_SCHEMA.PARTITIONS`) or legacy SQL (`$__PARTITIONS_SUMMARY__`). The flag defaulted to `false` when first introduced in this release, but has been flipped to `true` by default ahead of BigQuery's [legacy SQL deprecation on June 1, 2026](https://docs.cloud.google.com/bigquery/docs/release-notes#February_25_2026). To revert to legacy SQL, set the flag to `false` in `dbt_project.yml`. * Added the [`bigquery_reject_wildcard_metadata_source_freshness`](https://docs.getdbt.com/reference/global-configs/bigquery-changes.md#the-bigquery_reject_wildcard_metadata_source_freshness-flag) flag. When you set this flag to `true`, dbt raises a `DbtRuntimeError` if you run metadata-based source freshness checks with wildcard table identifiers (for example, `events_*`), preventing incorrect freshness results. * You can configure BigQuery job link logging with `job_link_info_level_log`. By default, dbt logs job links at the debug level. To log job links at the info level, set `job_link_info_level_log: true` in your BigQuery profile. This makes job links visible in dbt logs for easier access to the BigQuery console. For more information, see [BigQuery setup](https://docs.getdbt.com/docs/local/connect-data-platform/bigquery-setup.md#job_link_info_level_log). * You can set `job_execution_timeout_seconds` per model, snapshot, seed, or test, in addition to the profile-level configuration. The per-resource value takes precedence over the default value set in the profile level. For more information, refer to [BigQuery setup](https://docs.getdbt.com/docs/local/connect-data-platform/bigquery-setup.md#job_execution_timeout_seconds). ##### Redshift * The [`redshift_skip_autocommit_transaction_statements`](https://docs.getdbt.com/reference/global-configs/redshift-changes.md#redshift_skip_autocommit_transaction_statements-flag) flag defaults to `false`, preserving legacy behavior of sending `BEGIN`/`COMMIT`/`ROLLBACK` statements even when autocommit is enabled. To skip unnecessary transaction statements and improve performance, set the flag to `true`. * Added support for the `query_group` session parameter, allowing dbt to tag queries for Redshift Workload Manager routing and query logging. When configured in a profile, dbt sets `query_group` when opening a connection and the value applies for the duration of that session. You can also configure `query_group` at the model level to temporarily override the default value for a specific model, and dbt reverts the value at the end of model materialization. For more information, see [Redshift configurations](https://docs.getdbt.com/reference/resource-configs/redshift-configs.md#session-configuration). ##### Databricks * You can use the [`row_filter`](https://docs.getdbt.com/reference/resource-configs/databricks-configs.md#setting-row-filters) config to apply a [Unity Catalog row filter](https://docs.databricks.com/aws/en/tables/row-and-column-filters) to a model, restricting which rows a query returns based on a SQL UDF. Row filters are supported on `table`, `incremental`, `materialized_view`, and `streaming_table` materializations. They are not supported on regular views or Hive Metastore relations. * [`databricks_tags`](https://docs.getdbt.com/reference/resource-configs/databricks-configs.md#databricks_tags) has two updates in v1.12: * You can now set a tag's value to `''` or `None` to apply a key-only tag with no value. * Configs now merge additively across hierarchy levels (for example, project-level and model-level) instead of lower-level configs completely replacing higher-level ones. When the same tag key is defined at multiple levels, the lower-level value takes precedence. #### Quick hits * dbt now surfaces occasional, non-blocking hints that suggest ways to optimize your project. Hints are enabled by default. Disable them with the [`hints_enabled` flag](https://docs.getdbt.com/reference/global-configs/about-global-configs.md). * dbt v1.12 now tolerates dbt v2-specific names in [`warn_error_options`](https://docs.getdbt.com/reference/global-configs/warnings.md) instead of raising an error. If your config includes a v2-specific name (for example, `StaticAnalysis`, `PackageParsingCompatibility`), dbt v1 ignores it and emits a note: `<name> is not being used because it's specific to the dbt Fusion engine.` This lets you share `warn_error_options` configs across dbt v1 and dbt v2. * Macros invoked with the [`dbt run-operation`](https://docs.getdbt.com/reference/commands/run-operation.md) command can now `ref()` models with `private` or `protected` [access](https://docs.getdbt.com/reference/resource-configs/access.md) without raising a `DbtReferenceError`. Because macros are not part of the group and access control system, dbt doesn't enforce group membership when a macro called by `run-operation` references a model. * `dbt seed` now supports the [`--empty`](https://docs.getdbt.com/reference/commands/seed.md#the---empty-flag) flag. Use it to create seed tables with the correct schema but without loading any data. * dbt v1 now automatically loads environment variables from a `.env` file in your current working directory. Shell environment variables take precedence over `.env` values. New projects created with `dbt init` include `.env` in the default `.gitignore`. For more information, refer to [About env\_var function](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md#using-the-env-file). * `dbt compile` writes compiled SQL for [snapshots](https://docs.getdbt.com/docs/build/snapshots.md) to `target/compiled/`, consistent with models, tests, analyses, and functions. Each snapshot gets its own output file, named from the snapshot identifier, so multiple snapshot blocks in the same source file do not share one compiled path. For more information, refer to [About dbt compile](https://docs.getdbt.com/reference/commands/compile.md). --- ### Upgrading to v1.2 Available in v1 ##### Resources * [Changelog](https://github.com/dbt-labs/dbt/blob/1.2.latest/CHANGELOG.md) * [dbt v1 CLI Installation guide](https://docs.getdbt.com/docs/local/install-dbt.md) * [Cloud upgrade guide](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) #### What to know before upgrading There are no breaking changes for code in dbt projects and packages. We are committed to providing backwards compatibility for all versions 1.x. If you encounter an error upon upgrading, please let us know by [opening an issue](https://github.com/dbt-labs/dbt/issues/new). ##### For consumers of dbt artifacts (metadata) The manifest schema version has been updated to `v6`. The relevant changes are: * Change to `config` default, which includes a new `grants` property with default value `{}` * Addition of a `metrics` property, to any node which could reference metrics using the `metric()` function For users of [state-based selection](https://docs.getdbt.com/reference/node-selection/syntax.md#about-node-selection): This release also includes new logic declaring forwards compatibility for older manifest versions. While running dbt v1.2, it should be possible to use `state:modified --state ...` selection against a manifest produced by dbt v1.0 or v1.1. #### For maintainers of adapter plugins See GitHub discussion [dbt-labs/dbt#5468](https://github.com/dbt-labs/dbt/discussions/5468) for detailed information #### New and changed functionality * **[Grants](https://docs.getdbt.com/reference/resource-configs/grants.md)** are natively supported in `dbt-core` for the first time. That support extends to all standard materializations, and the most popular adapters. If you already use hooks to apply simple grants, we encourage you to use built-in `grants` to configure your models, seeds, and snapshots instead. This will enable you to [DRY](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself) up your duplicated or boilerplate code. * **[Metrics](https://docs.getdbt.com/docs/build/build-metrics-intro.md)** now support an `expression` type (metrics-on-metrics), as well as a `metric()` function to use when referencing metrics from within models, macros, or `expression`-type metrics. For more information on how to use expression metrics, check out the [**`dbt_metrics` package**](https://github.com/dbt-labs/dbt_metrics) * **[dbt-Jinja functions](https://docs.getdbt.com/reference/dbt-jinja-functions-context-variables.md)** now include the [`itertools` Python module](https://docs.getdbt.com/reference/dbt-jinja-functions/modules.md#itertools), as well as the [set](https://docs.getdbt.com/reference/dbt-jinja-functions/set.md) and [zip](https://docs.getdbt.com/reference/dbt-jinja-functions/zip.md) functions. * **[Node selection](https://docs.getdbt.com/reference/node-selection/syntax.md)** includes a [file selection method](https://docs.getdbt.com/reference/node-selection/methods.md#file) (`-s model.sql`), and [yaml selector](https://docs.getdbt.com/reference/node-selection/yaml-selectors.md) inheritance. * **[Global configs](https://docs.getdbt.com/reference/global-configs/about-global-configs.md)** now include CLI flag and environment variable settings for [`target-path`](https://docs.getdbt.com/reference/global-configs/json-artifacts.md) and [`log-path`](https://docs.getdbt.com/reference/global-configs/logs.md), which can be used to override the values set in `dbt_project.yml` ##### Specific adapters * [Postgres](https://docs.getdbt.com/docs/local/connect-data-platform/postgres-setup.md) and [Redshift](https://docs.getdbt.com/docs/local/connect-data-platform/redshift-setup.md) profiles support a `retries` config, if dbt encounters an operational error or timeout when opening a connection. The default is 1 retry. --- ### Upgrading to v1.3 Available in v1 dbt v1.3 – v1.7 will be deprecated on January 31, 2027 dbt v1 versions v1.3-v1.7 have reached [end of life](https://docs.getdbt.com/docs/dbt-versions.md#end-of-life-versions) and will be deprecated on January 31, 2027. After that date, these versions are no longer maintained by dbt Labs and will be removed from dbt platform. Upgrade your environments to a [supported dbt version](https://docs.getdbt.com/docs/dbt-versions.md) or a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) before then to keep receiving updates and support. For more information, check out the [Migrate off legacy dbt versions](https://docs.getdbt.com/guides/migrate-off-legacy-dbt-versions.md?step=1) guide. ##### Resources * [Changelog](https://github.com/dbt-labs/dbt/blob/1.3.latest/CHANGELOG.md) * [dbt v1 CLI Installation guide](https://docs.getdbt.com/docs/local/install-dbt.md) * [Cloud upgrade guide](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) #### What to know before upgrading We are committed to providing backward compatibility for all versions 1.x. If you encounter an error upon upgrading, please let us know by [opening an issue](https://github.com/dbt-labs/dbt/issues/new). There are three changes in dbt v1.3 that may require action from some users: 1. If you have a `profiles.yml` file located in the root directory where you run dbt, dbt will start preferring that profiles file over the default location on your machine. [You can read more details here](https://docs.getdbt.com/docs/local/profiles.yml.md#advanced-customizing-a-profile-directory). 2. If you already have `.py` files defined in the `model-paths` of your dbt project, dbt will start trying to read them as Python models. You can use [the new `.dbtignore` file](https://docs.getdbt.com/reference/dbtignore.md) to tell dbt to ignore those files. 3. If you have custom code accessing the `raw_sql` property of models (with the [model](https://docs.getdbt.com/reference/dbt-jinja-functions/model.md) or [graph](https://docs.getdbt.com/reference/dbt-jinja-functions/graph.md) objects), it has been renamed to `raw_code`. This is a change to the manifest contract, described in more detail below. ##### For users of dbt Metrics The names of metric properties have changed, with backward compatibility. Those changes are: * Renamed `type` to `calculation_method` * Renamed `sql` to `expression` * Renamed `expression` calculation method metrics to `derived` calculation method metrics We plan to keep backward compatibility for a full minor version. Defining metrics with the old names will raise an error in dbt v1.4. ##### For consumers of dbt artifacts (metadata) We have updated the manifest schema version to `v7`. This includes the changes to metrics described above and a few other changes related to the addition of Python models: * Renamed `raw_sql` to `raw_code` * Renamed `compiled_sql` to `compiled_code` * A new top-level node property, `language` (`'sql'` or `'python'`) For users of [state-based selection](https://docs.getdbt.com/reference/node-selection/syntax.md#about-node-selection): This release includes logic providing backward and forward compatibility for older manifest versions. While running dbt v1.3, it should be possible to use `state:modified --state ...` selection against a manifest produced by dbt v1.0 and higher. ##### For maintainers of adapter plugins GitHub discussion with details: [dbt-labs/dbt#6011](https://github.com/dbt-labs/dbt/discussions/6011) #### New and changed documentation * **[Python models](https://docs.getdbt.com/docs/build/python-models.md)** are natively supported in `dbt-core` for the first time, on data warehouses that support Python runtimes. * Updates made to **[Metrics](https://docs.getdbt.com/docs/build/build-metrics-intro.md)** reflect their new syntax for definition, as well as additional properties that are now available. * Plus, a few related updates to **[exposure properties](https://docs.getdbt.com/reference/exposure-properties.md)**: `config`, `label`, and `name` validation. * **[Custom `node_color`](https://docs.getdbt.com/reference/resource-configs/docs.md)** in `dbt-docs`. For the first time, you can control the colors displayed in dbt's DAG. Want bronze, silver, and gold layers? It's at your fingertips. * **[`Profiles.yml`](https://docs.getdbt.com/docs/local/profiles.yml.md#advanced-customizing-a-profile-directory)** search order now looks in the current working directory before `~/.dbt`. ##### Quick hits * **["Full refresh"](https://docs.getdbt.com/reference/resource-configs/full_refresh.md)** flag supports a short name, `-f`. * **[The "config" selection method](https://docs.getdbt.com/reference/node-selection/methods.md#config)** supports boolean and list config values, in addition to strings. * Two new dbt-Jinja context variables for accessing invocation metadata: [`invocation_args_dict`](https://docs.getdbt.com/reference/dbt-jinja-functions/flags.md#invocation_args_dict) and [`dbt_metadata_envs`](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md#custom-metadata). --- ### Upgrading to v1.4 Available in v1 dbt v1.3 – v1.7 will be deprecated on January 31, 2027 dbt v1 versions v1.3-v1.7 have reached [end of life](https://docs.getdbt.com/docs/dbt-versions.md#end-of-life-versions) and will be deprecated on January 31, 2027. After that date, these versions are no longer maintained by dbt Labs and will be removed from dbt platform. Upgrade your environments to a [supported dbt version](https://docs.getdbt.com/docs/dbt-versions.md) or a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) before then to keep receiving updates and support. For more information, check out the [Migrate off legacy dbt versions](https://docs.getdbt.com/guides/migrate-off-legacy-dbt-versions.md?step=1) guide. ##### Resources * [Changelog](https://github.com/dbt-labs/dbt/blob/1.4.latest/CHANGELOG.md) * [dbt v1 CLI Installation guide](https://docs.getdbt.com/docs/local/install-dbt.md) * [Cloud upgrade guide](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) **Final release:** January 25, 2023 dbt v1.4 is a "behind-the-scenes" release. We've been hard at work rebuilding `dbt-core` internals on top of more-solid foundations, to enable an exciting year of new feature development. Check out the [v1.5 milestone](https://github.com/dbt-labs/dbt/milestone/82) in GitHub for a preview of what's planned for April. #### What to know before upgrading dbt Labs is committed to providing backward compatibility for all versions 1.x. If you encounter an error upon upgrading, please let us know by [opening an issue](https://github.com/dbt-labs/dbt/issues/new). ##### For consumers of dbt artifacts (metadata) The manifest schema version has updated to `v8`. These changes are relevant for people who parse or analyze the contents of the `manifest.json` file, or who have custom code accessing the [`model`](https://docs.getdbt.com/reference/dbt-jinja-functions/model.md) or [`graph`](https://docs.getdbt.com/reference/dbt-jinja-functions/graph.md) variables, for example, `{{ model.root_path }}`. Relevant changes: * The `root_path` attribute has been removed for non-seed nodes to reduce duplicative information. * Unused attributes have been removed from seed nodes (including `depends_on.nodes`), and from `macros` (including `tags`). * The `unique_id` of docs blocks now start with `doc` for consistency with other resource types. ##### For maintainers of adapter plugins > **TL;DR** Not much heavy lifting for this minor version. We anticipate more work for `1.5.0`. We plan to release betas early & often, and provide guidance on upgrading. The high-level changes are: * Add support for Python 3.11 * Rename/replace deprecated exception functions * Add support for Incremental Predicates (if applicable) * Make use of new adapter-zone tests For more detailed information and to ask any questions, please visit [dbt v1/discussions/6624](https://github.com/dbt-labs/dbt/discussions/6624). #### New and changed documentation * [**Events and structured logging**](https://docs.getdbt.com/reference/events-logging.md): dbt's event system got a makeover. Expect more consistency in the availability and structure of information, backed by type-safe event schemas. * [**Python support**](https://docs.getdbt.com/faqs/Core/install-python-compatibility.md): Python 3.11 was released in October 2022. It is officially supported in dbt-core v1.4, although full support depends also on the adapter plugin for your data platform. According to the Python maintainers, "Python 3.11 is between 10-60% faster than Python 3.10." We encourage you to try [`dbt parse`](https://docs.getdbt.com/reference/commands/parse.md) with dbt v1.4 + Python 3.11, and compare the timing with dbt v1.3 + Python 3.10. Let us know what you find! * [**Metrics**](https://docs.getdbt.com/docs/build/build-metrics-intro.md): `time_grain` is optional, to provide better ergonomics around metrics that aren't time-bound. * **dbt-Jinja context:** The [local\_md5](https://docs.getdbt.com/reference/dbt-jinja-functions/local_md5.md) context method will calculate an [MD5 hash](https://en.wikipedia.org/wiki/MD5) for use *within* dbt. (Not to be confused with SQL md5!) * [**Exposures**](https://docs.getdbt.com/docs/build/exposures.md) can now depend on `metrics`. * [**"Tarball" packages**](https://docs.getdbt.com/docs/build/packages.md#internally-hosted-tarball-URL): Some organizations have security requirements to pull resources only from internal services. To address the need to install packages from hosted environments (such as Artifactory or cloud storage buckets), it's possible to specify any accessible URL where a compressed dbt package can be downloaded. * [**Granular "warn error" configuration**](https://docs.getdbt.com/reference/global-configs/warnings.md): Thanks to a full cleanup and consolidation of warning and exception classes within `dbt-core`, it is now possible to define a more granular `--warn-error-options` configuration that specifies the exact warnings you do (or don't) want dbt to treat as errors. * [**Deferral**](https://docs.getdbt.com/reference/node-selection/defer.md#favor-state) supports an optional configuration, `--favor-state`. ##### Advanced configurations for incremental models * [**`incremental_predicates`** config](https://docs.getdbt.com/docs/build/incremental-strategy.md#about-incremental_predicates) is now supported on the most popular adapters, enabling greater flexibility when tuning performance in `merge` and `delete` statements against large datasets. * **BigQuery:** The `insert_overwrite` incremental strategy supports a new (old) mechanism, [`time_ingestion_partitioning`](https://docs.getdbt.com/reference/resource-configs/bigquery-configs.md#partitioning-by-an-ingestion-date-or-timestamp) + [`copy_partitions`](#copying-ingestion-time-partitions), that can yield significant savings in cost + time for large datasets. ##### Updates to Python models * Python models are [configured to materialize](https://docs.getdbt.com/docs/build/python-models.md) as `table` by default. * Python models [running on Snowpark](https://docs.getdbt.com/docs/build/python-models.md) will use "anonymous" stored procedures by default, enabling a small speedup and a cleaner query history. --- ### Upgrading to v1.5 Available in v1 dbt v1.3 – v1.7 will be deprecated on January 31, 2027 dbt v1 versions v1.3-v1.7 have reached [end of life](https://docs.getdbt.com/docs/dbt-versions.md#end-of-life-versions) and will be deprecated on January 31, 2027. After that date, these versions are no longer maintained by dbt Labs and will be removed from dbt platform. Upgrade your environments to a [supported dbt version](https://docs.getdbt.com/docs/dbt-versions.md) or a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) before then to keep receiving updates and support. For more information, check out the [Migrate off legacy dbt versions](https://docs.getdbt.com/guides/migrate-off-legacy-dbt-versions.md?step=1) guide. dbt v1.5 is a feature release, with two significant additions: 1. [**Model governance**](https://docs.getdbt.com/docs/mesh/govern/about-model-governance.md) — access, contracts, versions — the first phase of [multi-project deployments](https://github.com/dbt-labs/dbt/discussions/6725) 2. A Python entry point for [**programmatic invocations**](https://docs.getdbt.com/reference/programmatic-invocations.md), at parity with the CLI #### Resources * [Changelog](https://github.com/dbt-labs/dbt/blob/1.5.latest/CHANGELOG.md) * [dbt v1 CLI Installation guide](https://docs.getdbt.com/docs/local/install-dbt.md) * [Cloud upgrade guide](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) * [Release schedule](https://github.com/dbt-labs/dbt/issues/6715) #### What to know before upgrading dbt Labs is committed to providing backward compatibility for all versions 1.x, with the exception of any changes explicitly mentioned below. If you encounter an error upon upgrading, please let us know by [opening an issue](https://github.com/dbt-labs/dbt/issues/new). ##### Behavior changes Why changes to previous behavior? This release includes significant new features, and rework to `dbt-core`'s CLI and initialization flow. As part of refactoring its internals from [`argparse`](https://docs.python.org/3/library/argparse.html) to [`click`](https://click.palletsprojects.com), we made a handful of changes to runtime configuration. The net result of these changes is more consistent and practical configuration options, and a more legible codebase. ***Wherever possible, we will provide backward compatibility and deprecation warnings for at least one minor version before actually removing the old functionality.*** In those cases, we still reserve the right to fully remove backwards compatibility for deprecated functionality in a future v1.x minor version of `dbt-core`. Setting `log-path` and `target-path` in `dbt_project.yml` has been deprecated for consistency with other invocation-specific runtime configs ([dbt-labs/dbt#6882](https://github.com/dbt-labs/dbt/issues/6882)). We recommend setting via env var or CLI flag instead. The `dbt list` command will now include `INFO` level logs by default. Previously, the `list` command (and *only* the `list` command) had `WARN`-level stdout logging, to support piping its results to [`jq`](https://jqlang.github.io/jq/manual/), a file, or another process. To achieve that goal, you can use either of the following parameters: * `dbt list --log-level warn` (recommended; equivalent to previous default) * `dbt list --quiet` (suppresses all logging less than ERROR level, except for "printed" messages and `list` output) The following env vars have been renamed, for consistency with the convention followed by all other parameters: * `DBT_DEFER_TO_STATE` → `DBT_DEFER` * `DBT_FAVOR_STATE_MODE` → `DBT_FAVOR_STATE` * `DBT_NO_PRINT` → `DBT_PRINT` * `DBT_ARTIFACT_STATE_PATH` → `DBT_STATE` As described in [dbt-labs/dbt#7169](https://github.com/dbt-labs/dbt/pull/7169), command-line parameters that could be silent before will no longer be silent. See [dbt-labs/dbt#7158](https://github.com/dbt-labs/dbt/issues/7158) and [dbt-labs/dbt#6800](https://github.com/dbt-labs/dbt/issues/6800) for more examples of the behavior we are fixing. An empty `tests:` key in a yaml file will now raise a validation error, instead of being silently skipped. You can resolve this by removing the empty `tests:` key, or by setting it to an empty list explicitly: ```yml # ❌ this will raise an error models: - name: my_model tests: config: ... # ✅ this is fine models: - name: my_model tests: [] # todo! add tests later config: ... ``` Some options that could previously be specified *after* a subcommand can now only be specified *before*. This includes the inverse of the option, `--write-json` and `--no-write-json`, for example. The list of affected options are: List of affected options ```bash --cache-selected-only | --no-cache-selected-only --debug, -d | --no-debug --deprecated-print | --deprecated-no-print --enable-legacy-logger | --no-enable-legacy-logger --fail-fast, -x | --no-fail-fast --log-cache-events | --no-log-cache-events --log-format --log-format-file --log-level --log-level-file --log-path --macro-debugging | --no-macro-debugging --partial-parse | --no-partial-parse --partial-parse-file-path --populate-cache | --no-populate-cache --print | --no-print --printer-width --quiet, -q | --no-quiet --record-timing-info, -r --send-anonymous-usage-stats | --no-send-anonymous-usage-stats --single-threaded | --no-single-threaded --static-parser | --no-static-parser --use-colors | --no-use-colors --use-colors-file | --no-use-colors-file --use-experimental-parser | --no-use-experimental-parser --version, -V, -v --version-check | --no-version-check --warn-error --warn-error-options --write-json | --no-write-json ``` Additionally, some options that could be previously specified *before* a subcommand can now only be specified *after*. Any option *not* in the above list must appear *after* the subcommand from v1.5 and later. For example, `--profiles-dir`. The built-in [collect\_freshness](https://github.com/dbt-labs/dbt/blob/1.5.latest/core/dbt/include/global_project/macros/adapters/freshness.sql) macro now returns the entire `response` object, instead of just the `table` result. If you're using a custom override for `collect_freshness`, make sure you're also returning the `response` object; otherwise, some of your dbt commands will never finish. For example: ```sql {{ return(load_result('collect_freshness')) }} ``` Finally: The [built-in `generate_alias_name` macro](https://github.com/dbt-labs/dbt/blob/1.5.latest/core/dbt/include/global_project/macros/get_custom_name/get_custom_alias.sql) now includes logic to handle versioned models. If your project has reimplemented the `generate_alias_name` macro with custom logic, and you want to start using [model versions](https://docs.getdbt.com/docs/mesh/govern/model-versions.md), you will need to update the logic in your macro. Note that, while this is **not** a prerequisite for upgrading to v1.5—only for using the new feature—we recommend that you do this during your upgrade, whether you're planning to use model versions tomorrow or far in the future. Likewise, if your project has reimplemented the `ref` macro with custom logic, you will need to update the logic in your macro as described [here](https://docs.getdbt.com/reference/dbt-jinja-functions/builtins.md). ##### For consumers of dbt artifacts (metadata) The [manifest](https://docs.getdbt.com/reference/artifacts/manifest-json.md) schema version will be updated to `v9`. Specific changes: * Addition of `groups` as a top-level key * Addition of `access`, `constraints`, `version`, `latest_version` as a top-level node attributes for models * Addition of `constraints` as a column-level attribute * Addition of `group` and `contract` as node configs * To support model versions, the type of `refs` has changed from `List[List[str]]` to `List[RefArgs]`, with nested keys `name: str`, `package: Optional[str] = None`, and `version: Union[str, float, NoneType] = None)`. ##### For maintainers of adapter plugins For more detailed information and to ask questions, please read and comment on the GH discussion: [dbt-labs/dbt#7213](https://github.com/dbt-labs/dbt/discussions/7213). #### New and changed documentation ##### Model governance The first phase of supporting dbt deployments at scale, across multiple projects with clearly defined ownership and interface boundaries. [Read about model governance](https://docs.getdbt.com/docs/mesh/govern/about-model-governance.md), all of which is new in v1.5. ##### Revamped CLI Compile and preview dbt models and `--inline` dbt-SQL queries on the CLI using: * [`dbt compile`](https://docs.getdbt.com/reference/commands/compile.md) * [`dbt show`](https://docs.getdbt.com/reference/commands/show.md) (new!) [Node selection methods](https://docs.getdbt.com/reference/node-selection/methods.md) can use Unix-style wildcards to glob nodes matching a pattern: ```text dbt ls --select "tag:team_*" ``` And (!): a first-ever entry point for [programmatic invocations](https://docs.getdbt.com/reference/programmatic-invocations.md), at parity with CLI commands. Run `dbt --help` to see new & improved help documentation :) ##### Quick hits * The [`version: 2` top-level key](https://docs.getdbt.com/reference/project-configs/version.md) is now **optional** in all YAML files. Also, the [`config-version: 2`](https://docs.getdbt.com/reference/project-configs/config-version.md) and `version:` top-level keys are now optional in `dbt_project.yml` files. * [Events and logging](https://docs.getdbt.com/reference/events-logging.md): Added `node_relation` (`database`, `schema`, `identifier`) to the `node_info` dictionary, available on node-specific events * Support setting `--project-dir` via environment variable: [`DBT_PROJECT_DIR`](https://docs.getdbt.com/reference/dbt_project.yml.md) * More granular configurations for logging (to set [log format](https://docs.getdbt.com/reference/global-configs/logs.md#log-formatting), [log levels](https://docs.getdbt.com/reference/global-configs/logs.md#log-level), and [colorization](https://docs.getdbt.com/reference/global-configs/logs.md#color)) and [cache population](https://docs.getdbt.com/reference/global-configs/cache.md#cache-population) * [dbt overwrites the `manifest.json` file](https://docs.getdbt.com/reference/node-selection/state-comparison-caveats.md#overwrites-the-manifestjson) during parsing, which means when you reference `--state` from the `target/ directory`, you may encounter a warning indicating that the saved manifest wasn't found. --- ### Upgrading to v1.6 Available in v1 dbt v1.3 – v1.7 will be deprecated on January 31, 2027 dbt v1 versions v1.3-v1.7 have reached [end of life](https://docs.getdbt.com/docs/dbt-versions.md#end-of-life-versions) and will be deprecated on January 31, 2027. After that date, these versions are no longer maintained by dbt Labs and will be removed from dbt platform. Upgrade your environments to a [supported dbt version](https://docs.getdbt.com/docs/dbt-versions.md) or a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) before then to keep receiving updates and support. For more information, check out the [Migrate off legacy dbt versions](https://docs.getdbt.com/guides/migrate-off-legacy-dbt-versions.md?step=1) guide. dbt v1.6 has three significant areas of focus: 1. Next milestone of [multi-project deployments](https://github.com/dbt-labs/dbt/discussions/6725): improvements to contracts, groups/access, versions; and building blocks for cross-project `ref` 2. Semantic layer re-launch: dbt v1 and [MetricFlow](https://docs.getdbt.com/docs/build/about-metricflow.md) integration 3. Mechanisms to support mature deployment at scale (`dbt clone` and `dbt retry`) #### Resources * [Changelog](https://github.com/dbt-labs/dbt/blob/1.6.latest/CHANGELOG.md) * [dbt v1 installation guide](https://docs.getdbt.com/docs/local/install-dbt.md) * [Cloud upgrade guide](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) * [Release schedule](https://github.com/dbt-labs/dbt/issues/7481) #### What to know before upgrading dbt Labs is committed to providing backward compatibility for all versions 1.x, with the exception of any changes explicitly mentioned below. If you encounter an error upon upgrading, please let us know by [opening an issue](https://github.com/dbt-labs/dbt/issues/new). ##### Behavior changes Action required if your project defines `metrics` The [spec for metrics](https://github.com/dbt-labs/dbt/discussions/7456) has changed and now uses [MetricFlow](https://docs.getdbt.com/docs/build/about-metricflow.md). If your dbt project defines metrics, you must migrate to dbt v1.6 because the YAML spec has moved from dbt\_metrics to MetricFlow. Any tests you have won't compile on v1.5 or older. * dbt v1.6 does not support Python 3.7, which reached End Of Life on June 23. Support Python versions are 3.8, 3.9, 3.10, and 3.11. * As part of the [dbt Semantic layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) re-launch, the spec for `metrics` has changed significantly. * The manifest schema version is now v10. * dbt Labs is ending support for Homebrew installation of dbt v1 and adapters. See [the discussion](https://github.com/dbt-labs/dbt/discussions/8277) for more details. ##### For consumers of dbt artifacts (metadata) The [manifest](https://docs.getdbt.com/reference/artifacts/manifest-json.md) schema version has been updated to `v10`. Specific changes: * Addition of `semantic_models` and changes to `metrics` attributes * Addition of `deprecation_date` as a model property * Addition of `on_configuration_change` as default node configuration (to support materialized views) * Small type changes to `contracts` and `constraints` * Manifest `metadata` includes `project_name` ##### For maintainers of adapter plugins For more detailed information and to ask questions, please read and comment on the GH discussion: [dbt-labs/dbt-core#7958](https://github.com/dbt-labs/dbt/discussions/7958). #### New and changed documentation ##### MetricFlow * [**Build your metrics**](https://docs.getdbt.com/docs/build/build-metrics-intro.md) with MetricFlow, a key component of the Semantic Layer. You can define your metrics and build semantic models with MetricFlow, available on the command line (CLI) for dbt v1.6 beta or higher. ##### Materialized views Supported on: * [Postgres](https://docs.getdbt.com/reference/resource-configs/postgres-configs.md#materialized-view) * [Redshift](https://docs.getdbt.com/reference/resource-configs/redshift-configs.md#materialized-view) * [Snowflake](https://docs.getdbt.com/reference/resource-configs/snowflake-configs.md#dynamic-tables) * [Databricks](https://docs.getdbt.com/reference/resource-configs/databricks-configs.md#materialized-views-and-streaming-tables) ##### New commands for mature deployment [`dbt retry`](https://docs.getdbt.com/reference/commands/retry.md) executes the previously run command from the point of failure. Rebuild just the nodes that errored or skipped in a previous run/build/test, rather than starting over from scratch. [`dbt clone`](https://docs.getdbt.com/reference/commands/clone.md) leverages each data platform's functionality for creating lightweight copies of dbt models from one environment into another. Useful when quickly spinning up a new development environment, or promoting specific models from a staging environment into production. ##### Multi-project collaboration [**Deprecation date**](https://docs.getdbt.com/reference/resource-properties/deprecation_date.md): Models can declare a deprecation date that will warn model producers and downstream consumers. This enables clear migration windows for versioned models, and provides a mechanism to facilitate removal of immature or little-used models, helping to avoid project bloat. [Model names](https://docs.getdbt.com/faqs/Project/unique-resource-names.md) can be duplicated across different namespaces (projects/packages), so long as they are unique within each project/package. We strongly encourage using [two-argument `ref`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models) when referencing a model from a different package/project. More consistency and flexibility around packages. Resources defined in a package will respect variable and global macro definitions within the scope of that package. * `vars` defined in a package's `dbt_project.yml` are now available in the resolution order when compiling nodes in that package, though CLI `--vars` and the root project's `vars` will still take precedence. See ["Variable Precedence"](https://docs.getdbt.com/docs/build/project-variables.md#variable-precedence) for details. * `generate_x_name` macros (defining custom rules for database, schema, alias naming) follow the same pattern as other "global" macros for package-scoped overrides. See [macro dispatch](https://docs.getdbt.com/reference/dbt-jinja-functions/dispatch.md) for an overview of the patterns that are possible. Closed Beta - dbt Enterprise [**Project dependencies**](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md): Introduces `dependencies.yml` and dependent `projects` as a feature of dbt Enterprise. Allows enforcing model access (public vs. protected/private) across project/package boundaries. Enables cross-project `ref` of public models, without requiring the installation of upstream source code. ##### Deprecated functionality The ability for installed packages to override built-in materializations without explicit opt-in from the user is being deprecated. * Overriding a built-in materialization from an installed package raises a deprecation warning. * Using a custom materialization from an installed package does not raise a deprecation warning. * Using a built-in materialization package override from the root project via a wrapping materialization is still supported. For example: ```text {% materialization view, default %} {{ return(my_cool_package.materialization_view_default()) }} {% endmaterialization %} ``` ##### Quick hits * [`state:unmodified` and `state:old`](https://docs.getdbt.com/reference/node-selection/methods.md#state) for [MECE](https://en.wikipedia.org/wiki/MECE_principle) stateful selection * [`invocation_args_dict`](https://docs.getdbt.com/reference/dbt-jinja-functions/flags.md#invocation_args_dict) includes full `invocation_command` as string * [`dbt debug --connection`](https://docs.getdbt.com/reference/commands/debug.md) to test just the data platform connection specified in a profile * [`dbt docs generate --empty-catalog`](https://docs.getdbt.com/reference/commands/cmd-docs.md) to skip catalog population while generating docs * [`--defer-state`](https://docs.getdbt.com/reference/node-selection/defer.md) enables more-granular control * [`dbt ls`](https://docs.getdbt.com/reference/commands/list.md) adds the Semantic model selection method to allow for `dbt ls -s "semantic_model:*"` and the ability to execute `dbt ls --resource-type semantic_model`. * Syntax for `DBT_ENV_SECRET_` has changed to `DBT_ENV_SECRET` and no longer requires the closing underscore. --- ### Upgrading to v1.7 Available in v1 dbt v1.3 – v1.7 will be deprecated on January 31, 2027 dbt v1 versions v1.3-v1.7 have reached [end of life](https://docs.getdbt.com/docs/dbt-versions.md#end-of-life-versions) and will be deprecated on January 31, 2027. After that date, these versions are no longer maintained by dbt Labs and will be removed from dbt platform. Upgrade your environments to a [supported dbt version](https://docs.getdbt.com/docs/dbt-versions.md) or a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) before then to keep receiving updates and support. For more information, check out the [Migrate off legacy dbt versions](https://docs.getdbt.com/guides/migrate-off-legacy-dbt-versions.md?step=1) guide. #### Resources * [Changelog](https://github.com/dbt-labs/dbt/blob/1.7.latest/CHANGELOG.md) * [dbt v1 CLI Installation guide](https://docs.getdbt.com/docs/local/install-dbt.md) * [Cloud upgrade guide](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) * [Release schedule](https://github.com/dbt-labs/dbt/issues/8260) #### What to know before upgrading dbt Labs is committed to providing backward compatibility for all versions 1.x, with the exception of any changes explicitly mentioned below. If you encounter an error upon upgrading, please let us know by [opening an issue](https://github.com/dbt-labs/dbt/issues/new). Snowflake column size change [Snowflake plans to increase](https://docs.snowflake.com/en/release-notes/bcr-bundles/un-bundled/bcr-2118) the default column size for string and binary data types in September 2026. `dbt-snowflake` versions below v1.10.6 may fail to build certain incremental models when this change is deployed.  Assess impact and required actions If you're using a `dbt-snowflake` version below v1.10.6 or have not yet migrated to a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) in the dbt platform, your adapter version is incompatible with this change and may fail to build incremental models that meet *both* of the following conditions: * Contain string columns with collation defined * Use the `on_schema_change='sync_all_columns'` config To check whether this change affects your project, run the following [list](https://docs.getdbt.com/reference/commands/list.md) command: ```bash dbt ls -s config.materialized:incremental,config.on_schema_change:sync_all_columns --resource-type model ``` * If the command returns `No nodes selected!`, no action is required. * If the command returns one or more models (for example, `Found 1000 models, 644 macros`), you may be impacted if those models have string columns that don't specify a width. In that case, upgrade to a version that includes the fix: * **dbt v1**: `dbt-snowflake` v1.10.6 or later. For upgrade instructions, refer to [Upgrade adapters](https://docs.getdbt.com/docs/local/install-dbt.md) in the dbt v1 installation instructions. * **dbt platform**: Any release track (**v1 Latest**, **v1 Compatible**, **v1 Extended**, or **v1 Fallback**). * **dbt v2**: v2.0.0. This ensures your incremental models can safely handle schema changes while maintaining required collation settings. ##### Behavior changes dbt v1.7 expands the amount of sources you can configure freshness for. Previously, freshness was limited to sources with a `loaded_at_field`; now, freshness can be generated from warehouse metadata tables when available. As part of this change, the `loaded_at_field` is no longer required to generate source freshness. If a source has a `freshness:` block, dbt will attempt to calculate freshness for that source: * If a `loaded_at_field` is provided, dbt will calculate freshness via a select query (previous behavior). * If a `loaded_at_field` is *not* provided, dbt will calculate freshness via warehouse metadata tables when possible (new behavior). This is a relatively small behavior change, but worth calling out in case you notice that dbt is calculating freshness for *more* sources than before. To exclude a source from freshness calculations, explicitly set `freshness: null`. Beginning with v1.7, running [`dbt deps`](https://docs.getdbt.com/reference/commands/deps.md) creates or updates the `package-lock.yml` file in the *project\_root* where `packages.yml` is recorded. The `package-lock.yml` file contains a record of all packages installed and, if subsequent `dbt deps` runs contain no updated packages in `dependencies.yml` or `packages.yml`, dbt-core installs from `package-lock.yml`. To retain the behavior prior to v1.7, there are two main options: 1. Use `dbt deps --upgrade` everywhere `dbt deps` was used previously. 2. Add `package-lock.yml` to your `.gitignore` file. #### New and changed features and functionality * [`dbt docs generate`](https://docs.getdbt.com/reference/commands/cmd-docs.md) now supports `--select` to generate [catalog metadata](https://docs.getdbt.com/reference/artifacts/catalog-json.md) for a subset of your project. * [Source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) can now be generated from warehouse metadata tables. ##### MetricFlow enhancements * Automatically create metrics on measures with [`create_metric: true`](https://docs.getdbt.com/docs/build/semantic-models.md). * Optional [`label`](https://docs.getdbt.com/docs/build/semantic-models.md) in semantic\_models, measures, dimensions, and entities. * New configurations for semantic models - [enable/disable](https://docs.getdbt.com/reference/resource-configs/enabled.md), [group](https://docs.getdbt.com/reference/resource-configs/group.md), and [meta](https://docs.getdbt.com/reference/resource-configs/meta.md). * Support `fill_nulls_with` and `join_to_timespine` for metric nodes. * `saved_queries` extends governance beyond the semantic objects to their consumption. ##### For consumers of dbt artifacts (metadata) * The [manifest](https://docs.getdbt.com/reference/artifacts/manifest-json.md) schema version has been updated to v11. * The [run\_results](https://docs.getdbt.com/reference/artifacts/run-results-json.md) schema version has been updated to v5. * There are a few specific changes to the [catalog.json](https://docs.getdbt.com/reference/artifacts/catalog-json.md): * Added [node attributes](https://docs.getdbt.com/reference/artifacts/run-results-json.md) related to compilation (`compiled`, `compiled_code`, `relation_name`) to the `catalog.json`. * The nodes dictionary in the `catalog.json` can now be "partial" if `dbt docs generate` is run with a selector. ##### Model governance dbt v1.5 introduced model governance which we're continuing to refine. v1.7 includes these additional features and functionality: * **[Breaking change detection](https://docs.getdbt.com/reference/resource-properties/versions.md#detecting-breaking-changes) for models with contracts enforced:** When dbt detects a breaking change to a model with an enforced contract during state comparison, it will now raise an error for versioned models and a warning for models that are not versioned. * **[Set `access` as a config](https://docs.getdbt.com/reference/resource-configs/access.md):** You can now set a model's `access` within config blocks in the model's SQL file or in the project YAML file (`dbt_project.yml`) for an entire subfolder at once. * **[Type aliasing for model contracts](https://docs.getdbt.com/reference/resource-configs/contract.md):** dbt will use each adapter's built-in type aliasing for user-provided data types—meaning you can now write `string` always, and dbt will translate to `text` on Postgres/Redshift. This is "on" by default, but you can opt-out. * **[Raise warning for numeric types](https://docs.getdbt.com/reference/resource-configs/contract.md):** Because of issues when putting `numeric` in model contracts without considering that default values such as `numeric(38,0)` might round decimals accordingly. dbt will now warn you if it finds a numeric type without specified precision/scale. ##### dbt clean [dbt clean](https://docs.getdbt.com/reference/commands/clean.md) only cleans paths within the current working directory. The `--no-clean-project-files-only` flag will delete all paths specified in the `clean-targets` section of `dbt_project.yml`, even if they're outside the dbt project. Supported flags: * `--clean-project-files-only` (default) * `--no-clean-project-files-only` ##### Additional attributes in run\_results.json The run\_results.json now includes three attributes related to the `applied` state that complement `unique_id`: * `compiled`: Boolean entry of the node compilation status (`False` after parsing, but `True` after compiling). * `compiled_code`: Rendered string of the code that was compiled (empty after parsing, but full string after compiling). * `relation_name`: The fully-qualified name of the object that was (or will be) created/updated within the database. ##### Deprecated functionality The ability for installed packages to override built-in materializations without explicit opt-in from the user is being deprecated. * Overriding a built-in materialization from an installed package raises a deprecation warning. * Using a custom materialization from an installed package does not raise a deprecation warning. * Using a built-in materialization package override from the root project via a wrapping materialization is still supported. For example: ```text {% materialization view, default %} {{ return(my_cool_package.materialization_view_default()) }} {% endmaterialization %} ``` ##### Quick hits With these quick hits, you can now: * Configure a [`delimiter`](https://docs.getdbt.com/reference/resource-configs/delimiter.md) for a seed file. * Use packages with the same git repo and unique subdirectory. * Access the `date_spine` macro directly from dbt-core (moved over from dbt-utils). * Syntax for `DBT_ENV_SECRET_` has changed to `DBT_ENV_SECRET` and no longer requires the closing underscore. --- ### Upgrading to v1.8 Available in v1 #### Resources * [Changelog](https://github.com/dbt-labs/dbt/blob/1.8.latest/CHANGELOG.md) * [dbt v1 CLI Installation guide](https://docs.getdbt.com/docs/local/install-dbt.md) * [Cloud upgrade guide](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) #### What to know before upgrading dbt Labs is committed to providing backward compatibility for all versions 1.x, except for any changes explicitly mentioned on this page. If you encounter an error upon upgrading, please let us know by [opening an issue](https://github.com/dbt-labs/dbt/issues/new). #### Release tracks Starting in 2024, dbt provides the functionality from new versions of dbt v1 via [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) with automatic upgrades. Select a release track in your development, staging, and production [environments](https://docs.getdbt.com/docs/deploy/deploy-environments.md) to access everything in dbt v1.8+ and more. To upgrade an environment in the [dbt Admin API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md) or [Terraform](https://registry.terraform.io/providers/dbt-labs/dbtcloud/latest), set `dbt_version` to the string `latest`. #### New and changed features and functionality Features and functionality new in dbt v1.8. Snowflake column size change [Snowflake plans to increase](https://docs.snowflake.com/en/release-notes/bcr-bundles/un-bundled/bcr-2118) the default column size for string and binary data types in September 2026. `dbt-snowflake` versions below v1.10.6 may fail to build certain incremental models when this change is deployed.  Assess impact and required actions If you're using a `dbt-snowflake` version below v1.10.6 or have not yet migrated to a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) in the dbt platform, your adapter version is incompatible with this change and may fail to build incremental models that meet *both* of the following conditions: * Contain string columns with collation defined * Use the `on_schema_change='sync_all_columns'` config To check whether this change affects your project, run the following [list](https://docs.getdbt.com/reference/commands/list.md) command: ```bash dbt ls -s config.materialized:incremental,config.on_schema_change:sync_all_columns --resource-type model ``` * If the command returns `No nodes selected!`, no action is required. * If the command returns one or more models (for example, `Found 1000 models, 644 macros`), you may be impacted if those models have string columns that don't specify a width. In that case, upgrade to a version that includes the fix: * **dbt v1**: `dbt-snowflake` v1.10.6 or later. For upgrade instructions, refer to [Upgrade adapters](https://docs.getdbt.com/docs/local/install-dbt.md) in the dbt v1 installation instructions. * **dbt platform**: Any release track (**v1 Latest**, **v1 Compatible**, **v1 Extended**, or **v1 Fallback**). * **dbt v2**: v2.0.0. This ensures your incremental models can safely handle schema changes while maintaining required collation settings. ##### Unit Tests Historically, dbt's test coverage was confined to [“data” tests](https://docs.getdbt.com/docs/build/data-tests.md), assessing the quality of input data or resulting datasets' structure. In v1.8, we're introducing native support for [unit testing](https://docs.getdbt.com/docs/build/unit-tests.md). Unit tests validate your SQL modeling logic on a small set of static inputs **before** you materialize your full model in production. They support a test-driven development approach, improving both the efficiency of developers and the reliability of code. Starting from v1.8, when you execute the `dbt test` command, it will run both unit and data tests. Use the [`test_type`](https://docs.getdbt.com/reference/node-selection/methods.md#test_type) method to run only unit or data tests: ```shell dbt test --select "test_type:unit" # run all unit tests dbt test --select "test_type:data" # run all data tests ``` Unit tests are defined in YML files in your `models/` directory and are currently only supported on SQL models. To distinguish between the two, the `tests:` config has been renamed to `data_tests:`. Both are currently supported for backward compatibility. ###### New `data_tests:` syntax The `tests:` syntax is changing to reflect the addition of unit tests. Start migrating your [data test](https://docs.getdbt.com/docs/build/data-tests.md#new-data_tests-syntax) YML to use `data_tests:` after you upgrade to v1.8 to prevent issues in the future. ```yml models: - name: orders columns: - name: order_id data_tests: - unique - not_null ``` ###### The `--empty` flag The [`run`](https://docs.getdbt.com/reference/commands/run.md#the-%60--empty%60-flag) and [`build`](https://docs.getdbt.com/reference/commands/build.md#the---empty-flag) commands now support the `--empty` flag for building schema-only dry runs. The `--empty` flag limits the refs and sources to zero rows. dbt will still execute the model SQL against the target data warehouse but will avoid expensive reads of input data. This validates dependencies and ensures your models will build properly. ##### dbt-core and adapters are decoupled Before v1.8, dbt adapters directly depended on components of `dbt-core`, and `dbt-core` depended on the adapter for execution. This bidirectional dependency made it difficult to develop adapters independently. Starting in dbt v1.8, [`dbt-core` and adapters are decoupled](https://github.com/dbt-labs/dbt-adapters/discussions/87), making it easier to maintain and evolve them independent of each other. For backward compatibility, adapter packages continue to depend on `dbt-core` at install time. Running a dbt project still requires both `dbt-core` and an adapter, and since v1.0, many users have installed both together using [`pip install dbt-<adapter>`](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.0.md#installation>). This behavior remains unchanged. For example, `pip install dbt-snowflake` installs the latest versions of both `dbt-core` and `dbt-snowflake`. ##### Deprecated functionality The ability for installed packages to override built-in materializations without explicit opt-in from the user is being deprecated. * Overriding a built-in materialization from an installed package raises a deprecation warning. * Using a custom materialization from an installed package does not raise a deprecation warning. * Using a built-in materialization package override from the root project via a wrapping materialization is still supported. For example: ```sql {% materialization view, default %} {{ return(my_cool_package.materialization_view_default()) }} {% endmaterialization %} ``` ##### Managing changes to legacy behaviors dbt v1.8 has introduced flags for [managing changes to legacy behaviors](https://docs.getdbt.com/reference/global-configs/behavior-changes.md). You may opt into recently introduced changes (disabled by default), or opt out of mature changes (enabled by default), by setting `true` / `false` values, respectively, for `flags` in `dbt_project.yml`. You can read more about each of these behavior changes in the following links: * (Mature, enabled by default) [Require explicit package overrides for builtin materializations](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_explicit_package_overrides_for_builtin_materializations.md) * (Introduced, disabled by default) [Require resource names without spaces](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_resource_names_without_spaces.md) * (Introduced, disabled by default) [Run project hooks (`on-run-*`) in the `dbt source freshness` command](https://docs.getdbt.com/reference/global-configs/behavior-flags/source_freshness_run_project_hooks.md) #### Quick hits * Custom defaults of [global config flags](https://docs.getdbt.com/reference/global-configs/about-global-configs.md) should be set in the `flags` dictionary in [`dbt_project.yml`](https://docs.getdbt.com/reference/dbt_project.yml.md), instead of in [`profiles.yml`](https://docs.getdbt.com/docs/local/profiles.yml.md). Support for `profiles.yml` has been deprecated. * New CLI flag [`--resource-type`/`--exclude-resource-type`](https://docs.getdbt.com/reference/global-configs/resource-type.md) for including/excluding resources from dbt `build`, `run`, and `clone`. * To improve performance, dbt now issues a single (batch) query when calculating `source freshness` through metadata, instead of executing a query per source. * Syntax for `DBT_ENV_SECRET_` has changed to `DBT_ENV_SECRET` and no longer requires the closing underscore. * The `--select` flag now reliably limits `catalog.json` to the selected nodes in all projects, regardless of project size. In earlier versions, node selectors were ignored for projects with 100 or more nodes, and sources were omitted from `catalog.json` entirely when more than 100 nodes were selected. --- ### Upgrading to v1.9 Available in v1 #### Resources * [dbt 1.9 changelog](https://github.com/dbt-labs/dbt/blob/1.9.latest/CHANGELOG.md) * [dbt v1 CLI Installation guide](https://docs.getdbt.com/docs/local/install-dbt.md) * [dbt platform upgrade guide](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md#release-tracks) #### What to know before upgrading dbt Labs is committed to providing backward compatibility for all versions 1.x. Any behavior changes will be accompanied by a [behavior change flag](https://docs.getdbt.com/reference/global-configs/behavior-changes.md#behavior-change-flags) to provide a migration window for existing projects. If you encounter an error upon upgrading, please let us know by [opening an issue](https://github.com/dbt-labs/dbt/issues/new). Starting in 2024, dbt provides the functionality from new versions of dbt v1 via [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) with automatic upgrades. If you have selected the **v1 Latest** release track in dbt, you already have access to all the features, fixes, and other functionality that is included in dbt v1.9! If you have selected the **v1 Compatible** release track, you will have access in the next monthly **v1 Compatible** release after the dbt v1.9 final release. #### New and changed features and functionality Features and functionality new in dbt v1.9. ##### Microbatch `incremental_strategy` info If you use a custom microbatch macro, set the [`require_batched_execution_for_custom_microbatch_strategy`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_batched_execution_for_custom_microbatch_strategy.md) behavior flag in your `dbt_project.yml` to enable batched execution. If you don't have a custom microbatch macro, you don't need to set this flag as dbt will handle microbatching automatically for any model using the microbatch strategy. Incremental models are, and have always been, a *performance optimization* — for datasets that are too large to be dropped and recreated from scratch every time you do a `dbt run`. Learn more about [incremental models](https://docs.getdbt.com/docs/build/incremental-models-overview.md). Historically, managing incremental models involved several manual steps and responsibilities, including: * Add a snippet of dbt code (in an `is_incremental()` block) that uses the already-existing table (`this`) as a rough bookmark, so that only new data gets processed. * Pick one of the strategies for smushing old and new data together (`append`, `delete+insert`, or `merge`). * If anything goes wrong, or your schema changes, you can always "full-refresh", by running the same simple query that rebuilds the whole table from scratch. While this works for many use-cases, there’s a clear limitation with this approach: *Some datasets are just too big to fit into one query.* Starting in dbt 1.9, you can use the new [microbatch strategy](https://docs.getdbt.com/docs/build/incremental-microbatch.md#what-is-microbatch-in-dbt) to optimize your largest datasets -- **process your event data in discrete periods with their own SQL queries, rather than all at once.** The benefits include: * Simplified query design: Write your model query for a single batch of data. dbt will use your `event_time`, `lookback`, and `batch_size` configurations to automatically generate the necessary filters for you, making the process more streamlined and reducing the need for you to manage these details. * Independent batch processing: dbt automatically breaks down the data to load into smaller batches based on the specified `batch_size` and processes each batch independently, improving efficiency and reducing the risk of query timeouts. If some of your batches fail, you can use `dbt retry` to load only the failed batches. * Targeted reprocessing: To load a *specific* batch or batches, you can use the CLI arguments `--event-time-start` and `--event-time-end`. * [Automatic parallel batch execution](https://docs.getdbt.com/docs/build/parallel-batch-execution.md): Process multiple batches at the same time, instead of one after the other (sequentially) for faster processing of your microbatch models. dbt intelligently auto-detects if your batches can run in parallel, while also allowing you to manually override parallel execution with the [`concurrent_batches` config](https://docs.getdbt.com/reference/resource-properties/concurrent_batches.md). Currently microbatch is supported on these adapters with more to come: * postgres * redshift * snowflake * bigquery * spark * databricks ##### Snapshots improvements Beginning in dbt 1.9, we've streamlined snapshot configuration and added a handful of new configurations to make dbt **snapshots easier to configure, run, and customize.** These improvements include: * New snapshot specification: Snapshots can now be configured in a YAML file, which provides a cleaner and more consistent set up. * New `snapshot_meta_column_names` config: Allows you to customize the names of meta fields (for example, `dbt_valid_from`, `dbt_valid_to`, etc.) that dbt automatically adds to snapshots. This increases flexibility to tailor metadata to your needs. * `target_schema` is now optional for snapshots: When omitted, snapshots will use the schema defined for the current environment. * Standard `schema` and `database` configs supported: Snapshots will now be consistent with other dbt resource types. You can specify where environment-aware snapshots should be stored. * Warning for incorrect `updated_at` data type: To ensure data integrity, you'll see a warning if the `updated_at` field specified in the snapshot configuration is not the proper data type or timestamp. * Set a custom current indicator for the value of `dbt_valid_to`: Use the [`dbt_valid_to_current` config](https://docs.getdbt.com/reference/resource-configs/dbt_valid_to_current.md) to set a custom indicator for the value of `dbt_valid_to` in current snapshot records (like a future date). By default, this value is `NULL`. When configured, dbt will use the specified value instead of `NULL` for `dbt_valid_to` for current records in the snapshot table. * Use the [`hard_deletes`](https://docs.getdbt.com/reference/resource-configs/hard-deletes.md) configuration to get more control on how to handle deleted rows from the source. Supported methods are `ignore` (default), `invalidate` (replaces legacy `invalidate_hard_deletes=true`), and `new_record`. Setting `hard_deletes='new_record'` allows you to track hard deletes by adding a new record when row becomes "deleted" in source. Read more about [Snapshots meta fields](https://docs.getdbt.com/docs/build/snapshots.md#snapshot-meta-fields). To learn how to safely migrate existing snapshots, refer to [Snapshot configuration migration](https://docs.getdbt.com/reference/snapshot-configs.md#snapshot-configuration-migration) for more information. ##### Some `properties` moved to `configs` The following `properties` were moved to `configs` in [dbt v1.10](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.10.md) and backported to dbt v1.9: * [`freshness`](https://docs.getdbt.com/reference/resource-properties/freshness.md) for sources * [`meta`](https://docs.getdbt.com/reference/resource-configs/meta.md) under `columns` * [`tags`](https://docs.getdbt.com/reference/resource-configs/tags.md) under `columns` ##### `state:modified` improvements We’ve made improvements to `state:modified` behaviors to help reduce the risk of false positives and negatives. Read more about [the `state:modified` behavior flag](#managing-changes-to-legacy-behaviors) that unlocks this improvement: * Added environment-aware enhancements for environments where the logic purposefully differs (for example, materializing as a table in `prod` but a `view` in dev). ##### Managing changes to legacy behaviors dbt v1.9 has a handful of new flags for [managing changes to legacy behaviors](https://docs.getdbt.com/reference/global-configs/behavior-changes.md). You may opt into recently introduced changes (disabled by default), or opt out of mature changes (enabled by default), by setting `true` / `false` values, respectively, for `flags` in `dbt_project.yml`. You can read more about each of these behavior changes in the following links: * (Introduced, disabled by default) [`state_modified_compare_more_unrendered_values`](https://docs.getdbt.com/reference/global-configs/behavior-flags/state_modified_compare_more_unrendered_values.md). Set to `true` to persist `unrendered_config` during model parsing and `unrendered_database`/`unrendered_schema` during source parsing, then compare on unrendered values during `state:modified` checks. This reduces false positives from environment-aware logic, but requires rebuilding your state directory; without a rebuild, `state:modified` may select nodes with Jinja in YAML configs that haven't actually changed. * (Introduced, disabled by default) [`skip_nodes_if_on_run_start_fails` project config flag](https://docs.getdbt.com/reference/global-configs/behavior-flags/skip_nodes_if_on_run_start_fails.md). If the flag is set and **any** `on-run-start` hook fails, mark all selected nodes as skipped. * `on-run-start/end` hooks are **always** run, regardless of whether they passed or failed last time. - (Introduced, disabled by default) [`require_nested_cumulative_type_params`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_nested_cumulative_type_params.md). If the flag is set to `true`, users will receive an error instead of a warning if they're not properly formatting cumulative metrics using the new [`cumulative_type_params`](https://docs.getdbt.com/docs/build/cumulative.md#parameters) nesting. - (Introduced, disabled by default) [`require_batched_execution_for_custom_microbatch_strategy`](https://docs.getdbt.com/reference/global-configs/behavior-flags/require_batched_execution_for_custom_microbatch_strategy.md). Set to `true` if you use a custom microbatch macro to enable batched execution. If you don't have a custom microbatch macro, you don't need to set this flag as dbt will handle microbatching automatically for any model using the microbatch strategy. #### Adapter-specific features and functionalities Snowflake column size change [Snowflake plans to increase](https://docs.snowflake.com/en/release-notes/bcr-bundles/un-bundled/bcr-2118) the default column size for string and binary data types in September 2026. `dbt-snowflake` versions below v1.10.6 may fail to build certain incremental models when this change is deployed.  Assess impact and required actions If you're using a `dbt-snowflake` version below v1.10.6 or have not yet migrated to a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) in the dbt platform, your adapter version is incompatible with this change and may fail to build incremental models that meet *both* of the following conditions: * Contain string columns with collation defined * Use the `on_schema_change='sync_all_columns'` config To check whether this change affects your project, run the following [list](https://docs.getdbt.com/reference/commands/list.md) command: ```bash dbt ls -s config.materialized:incremental,config.on_schema_change:sync_all_columns --resource-type model ``` * If the command returns `No nodes selected!`, no action is required. * If the command returns one or more models (for example, `Found 1000 models, 644 macros`), you may be impacted if those models have string columns that don't specify a width. In that case, upgrade to a version that includes the fix: * **dbt v1**: `dbt-snowflake` v1.10.6 or later. For upgrade instructions, refer to [Upgrade adapters](https://docs.getdbt.com/docs/local/install-dbt.md) in the dbt v1 installation instructions. * **dbt platform**: Any release track (**v1 Latest**, **v1 Compatible**, **v1 Extended**, or **v1 Fallback**). * **dbt v2**: v2.0.0. This ensures your incremental models can safely handle schema changes while maintaining required collation settings. ##### Redshift * Support IAM Role auth ##### Snowflake * Iceberg Table Format — Support will be available on three out-of-the-box materializations: table, incremental, dynamic tables. * Breaking change — When upgrading from dbt 1.8 to 1.9 `{{ target.account }}` replaces underscores with dashes. For example, if the `target.account` is set to `sample_company`, then the compiled code now generates `sample-company`. [Refer to the `dbt-snowflake` issue](https://github.com/dbt-labs/dbt-snowflake/issues/1286) for more information. ##### Bigquery * Can cancel running queries on keyboard interrupt * Auto-drop intermediate tables created by incremental models to save resources ##### Spark * Support overriding the ODBC driver connection string which now enables you to provide custom connections #### Quick hits We also made some quality-of-life improvements in dbt 1.9, enabling you to: * Maintain data quality now that dbt returns an error (versioned models) or warning (unversioned models) when someone [removes a contracted model by deleting, renaming, or disabling](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md#how-are-breaking-changes-handled) it. * Document [data tests](https://docs.getdbt.com/reference/resource-properties/description.md). * Use `ref` and `source` in [foreign key constraints](https://docs.getdbt.com/reference/resource-properties/constraints.md). * Use `dbt test` with the `--resource-type` / `--exclude-resource-type` flag, making it possible to include or exclude data tests (`test`) or unit tests (`unit_test`). * The [`enabled`](https://docs.getdbt.com/reference/resource-configs/enabled.md) config is now available for unit tests. Defaults to `true` if not defined. --- ### Upgrading to v2 Beta ### Upgrading to v2 [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Available in v2 v2 is the current era of dbt, delivered through dbt v2. When you install dbt, you get dbt v2 by default. This guide walks you through upgrading a v1 project to v2. v2 is faster and stricter, but your existing project language and DAG semantics carry over, so once you upgrade, your project works as before — just faster. important dbt v2 is currently available for installation in: * [Local command line interface (CLI) tools](https://docs.getdbt.com/docs/local/install-dbt.md?version=2) [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") * [VS Code and Cursor with the dbt extension](https://docs.getdbt.com/docs/install-dbt-extension.md) [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") * [dbt platform environments](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md#dbt-v2) Join the conversation in our Community Slack channel [`#dbt-fusion-engine`](https://getdbt.slack.com/archives/C088YCAB6GH). #### More information about dbt v2 * [About the dbt extension](https://docs.getdbt.com/docs/about-dbt-extension.md) * [Supported features matrix](https://docs.getdbt.com/docs/dbt/supported-features.md) * [Install dbt](https://docs.getdbt.com/docs/local/install-dbt.md) * [Quickstart for dbt v2](https://docs.getdbt.com/guides/dbt.md?step=1) * [Upgrade guide](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md) * [dbt v2 license agreement](https://www.getdbt.com/dbt-fusion-engine-license-agreement) #### Resources * [dbt v2 changelog](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md) #### Install dbt Upgrading to v2 is an install step. Install dbt using `pip` to get dbt v2 for v2: ```shell python -m pip install --pre dbt ``` For full instructions, including Homebrew, winget, and additional options, refer to [Install dbt](https://docs.getdbt.com/docs/local/install-dbt.md). #### What to know before upgrading If you have an older project that isn't ready to move to v2, or you need compatibility with existing tooling, packages, or workflows that haven't moved to v2 yet, you can stay on dbt v1.x, which remains fully supported. Over time, new capabilities will land in v2 only, so most people will eventually want to upgrade. To install or continue using v1.x, refer to [Install dbt v1.x](https://docs.getdbt.com/docs/local/install-dbt.md?version=1.12). This new major version is an opportunity to *strengthen the framework* by removing deprecated functionality, rationalizing confusing behavior, and providing more rigorous validation on erroneous inputs. This means that there is some work involved in preparing an existing dbt project for v2. That work is documented below — it should be simple, straightforward, and in many cases, auto-fixable with the [`dbt-autofix`](https://github.com/dbt-labs/dbt-autofix) helper or the [agent skill](https://github.com/dbt-labs/dbt-agent-skills/tree/main/skills/dbt-migration/skills/migrating-dbt-core-to-fusion). Test v2 parser compatibility from dbt v1.12 If you're on dbt v1.12, you can test the rust parser compatibility before fully migrating by using the opt-in [`--use-v2-parser`](https://docs.getdbt.com/reference/global-configs/parsing.md#opt-in-v2-parser) flag. This delegates parsing to the v2 parser without changing any other behavior, making it a low-risk way to catch compatibility issues early. ###### Upgrade considerations Keep in mind the following considerations during the upgrade process: * **Manifest compatibility** — dbt v2 produces a `v12` [manifest](https://docs.getdbt.com/reference/artifacts/manifest-json.md) that's compatible with dbt v1. The only differences are optional dbt v2-specific fields that only dbt v2 writes, which dbt v1 safely ignores. As a result, you can run dbt v2 and dbt v1 side by side. State-dependent features such as `state:modified`, `--defer`, and cross-environment `dbt docs generate` work across mixed dbt v2 and dbt v1 environments, so you can migrate to dbt v2 incrementally without breaking existing dbt v1 jobs. State-aware orchestration is now dbt State [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) works with all engines and environments: dbt v1, the dbt platform, and dbt v2. If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). ##### Supported adapters The following adapters are supported in v2:  BigQuery[Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") * Service Account / User Token * Native OAuth * External OAuth * [Workload Identity Federation](https://docs.getdbt.com/docs/platform/manage-access/set-up-bigquery-oauth.md#set-up-bigquery-workload-identity-federation) (Microsoft Entra) * [Required permissions](https://docs.getdbt.com/docs/local/connect-data-platform/bigquery-setup.md#required-permissions)  Databricks[Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") * Service Account / User Token * Native OAuth  Redshift[Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") * Username / Password * IAM profile  Snowflake * Username / Password * Native OAuth * External OAuth * Key pair using a modern PKCS#8 method * MFA  Apache Spark (CLI only)[Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") * Thrift * Simple Authentication and Security Layer (SASL) PLAIN * No SASL (NOSASL) * Livy * Basic authentication (username and password) * When deployed on Amazon Web Services (AWS): AWS Signature Version 4 * Supports authentication using single sign-on, service accounts, or user tokens  DuckDB (CLI only)[Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") DuckDB does not require authentication — it runs locally on your machine. *Note that adapter lifecycle may differ between the dbt platform and local development. An adapter can reach GA in the dbt platform before it reaches GA for local use.* ##### A clean slate v2 will not support any deprecated functionality (see the [Changes overview](https://docs.getdbt.com/reference/changes-overview.md) for details): * All [deprecation warnings](https://docs.getdbt.com/reference/deprecations.md) must be resolved before upgrading to the new engine. This includes historic deprecations and [new ones as of dbt v1.10](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.10.md#deprecation-warnings). * Some [behavior change flags](https://docs.getdbt.com/reference/global-configs/behavior-changes.md#behavior-change-flags) will be removed (generally enabled). You can no longer opt out of them using `flags:` in your `dbt_project.yml`. ##### Ecosystem packages The most popular `dbt-labs` packages (`dbt_utils`, `audit_helper`, `dbt_external_tables`, `dbt_project_evaluator`) are already compatible with v2. External packages published by organizations outside of dbt may use outdated code or incompatible features that fail to parse in v2. We're working with those package maintainers to make packages available for v2. Packages requiring an upgrade to a new release for v2 compatibility, will be documented in this upgrade guide. #### New and changed features and functionality ##### `dbt login` In dbt v2, [`dbt login`](https://docs.getdbt.com/reference/commands/login.md?version=2.0) enables browser-based authentication. It opens a browser window prompting you to sign in to your dbt platform account or create a free account. Run [`dbt login status`](https://docs.getdbt.com/reference/commands/login.md?version=2.0#dbt-login-status) to view your current authentication status. `dbt login` unlocks a broader set of features, such as advanced features in the [dbt VS Code extension](https://docs.getdbt.com/docs/about-dbt-extension.md). For details, refer to [`dbt login`](https://docs.getdbt.com/reference/commands/login.md?version=2.0). ##### dbt Docs v2 v2 introduces [dbt Docs v2](https://docs.getdbt.com/docs/build/view-documentation.md#dbt-docs-v2), a faster, statically hostable documentation experience that replaces the v1 static site. `dbt docs generate` compiles your project, produces the v2 Parquet artifacts, and exports a static site in a single command. `dbt docs serve` previews that site locally, and because the browser queries those artifacts directly with DuckDB-WASM (WebAssembly), you can also host the generated files on any static file host. You only need `--write-index` if you want to produce the artifacts from a separate `dbt compile` or `dbt build` command. To hydrate catalog metadata (`catalog.json`) for Catalog without building the site, use the [`--write-catalog` flag](https://docs.getdbt.com/reference/commands/cmd-docs.md#--write-catalog-flag) instead. For full usage, refer to [About dbt docs commands](https://docs.getdbt.com/reference/commands/cmd-docs.md). ##### Local execution of unit tests [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") v2 introduces the [`compute`](https://docs.getdbt.com/reference/resource-configs/compute.md) config for unit tests. Set your unit tests with `compute: local` and dbt runs the test with DuckDB instead of sending it to your data platform, which takes the warehouse round trip out of your development loop. This config is opt-in. For details, refer to [Run unit tests locally](https://docs.getdbt.com/docs/build/unit-tests.md#run-unit-tests-locally). ##### Changed functionality When developing v2, there were opportunities to improve the dbt framework — failing earlier (when possible), fixing bugs, optimizing run order, and deprecating flags that are no longer relevant. The result is a handful of specific and nuanced changes to existing behavior. When upgrading to v2, you should expect the following changes in functionality: ###### Parse time printing of relations will print out the full qualified name, instead of an empty string In dbt v1, when printing the result of `get_relation()`, the parse time output for that Jinja would print `None` (the undefined object coerces to the string "None"). In v2, to help with intelligent batching of `get_relation()` calls (and significantly speed up `dbt compile`), dbt needs to construct a relation object with the fully qualified name resolved at parse time for the `get_relation()` adapter call. Constructing a relation object with the fully qualified name in v2 produces different behavior than v1 in `print()`, `log()`, or any Jinja macro that outputs to `stdout` or `stderr` at parse time. Example: ```jinja {% set relation = adapter.get_relation( database=db_name, schema=db_schema, identifier='a') %} {{ print('relation: ' ~ relation) }} {% set relation_via_api = api.Relation.create( database=db_name, schema=db_schema, identifier='a' ) %} {{ print('relation_via_api: ' ~ relation_via_api) }} ``` The output after `dbt parse` in dbt v1: ```text relation: None relation_via_api: my_db.my_schema.my_table ``` The output after `dbt parse` in v2: ```text relation: my_db.my_schema.my_table relation_via_api: my_db.my_schema.my_table ``` ###### Deprecated flags Deprecated flags are command-line flags (like `--models`, `--print`) that you pass to dbt commands. These are being removed in v2. This is different from: * [Deprecation warnings](https://docs.getdbt.com/reference/deprecations.md) — Features in your project code (models, YAML, macros) that need to be updated * [Behavior change flags](https://docs.getdbt.com/reference/global-configs/behavior-changes.md) — Flags in `dbt_project.yml` that let you opt in/out of new behaviors See the [Changes overview](https://docs.getdbt.com/reference/changes-overview.md) for a full comparison. Some historic CLI flags from v1 will no longer do anything in v2. If you pass them into a dbt command in v2, the command will not error, but the flag will do nothing (and warn accordingly). | flag name | remediation | | ------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------- | | `--models` / `--model` / `-m` | Refer to [CLI flags that need changes](#cli-flags-that-need-changes). | | `dbt seed` [`--show`](https://docs.getdbt.com/reference/commands/seed.md) | N/A | | [`--print` / `--no-print`](https://docs.getdbt.com/reference/global-configs/print-output.md) | No action required | | [`--printer-width`](https://docs.getdbt.com/reference/global-configs/print-output.md#printer-width) | No action required | | [`--source`](https://docs.getdbt.com/reference/commands/deps.md#non-hub-packages) | No action required | | [`--record-timing-info` / `-r`](https://docs.getdbt.com/reference/global-configs/record-timing-info.md) | No action required | | [`--cache-selected-only` / `--no-cache-selected-only`](https://docs.getdbt.com/reference/global-configs/cache.md) | No action required | | [`--clean-project-files-only` / `--no-clean-project-files-only`](https://docs.getdbt.com/reference/commands/clean.md#--clean-project-files-only) | No action required | | `--single-threaded` / `--no-single-threaded` | No action required | | `dbt source freshness` [`--output` / `-o`](https://docs.getdbt.com/docs/deploy/source-freshness.md) | | | [`--config-dir`](https://docs.getdbt.com/reference/commands/debug.md) | No action required | | [`--resource-type` / `--exclude-resource-type`](https://docs.getdbt.com/reference/global-configs/resource-type.md) | Refer to [CLI flags that need changes](#cli-flags-that-need-changes). | | `--show-resource-report` / `--no-show-resource-report` | No action required | | [`--log-cache-events` / `--no-log-cache-events`](https://docs.getdbt.com/reference/global-configs/logs.md#logging-relational-cache-events) | No action required | | `--use-experimental-parser` / `--no-use-experimental-parser` | No action required | | [`--empty-catalog`](https://docs.getdbt.com/reference/commands/cmd-docs.md#dbt-docs-generate) | | | [`--compile` / `--no-compile`](https://docs.getdbt.com/reference/commands/cmd-docs.md#dbt-docs-generate) | | | `--inline-direct` | No action required | | `--partial-parse-file-diff` / `--no-partial-parse-file-diff` | No action required | | `--partial-parse-file-path` | No action required | | `--populate-cache` / `--no-populate-cache` | No action required | | `--static-parser` / `--no-static-parser` | No action required | | `--use-fast-test-edges` / `--no-use-fast-test-edges` | No action required | | `--inject-ephemeral-ctes` / `--no-inject-ephemeral-ctes` | | | [`--partial-parse` / `--no-partial-parse`](https://docs.getdbt.com/reference/parsing.md#partial-parsing) | Refer to [CLI flags that need changes](#cli-flags-that-need-changes). | ###### CLI flags that need changes The following deprecated flags require updates in your job definitions or scripts: * **`--models` / `--model` / `-m`:** Use `--select` / `-s` instead (renamed in dbt v0.21). dbt raises an error in v2 if you use the old flags. Do not pass `--models` as the value to `-s` (for example, `dbt run -s --models`); v1 treated that as a model name, but v2 requires a valid selector. * **`--resource-type` / `--exclude-resource-type`:** Use `--resource-types` / `--exclude-resource-types`. For more information, see [Resource type flags](https://docs.getdbt.com/reference/global-configs/resource-type.md). dbt v2 job runs no longer support the `--partial-parse` and `--no-partial-parse` CLI flags. If you pass them (for example, from a dbt v1 command or script), dbt logs deprecation warning `dbt1700`. Remove these flags from your dbt v2 job commands. For more information, refer to [Deprecated flags](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md#deprecated-flags) in the guide to upgrading to dbt v2. ###### Conflicting package versions when a local package depends on a hub package which the root package also wants will error If a local package depends on a hub package that the root package also wants, `dbt deps` doesn't resolve conflicting versions in dbt v1; it will install whatever the root project requests. v2 will present an error: ```bash error: dbt8999: Cannot combine non-exact versions: =0.8.3 and =1.1.1 ``` ###### Parse will fail on nonexistent macro invocations and adapter methods When you call a nonexistent macro in dbt: ```sql select id as payment_id, # my_nonexistent_macro is a macro that DOES NOT EXIST {{ my_nonexistent_macro('amount') }} as amount_usd, from app_data.payments ``` Or a nonexistent adapter method: ```sql {{ adapter.does_not_exist() }} ``` In v1, `dbt parse` passes, but `dbt compile` fails. In v2, dbt will error out during `parse`. ###### Parse will fail on missing generic test When you have an undefined generic test in your project: ```yaml models: - name: dim_wizards data_tests: - does_not_exist ``` In v1, `dbt parse` passes, but `dbt compile` fails. In v2, dbt will error out during `parse`. ###### Parse will fail on missing variable When you have an undefined variable in your project: ```sql select {{ var('does_not_exist') }} as my_column ``` In v1, `dbt parse` passes, but `dbt compile` fails. In v2, dbt will error out during `parse`. ###### Stricter evaluation of duplicate docs blocks In v1, it was possible to create scenarios with duplicate [docs blocks](https://docs.getdbt.com/docs/build/documentation.md#using-docs-blocks). For example, you can have two packages with identical docs blocks referenced by an unqualified name in your dbt project. In this case, v1 would use whichever docs block is referenced without any warnings or errors. v2 adds stricter evaluation of names of docs blocks to prevent such ambiguity. It will present an error if it detects duplicate names: ```bash dbt found two docs with the same name: 'docs_block_title' in files: 'models/crm/_crm.md' and 'docs/crm/business_class_marketing.md' ``` To resolve this error, rename any duplicate docs blocks. ###### `dbt clean` will not delete any files in configured resource paths or files outside the project directory In dbt v1, `dbt clean` deletes: * Any files outside the project directory if `clean-targets` is configured with an absolute path or relative path containing `../`, though there is an opt-in config to disable this (`--clean-project-files-only` / `--no-clean-project-files-only`). * Any files in the `asset-paths` or `doc-paths` (even though other resource paths, like `model-paths` and `seed-paths`, are restricted). In v2, `dbt clean` will not delete any files in configured resource paths or files outside the project directory. ###### All unit tests are run first in `dbt build` In dbt v1, the direct parents of the model being unit tested needed to exist in the warehouse to retrieve the needed column name and type information. `dbt build` runs the unit tests (and their dependent models) *in lineage order*. In v2, `dbt build` runs *all* of the unit tests *first*, and then builds the rest of the DAG, due to built-in column name and type awareness. ###### Configuring `--threads` dbt v1 runs with `--threads 1` by default. You can increase this number to run more nodes in parallel on the remote data platform, up to the max parallelism enabled by the DAG. v2 handles threading differently depending on your data platform: | Adapter | Behavior | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Snowflake** | dbt v2 automatically manages connection parallelism based on platform limits and backpressure. The `threads` setting acts as a maximum connection cap if set, but dbt v2 is designed to work optimally without it configured. If you're experiencing timeout or rate limit issues, setting `threads` to a lower value can help. | | **Databricks** | dbt v2 automatically manages connection parallelism based on platform limits and backpressure. The `threads` setting acts as a maximum connection cap if set, but dbt v2 is designed to work optimally without it configured. If you're experiencing timeout or rate limit issues, setting `threads` to a lower value can help. | | **BigQuery** | dbt v2 respects user-set threads to manage API rate limits.<br />Setting `--threads 0` (or omitting the setting) allows dbt v2 to dynamically optimize parallelism. | | **Redshift** | dbt v2 respects user-set threads to manage concurrency limits.<br />Setting `--threads 0` (or omitting the setting) allows dbt v2 to dynamically optimize parallelism. | For more information, refer to [Using threads](https://docs.getdbt.com/docs/running-a-dbt-project/using-threads.md#dbt-v2-thread-optimization). ###### Continue to compile unrelated nodes after hitting a compile error As soon as v1's `compile` encounters an error compiling one of your models, dbt stops and doesn't compile anything else. When v2's `compile` encounters an error, it will skip nodes downstream of the one that failed to compile, but it will keep compiling the rest of the DAG (in parallel, up to the number of configured / optimal threads). ###### Seeds with extra commas don't result in extra columns In dbt v1, if you have an additional comma on your seed, dbt creates a seed with an additional empty column. For example, the following seed file (with an extra comma): ```text animal, dog, cat, bear, ``` Will produce this table when `dbt seed` is executed: | animal | b | | ------ | - | | dog | | | cat | | | bear | | In v2, it will not produce this extra column in the table resulting from `dbt seed`: | animal | | ------ | | dog | | cat | | bear | ###### Move standalone anchors under `anchors:` key As part of the ongoing process of making the dbt authoring language more precise, unexpected top-level keys in a YAML file will result in errors. A common use case behind these unexpected keys is standalone anchor definitions at the top level of a YAML file. You can use the new top-level `anchors:` key as a container for these reusable configuration blocks. For example, rather than using this configuration: models/\_models.yml ```yml # id_column is not a valid name for a top-level key in the dbt authoring spec, and will raise an error id_column: &id_column_alias name: id description: This is a unique identifier. data_type: int data_tests: - not_null - unique models: - name: my_first_model columns: - *id_column_alias - name: unrelated_column_a description: This column is not repeated in other models. - name: my_second_model columns: - *id_column_alias ``` Move the anchor under the `anchors:` key instead: models/\_models.yml ```yml anchors: - &id_column_alias name: id description: This is a unique identifier. data_type: int data_tests: - not_null - unique models: - name: my_first_model columns: - *id_column_alias - name: unrelated_column_a description: This column is not repeated in other models - name: my_second_model columns: - *id_column_alias ``` This move is only necessary for fragments defined outside of the main YAML structure. For more information about this new key, see [anchors](https://docs.getdbt.com/reference/resource-properties/anchors.md). ###### Self-referential (recursive) YAML anchors are not supported In v1, dbt could parse a YAML anchor that merges into an element of the same sequence it's defined on, creating a self-referential (cyclic) anchor. For example, anchoring a full `tables:` sequence and then merging that anchor into one of the sequence's own elements: ```yml sources: - name: catalogue tables: &tables - name: anchor_item description: The first table in the sequence. - <<: *tables name: merged_item ``` This parsed successfully in v1 only because PyYAML (the YAML library dbt v1 depends on) incidentally allows self-referential anchors, a side effect of Python's own support for cyclic data structures, not an intentional YAML feature. No other major YAML implementation allows this pattern. In v2, parsing this pattern hits a recursion limit and raises an error, so the entire properties file fails to parse. This is a deliberate limitation, not a bug. v2 does not plan to support self-referential anchors. To resolve this, remove the self-reference. Anchor only the parts of the document that don't merge back into themselves, for example, anchor a single table mapping instead of the whole sequence: ```yml sources: - name: catalogue tables: - &anchor_item_alias name: anchor_item description: The first table in the sequence. - <<: *anchor_item_alias name: merged_item ``` ###### Algebraic operations in Jinja macros In v1, you can set algebraic functions in the return function of a Jinja macro: ```jinja {% macro my_macro() %} return('xyz') + 'abc' {% endmacro %} ``` This is no longer supported in v2 and will emit a warning: ```bash [warning] [JinjaTopLevelReturn (dbt1508)]: return is not at the top level of the block. Its value is final and cannot be modified by surrounding expressions. Example: return(0) + 1. The + 1 is ignored and the macro returns 0. ``` This is not a common use case and there is no deprecation warning for this behavior in v1. The supported format is: ```jinja {% macro my_macro() %} return('xyzabc') {% endmacro %} ``` ##### Accessing custom configurations in meta `config.get()` and `config.require()` don't return values from the `meta` dictionary. If you try to access a key that only exists in `meta`, dbt emits a warning: ```bash warning: The key 'my_key' was not found using config.get('my_key'), but was detected as a custom config under 'meta'. Please use config.meta_get('my_key') or config.meta_require('my_key') instead. ``` Behavior when a key exists only in meta: | Method | Behavior | | -------------------------- | ---------------------------------------------- | | `config.get('my_key')` | Returns the default value and emits a warning. | | `config.require('my_key')` | Raises an error and emits a warning. | To access custom configurations stored under meta, use the explicit methods: ```jinja {% set owner = config.meta_get('owner') %} {% set has_pii = config.meta_require('pii') %} ``` For more information, see [config.meta\_get()](https://docs.getdbt.com/reference/dbt-jinja-functions/config.md#configmeta_get) and [config.meta\_require()](https://docs.getdbt.com/reference/dbt-jinja-functions/config.md#configmeta_require). ##### v2 compiler ###### Snowflake model functions v2 supports [Snowflake ML model functions](https://docs.snowflake.com/en/guides-overview-ml-functions), which allow you to call machine learning models directly in SQL. Because model function return types are flexible and defined by the underlying model, v2 uses simplified type checking: * **Arguments:** v2 accepts any arguments without strict type validation. * **Return type:** v2 treats all model function results as `VARIANT`. To use the result in your models, cast it to the expected type: ```sql select my_model!predict(input_column)::float as prediction_score from {{ ref('my_table') }} ``` ##### Package support To determine if a package is compatible with dbt v2, visit the [dbt package hub](https://hub.getdbt.com/) and look for the dbt v2-compatible badge, or review the package's [`require-dbt-version` configuration](https://docs.getdbt.com/reference/project-configs/require-dbt-version.md#pin-to-a-range). * Packages with a `require-dbt-version` that equals or contains `2.0.0` are compatible with dbt v2. For example, `require-dbt-version: ">=1.10.0,<3.0.0"`. Even if a package doesn't reflect compatibility in the package hub, it may still work with v2. Work with package maintainers to track updates, and [thoroughly test packages](https://docs.getdbt.com/guides/dbt-package-compat?step=5) that aren't clearly compatible before deploying. * Package maintainers who would like to make their package compatible with v2 can refer to the [dbt v2 package upgrade guide](https://docs.getdbt.com/guides/dbt-package-compat.md) for instructions. Fivetran package considerations: * The Fivetran `source` and `transformation` packages have been combined into a single package. * If you manually installed source packages like `fivetran/github_source`, you need to ensure `fivetran/github` is installed and deactivate the transformation models. ###### Package compatibility messages Inconsistent v2 warnings and `dbt-autofix` logs dbt v2 warnings and `dbt-autofix` logs may show different messages about package compatibility. If you use [`dbt-autofix`](https://github.com/dbt-labs/dbt-autofix) while upgrading to v2 in the Studio IDE or dbt VS Code extension, you may see different messages about package compatibility between `dbt-autofix` and v2 warnings. Here's why: * dbt v2 warnings are emitted based on a package's `require-dbt-version` and whether `require-dbt-version` contains `2.0.0`. * Some packages are already v2-compatible even though package maintainers haven't yet updated `require-dbt-version`. * `dbt-autofix` knows about these compatible packages and will not try to upgrade a package that it knows is already compatible. This means that even if you see a v2 warning for a package that `dbt-autofix` identifies as compatible, you don't need to change the package. The message discrepancy is temporary while we implement and roll out `dbt-autofix`'s enhanced compatibility detection to v2 warnings. Here's an example of a v2 warning in the Studio IDE that says a package isn't compatible with v2 but `dbt-autofix` indicates it is compatible: ```text dbt1065: Package 'dbt_utils' requires dbt version [>=1.30,<2.0.0], but current version is 2.0.0-preview.72. This package may not be compatible with your dbt version. dbt(1065) [Ln 1, Col 1] ``` #### Distributions v2 is available in two distributions. For more information, refer to [dbt licensing](https://docs.getdbt.com/docs/dbt-licensing.md). | Distribution | Package | Use it when | | ------------ | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------- | | dbt v2 | `dbt` | The recommended v2 experience. | | dbt OSS | `dbt-core` | Your organization has a strict requirement to use the Apache 2.0 [open-source runtime](https://docs.getdbt.com/docs/local/install-dbt-v2.md). | If you have a older project that isn’t ready to move to v2, continue using v1.x for compatibility. For new or upgraded projects, we recommend v2. --- ### Use dbt Wizard locally Beta ### Use dbt Wizard locally [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Local development Install dbt Wizard locally and start an agentic dbt development session from your terminal. You can run the dbt Wizard CLI locally from any project on any dbt engine. Be warned, the wizard has been known to cast spells . Wizard usage and billing From September 1st, 2026, dbt Wizard usage is metered per token against your account's usage credits. All credit amounts are per account, not per user. Enterprise and Enterprise+ accounts get monthly credits. Developer and Starter plans start with a 30-day trial and $100 in credits, as do CLI users via a free dbt account. Refer to [Trial and billing](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/trial-and-billing.md) for what each plan gets, spend limits, and paid access. #### Prerequisites * Access to a [supported AI provider](#supported-ai-providers). Use a dbt managed provider or configure [BYOK](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) with your own provider credentials. * A dbt project with a built `target/` directory (run `dbt parse`, `dbt compile`, or `dbt build`) dbt Wizard is data warehouse agnostic and works with both the [dbt v2](https://docs.getdbt.com/docs/introduction.md) and [dbt v1](https://docs.getdbt.com/docs/local/install-dbt.md) — no specific engine is required. #### Supported AI providers dbt Wizard supports [managed models](https://docs.getdbt.com/docs/dbt-ai/pricing-billing/overview.md#dbt-managed-providers) (billed by dbt Labs, no key to manage) and [bring-your-own-key (BYOK)](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) models (billed directly by your provider). Here are the following AI providers supported depending on where you work. Refer to [Model Provider Rate table](https://www.getdbt.com/legal/dbt-wizard-token-costs-by-model) for the full list of available models. ##### dbt platform | Provider | Access | | ---------------------------------------------------------------------------- | ------------------- | | [OpenAI](https://openai.com/policies/row-terms-of-use/) (default) | dbt managed or BYOK | | [Anthropic](https://www.anthropic.com/legal/consumer-terms)† | dbt managed or BYOK | | Open weight models (like DeepSeek, Kimi, and so on). | dbt managed | | [Azure AI Foundry](https://www.microsoft.com/licensing/terms) / Azure OpenAI | BYOK | ##### Locally (CLI) | Provider | Access | | ---------------------------------------------------------------------------- | ------------------- | | [OpenAI](https://openai.com/policies/row-terms-of-use/) | dbt managed or BYOK | | [Anthropic](https://www.anthropic.com/legal/consumer-terms)† | dbt managed or BYOK | | Open weight models (like DeepSeek, Kimi, and so on). | dbt managed | | [Azure AI Foundry](https://www.microsoft.com/licensing/terms) / Azure OpenAI | BYOK | | [AWS Bedrock](https://aws.amazon.com/service-terms/) | BYOK | | [Google Gemini](https://ai.google.dev/gemini-api/terms) | BYOK | | [Snowflake Cortex](https://www.snowflake.com/en/legal/terms-of-service/) | BYOK | | [Databricks Unity AI Gateway](https://www.databricks.com/legal/mcsa) | BYOK | You can also connect a personal OpenAI ChatGPT subscription instead of a key. †Anthropic enterprise and subscription licenses (such as Claude Enterprise) aren't supported per Anthropic's [terms of service](https://www.anthropic.com/legal/consumer-terms). #### Install and set up dbt Wizard 12345 View all stepsNext 1 Install the dbt Wizard CLI Run the install script for your operating system: macOS/Linux: ```bash curl -fsSL https://public.cdn.getdbt.com/dbt-wizard/install/install-wizard.sh | sh ``` Windows (PowerShell): ```powershell irm https://public.cdn.getdbt.com/dbt-wizard/install/install-wizard.ps1 | iex ``` This installs dbt Wizard to `/usr/local/bin/wizard`, along with the dbt [metadata engine](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md#native-metadata-engine) that powers dbt Wizard's project-aware answers. For install and update details, refer to [Install dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-cli.md); to remove them, refer to [Uninstall](https://docs.getdbt.com/docs/dbt-ai/wizard-cli.md#uninstall). Best practices for using dbt Wizard Once you're set up, refer to [How to use dbt Wizard in your dbt project](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-1-intro.md) for recommended workflows on real project tasks. #### Useful terminal commands Use the following commands to get started: | Command | Description | Example | | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------- | | `wizard "[prompt]"` | Start an interactive session seeded with a prompt. Once you activate the session, you don't need to pass your prompt in quotes. | `wizard "summarize what this project does"` | | `wizard exec "[prompt]"` | Run a single prompt non-interactively and exit | `wizard exec "list all models with no tests"` | | `wizard review --uncommitted` | Non-interactive code review of uncommitted changes | `wizard review --uncommitted` | | `wizard review --base BRANCH` | Review diff against a base branch | `wizard review --base main` | | `wizard resume` | Resume a previous session | `wizard resume --last` | | `wizard apply` | Apply the latest Wizard diff to your working directory | `wizard apply TASK_ID` | | `wizard login` / `logout` | Authenticate with your dbt account | `wizard login` | | `wizard mcp` | Manage MCP server connections | `wizard mcp add dbt` | | `wizard update` | Update Wizard to the latest version | `wizard update` | Need to re-run setup? If you want to re-run onboarding — re-authenticate, reset project config, or retrigger the trusted folder prompt — refer to [Re-trigger onboarding flows](https://docs.getdbt.com/docs/dbt-ai/wizard-config.md#re-trigger-onboarding-flows). #### Next steps * [Use cases and examples](https://docs.getdbt.com/docs/dbt-ai/wizard-use-cases.md) for realistic analytics engineering scenarios * [Install and update reference](https://docs.getdbt.com/docs/dbt-ai/wizard-cli.md) for full install, update, and uninstall details * [Configure BYOK](https://docs.getdbt.com/docs/dbt-ai/wizard-byok.md) for managing your API key and choosing an AI model * [How to use dbt Wizard in your dbt project](https://docs.getdbt.com/best-practices/how-to-use-wizard/wizard-1-intro.md) for recommended workflows * [Use skills locally](https://docs.getdbt.com/docs/dbt-ai/wizard-skills.md) for giving Wizard reusable instructions for your project * [Use MCP servers](https://docs.getdbt.com/docs/dbt-ai/wizard-mcp.md) to connect dbt Wizard CLI to more tools and context * [Migrate from Claude Code](https://docs.getdbt.com/docs/dbt-ai/wizard-migrate.md) for bringing existing Claude Code project context into dbt Wizard See it in action and share your feedback Want to see dbt Wizard in action? Check out the [demo video](https://www.youtube.com/watch?v=-lIzh1xQWMA). We'd love to hear how dbt Wizard is working for you. Share your feedback by either running the `/feedback` slash command in your interactive terminal session or by going to the [#dbt-wizard](https://getdbt.slack.com/archives/C0B6KLW6T26) channel in the [dbt Community Slack](https://docs.getdbt.com/community/join?version=2.0). Thanks so much for your help in improving dbt Wizard and dbt data development! --- ### Use MCP servers with dbt Wizard in the dbt platform dbt platform | Usage-based dbt Wizard in the dbt platform includes built-in dbt context. Custom MCP server connections are not yet supported in the platform experience. In the dbt platform, dbt Wizard can use dbt project context, dbt metadata, and built-in dbt documentation tooling without requiring you to configure an MCP server. Refer to [Available MCP tools](https://docs.getdbt.com/docs/dbt-ai/mcp-available-tools.md?version=2.0) for more info. Although custom MCP server connections aren't supported yet for dbt Wizard in the dbt platform, you can use the [dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-mcp.md) if you need custom MCP servers for development workflows. To discuss dbt platform MCP support, contact your dbt Labs account representative. #### Related docs * [dbt Wizard in Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) * [dbt Wizard home tab](https://docs.getdbt.com/docs/platform/wizard-home.md) * [Use MCP servers with dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-mcp.md) * [About dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md) --- ### Use MCP servers with the dbt Wizard CLI Beta ### Use MCP servers with the dbt Wizard CLI [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Local development The Model Context Protocol (MCP) connects dbt Wizard to external tools and context. Add an MCP server and dbt Wizard can call its tools mid-session — query the dbt MCP server for governed project metadata, open a pull request through the GitHub MCP server, or pull in any other MCP-compatible service. See it in action and share your feedback Want to see dbt Wizard in action? Check out the [demo video](https://www.youtube.com/watch?v=-lIzh1xQWMA). We'd love to hear how dbt Wizard is working for you. Share your feedback by either running the `/feedback` slash command in your interactive terminal session or by going to the [#dbt-wizard](https://getdbt.slack.com/archives/C0B6KLW6T26) channel in the [dbt Community Slack](https://docs.getdbt.com/community/join?version=2.0). Thanks so much for your help in improving dbt Wizard and dbt data development! For background on MCP itself, refer to the [Model Context Protocol introduction](https://modelcontextprotocol.io/introduction). For the dbt-maintained server specifically, refer to the [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md). #### Locations and precedence MCP servers are configured under `[mcp_servers.NAME]` in `config.toml`. Use user-level config for servers you want in every local project, and project-level config for servers that should travel with a trusted repo. The following table summarizes where dbt Wizard looks for MCP server configuration. Use the intended locations for new MCP servers, and keep compatibility locations only when migrating existing setup. Project-level MCP config loads only for trusted projects. Within project-level locations, the current working directory wins over parent directories. Project-level servers win over user-level servers with the same name. | Location | Level | Use for new MCP servers? | Precedence | | --------------------------- | ------- | ----------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | | `.dbt/wizard/config.toml` | Project | Yes. Use `[mcp_servers.NAME]` here for repo-shared MCP servers. | Highest project MCP config location; closer to the current working directory wins over parent directories. | | `~/.dbt/wizard/config.toml` | User | Yes. Use this for MCP servers you want across all local projects. The `wizard mcp add` command writes here. | Below project config, above compatibility imports. | | `.mcp.json` | Project | No. Compatibility import for existing MCP setup. | Imported with the project config for the same directory. | | `~/.dbt/wizard/.mcp.json` | User | No. Compatibility import for existing MCP setup. | Imported with user config; takes precedence over `~/.mcp.json` if both exist. | | `~/.mcp.json` | User | No. Compatibility import for existing MCP setup. | User-level fallback. | Avoid defining the same MCP server name in more than one location unless you intentionally want the higher-precedence location to override it. If a compatibility `.mcp.json` file and `config.toml` in the same directory define the same server name, remove the duplicate and keep the intended `config.toml` entry. #### Why use an MCP server dbt Wizard natively understands your dbt project. An MCP server extends that reach to the other tools and systems your work depends on, so you can do more without leaving your [session](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md#sessions). Each server you add gives dbt Wizard a new set of tools it can call on your behalf. For example: * dbt MCP server for governed access to your models, metrics, and lineage. * GitHub server to read and review pull requests. * Data warehouse server, or another server, to pull in context that lives outside dbt. The dbt Wizard CLI lets you add, remove, authenticate, and customize MCP servers, including per-tool approvals, through the `config.toml` file. MCP servers are a CLI feature You can configure MCP servers only in the dbt Wizard CLI. You can't add your own MCP servers in the dbt platform (Studio IDE and the home app), but dbt Wizard includes built-in dbt tools, such as [dbt Agent skills](https://github.com/dbt-labs/dbt-agent-skills) and product documentation fetching through the dbt MCP server. #### Supported MCP server types dbt Wizard supports two transports: | Type | Description | | ------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | | STDIO server (standard input/output, a server that runs as a program on your own computer, instead of one you connect to over the internet) | Runs as a local process that dbt Wizard starts with a command (for example, `npx` or `uvx`). Supports environment variables. | | Streamable HTTP server | A server you reach at a URL. Supports bearer-token and OAuth authentication. | Server instructions For either transport, dbt Wizard reads the `instructions` field the server returns during initialization and uses it as cross-tool guidance. #### Add an MCP server Use the `wizard mcp add` command, or edit `~/.dbt/wizard/config.toml` directly. Either one writes user-level `[mcp_servers.NAME]` configuration. ##### Add a STDIO server ```bash wizard mcp add SERVER_NAME --env VAR1=value1 -- COMMAND ARGS ``` Where: * `SERVER_NAME` is a name you choose for the server (for example, `filesystem`). * `--env VAR1=value1` is optional and repeatable, and applies only to STDIO servers. * Everything after `--` is the command dbt Wizard runs to launch the server, so the space after `--` is intentional. For example, add a filesystem MCP server that runs locally through `npx`. This server needs no environment variables, so omit `--env`: ```bash wizard mcp add filesystem -- npx -y @modelcontextprotocol/server-filesystem /Users/you/my-project ``` To connect the [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md), use the streamable HTTP form below — refer to [dbt MCP server](#dbt-mcp-server) under Examples. ##### Add a streamable HTTP server ```bash wizard mcp add SERVER_NAME --url https://example.com/mcp --bearer-token-env-var MY_TOKEN ``` To see all MCP subcommands, run `wizard mcp --help`. For the full list of flags, refer to the [CLI command reference](https://docs.getdbt.com/docs/dbt-ai/wizard-cli-reference.md). ##### Edit config.toml directly Instead of the `wizard mcp add` command, you can edit `config.toml` yourself. dbt Wizard stores user-level MCP configuration in `~/.dbt/wizard/config.toml` alongside its other settings: \~/.dbt/wizard/config.toml ```toml # STDIO server (runs locally) [mcp_servers.filesystem] command = "npx" args = ["-y", "@modelcontextprotocol/server-filesystem", "/Users/you/my-project"] # Streamable HTTP server [mcp_servers.github] url = "https://api.githubcopilot.com/mcp/" bearer_token_env_var = "GITHUB_MCP_TOKEN" http_headers = { "X-Region" = "us-east-1" } ``` Restart `wizard` after editing `config.toml` — MCP servers are loaded at session start. For how settings resolve, refer to [Config precedence](https://docs.getdbt.com/docs/dbt-ai/wizard-config.md#config-precedence). #### Configuration keys These keys can be set under an `[mcp_servers.NAME]` block in `config.toml`. | Key | Applies to | Description | | ----------------------------- | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | `command` | STDIO | The command that launches the server (for example, `uvx` or `npx`). | | `args` | STDIO | Array of arguments passed to `command`. | | `env` | STDIO | Table of environment variables set when launching the server. | | `env_vars` | STDIO | Names of existing environment variables to pass through to the server. | | `url` | HTTP | The server endpoint for a streamable HTTP server. | | `bearer_token_env_var` | HTTP | Name of the environment variable to read a bearer token from. Sent as `Authorization: Bearer TOKEN`. | | `http_headers` | HTTP | Table of static HTTP headers to send with each request. | | `env_http_headers` | HTTP | HTTP headers whose values are read from environment variables. | | `scopes` | HTTP | Array of OAuth scopes to request during login. Overrides the scopes the server advertises. | | `oauth.client_id` | HTTP | OAuth client identifier presented during authorization and token exchange. | | `oauth_resource` | HTTP | OAuth resource parameter to include in login requests ([RFC 8707](https://datatracker.ietf.org/doc/html/rfc8707)). | | `enabled` | Both | Whether the server is active. Defaults to `true`. | | `required` | Both | When `true`, `wizard exec` errors if this server fails to initialize. | | `enabled_tools` | Both | Allowlist of tool names to expose from the server. | | `disabled_tools` | Both | Blocklist of tool names to hide from the server. | | `default_tools_approval_mode` | Both | Default approval mode for this server's tools: `prompt` (ask before each call) or `auto`/`approve` (run without asking, which behave the same). | | `startup_timeout_sec` | Both | How long to wait for the server to start and list its tools. | | `tool_timeout_sec` | Both | How long to wait for an individual tool call. | Set per-tool approvals with a `[mcp_servers.NAME.tools.TOOL_NAME]` block and an `approval_mode` of `auto`, `prompt`, or `approve`: ```toml [mcp_servers.github.tools.create_pull_request] approval_mode = "approve" ``` #### Authenticate a server If a streamable HTTP server uses OAuth, you must authenticate from the CLI before dbt Wizard can use it. Run: ```bash wizard mcp login SERVER_NAME wizard mcp logout SERVER_NAME ``` To request specific scopes at login, pass the `--scopes` CLI flag with a comma-separated list. This requests the same scopes as the `scopes` key in `config.toml`, but only for that login: ```bash wizard mcp login SERVER_NAME --scopes read,write ``` For servers that use a static token, set `bearer_token_env_var` to the name of an environment variable holding the token, and export that variable before starting `wizard`. #### Manage MCP servers (CLI) Manage your configured servers through the dbt Wizard CLI, or by editing `config.toml` directly. The CLI provides these commands: | Command | What it does | | ------------------------- | ---------------------------------------------------------------------- | | `wizard mcp list` | List configured MCP servers. Add `--json` for machine-readable output. | | `wizard mcp get NAME` | Show the configuration for one server. | | `wizard mcp add NAME ...` | Add a STDIO or streamable HTTP server. | | `wizard mcp remove NAME` | Remove a server's configuration. | | `wizard mcp login NAME` | Authenticate with an OAuth server. | | `wizard mcp logout NAME` | Sign out of an OAuth server. | #### Approvals and tool permissions MCP tool calls follow the same [approval and sandboxing](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md#approval-and-sandboxing) rules as the rest of dbt Wizard. Set `enabled_tools` and `disabled_tools` in `config.toml` to control which tools a server exposes (there's no dedicated CLI flag). That way, dbt Wizard calls only the tools you intend. #### Examples The following examples show common scenarios for adding an MCP server and how to configure them. ##### dbt MCP server The [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md) gives dbt Wizard governed access to your project's models, metrics, lineage, freshness, and platform APIs. You can connect it two ways: ##### Self-hosted (no account required) Runs on your machine through `uvx` and works with or without a dbt platform account — the best fit for development: ```bash wizard mcp add dbt -- uvx dbt-mcp ``` The self-hosted server reads its connection settings (such as `DBT_HOST`, `DBT_TOKEN`, and `DBT_PROJECT_DIR`) from environment variables, typically a `.env` file in your dbt project root. You don't need a URL. For setup, refer to [Run self-hosted dbt](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-cli.md) and [Set up self-hosted MCP](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md). ##### Remote (dbt platform account) Hosted on the dbt platform. Build the URL from your platform host (`https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/`, for example `https://cloud.getdbt.com/api/ai/v1/mcp/`), then authenticate: ```bash wizard mcp add dbt --url https://YOUR_DBT_HOST_URL/api/ai/v1/mcp/ wizard mcp login dbt ``` For finding your host and token, refer to [Connect to the remote dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-remote.md) and [Connections and authentication (MCP)](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md#connections-and-authentication-mcp). Then prompt dbt Wizard: ```text Use the dbt MCP server to find the most recent failed run for the nightly job and summarize the error. ``` ##### GitHub MCP server for pull request review Connect a GitHub MCP server so dbt Wizard can read a pull request and post review comments: \~/.dbt/wizard/config.toml ```toml [mcp_servers.github] url = "https://api.githubcopilot.com/mcp/" # This is the NAME of an environment variable, not the token itself. # Keep your real token out of this file. bearer_token_env_var = "GITHUB_MCP_TOKEN" ``` Then set that environment variable to your actual token before starting dbt Wizard: ```bash export GITHUB_MCP_TOKEN="your-real-token-here" ``` At runtime, dbt Wizard reads the token from the environment and sends it as `Authorization: Bearer <the token>`. Store only the variable name in `config.toml` to keep the secret out of your committed config. ```text Review the dbt model changes in PR #482 — check for missing tests on new columns and confirm downstream refs still resolve. ``` #### Related docs * [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/about-mcp.md): The dbt-maintained server and its available tools * [Use subagents with dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-subagents.md): Delegate work to specialized agents * [dbt Wizard CLI config](https://docs.getdbt.com/docs/dbt-ai/wizard-config.md): `config.toml` keys and precedence * [dbt Wizard CLI command reference](https://docs.getdbt.com/docs/dbt-ai/wizard-cli-reference.md): `wizard mcp` flags and subcommands * [How dbt Wizard works](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md): Approvals and sandboxing --- ### Use skills with dbt Wizard CLI Beta ### Use skills with dbt Wizard CLI [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Local development Skills are reusable instructions that help dbt Wizard follow your team's SQL conventions, naming rules, modeling patterns, and workflows without repeating them in every prompt. dbt Wizard also includes built-in skills from [dbt Agent skills](https://github.com/dbt-labs/dbt-agent-skills), maintained by dbt Labs. These are always available, no setup needed. When you initialize dbt Wizard in a project, it can also detect existing Claude Code skills in the current directory. For general project context (`AGENTS.md`, `CLAUDE.md`), refer to [Migrate to dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-migrate.md) — dbt Wizard reads those instruction files out of the box. #### Locations and precedence At the start of every session, dbt Wizard scans for project-level skills in `.agents/skills/` from your dbt project root down to your current working directory: ```text .agents/ skills/ your-skill-name/ SKILL.md ``` The CLI also discovers user-level skills in `~/.agents/skills/`, user-level Claude Code skills under `~/.claude/skills/`, and project-level Claude Code skills under `.claude/skills/`. ```text .claude/ skills/ your-skill-name/ SKILL.md ``` Custom skills use the [Agent Skills](https://agentskills.io/specification) format, the same format used by [dbt Agent Skills](https://github.com/dbt-labs/dbt-agent-skills) on GitHub. A skill file contains YAML frontmatter (`name`, `description`) and a Markdown body with instructions. The following table summarizes where dbt Wizard looks for skills. Use the intended locations for new skills, and keep compatibility locations only when migrating existing setup. Within project-level locations, the current working directory wins over parent directories. Project-level skills win over user-level and built-in skills. | Location | Level | Use for new skills? | Precedence | | ------------------------------------------------------------------------- | -------- | ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------- | | `.agents/skills/NAME/SKILL.md` | Project | Yes. Use this for skills shared with a repo or subdirectory. | Highest project skill location; closer to the current working directory wins over parent directories. | | `~/.agents/skills/NAME/SKILL.md` | User | Yes. Use this for skills you want across all local projects. | Below project skills, above compatibility imports and built-in skills. | | `.claude/skills/NAME/SKILL.md` | Project | No. Compatibility import for existing Claude Code skills. | Below intended project and user skill locations. | | `~/.claude/skills/NAME/SKILL.md` | User | No. Compatibility import for existing Claude Code skills. | Below intended project and user skill locations. | | Built-in [dbt Agent Skills](https://github.com/dbt-labs/dbt-agent-skills) | Built-in | No. Shipped with dbt Wizard. | Lowest. | Avoid duplicate skill names across locations. If you need to replace a built-in or imported skill, create the replacement in an intended project or user location and remove or rename the older copy. #### Create a skill From your terminal, use the following commands to create a skill folder and file: ```bash mkdir -p .agents/skills/my-team-style touch .agents/skills/my-team-style/SKILL.md ``` Then write your skill file. Refer to [Skill file format](#skill-file-format) in the next section for more information. Start a new `wizard` session to apply the changes. Skills are discovered at session start and changes made mid-session won't be picked up until you restart. #### Skill file format A skill file has two required parts: YAML frontmatter and a Markdown body. * **Frontmatter**: Includes at minimum, `name` and `description`. The `description` is especially important: it's how dbt Wizard decides when to apply the skill automatically. * **Body**: Markdown sections explaining when to use the skill, what it does, and any conventions or constraints. * **References** (optional): Supporting `.md` files in a `references/` subfolder for detail that would make `SKILL.md` too long. ##### Example skill file If you're new to skills, start with a small `SKILL.md` like the following example, then grow it over time: ```markdown --- name: my-team-style description: Apply Santi Corp's modeling conventions when editing or creating dbt models. Use when the user asks for refactors, new models, or YAML in this project. --- # My team style ## When to use Use this skill whenever you are changing SQL or YAML under `models/` and the user did not override these rules. ## Conventions - Staging models use prefix `stg_` and live in `models/staging/`. - Facts and dimensions use `fct_` and `dim_` prefixes respectively. - Document new columns in the same PR as the model change. ## Optional detail For edge cases, read `references/naming-conventions.md` in this skill folder before proposing renames. ``` For a full production-style example, refer to dbt's [`adding-dbt-unit-test` skill](https://github.com/dbt-labs/dbt-agent-skills/blob/main/skills/dbt/skills/adding-dbt-unit-test/SKILL.md). #### Folder layout ```text .agents/ skills/ my-team-style/ SKILL.md ← required references/ ← optional: extra detail Wizard loads when needed naming-conventions.md sql-patterns.md ``` The `references/` subfolder is for supporting material that would make `SKILL.md` too long. dbt Wizard reads reference files when the skill is active and the task is relevant. #### Use a skill dbt Wizard applies skills automatically when it detects a match between the skill's `description` and your task. You can also invoke a skill explicitly: Reference the skill by name in your prompt: ```text Use my-team-style to refactor fct_orders. ``` Or point to the file directly with `@`: ```text @.agents/skills/my-team-style/SKILL.md — apply these conventions to the new model. ``` #### Global skills (CLI only) If you use the CLI, you can store skills in `~/.agents/skills/` to make them available across all your dbt projects. This is useful for personal conventions you want everywhere: \~/.agents/skills/ ```text personal-defaults/ SKILL.md ``` If a project skill and a global skill use the same name, the project skill takes precedence. #### Built-in dbt skills dbt Wizard ships with skills from [dbt Agent Skills](https://github.com/dbt-labs/dbt-agent-skills), maintained by dbt Labs and the community. These capture analytics engineering knowledge for common workflows and are always available. The agent loads the relevant skill automatically when your prompt matches its use case. The following skills all ship bundled with dbt Wizard. You don't call them directly as the agent loads whichever one best matches your prompt. Refer to the [dbt Agent Skills repository](https://github.com/dbt-labs/dbt-agent-skills) for installation in other AI clients, contributing new skills, and the latest catalog. | Skill | What it does | | ----------------------------------------------- | ---------------------------------------------------------------------------- | | `using-dbt-for-analytics-engineering` | Build and modify dbt models, debug errors, explore data sources, write tests | | `adding-dbt-unit-test` | Add unit tests for dbt models; practice test-driven development | | `building-dbt-semantic-layer` | Create semantic models, metrics, and dimensions with MetricFlow | | `answering-natural-language-questions-with-dbt` | Answer business questions by querying the semantic layer | | `working-with-dbt-mesh` | Implement dbt Mesh governance and cross-project collaboration | | `troubleshooting-dbt-job-errors` | Diagnose and resolve dbt platform job failures | | `configuring-dbt-mcp-server` | Set up the dbt MCP server for Claude, Cursor, or VS Code | | `fetching-dbt-docs` | Look up dbt documentation efficiently | | `running-dbt-commands` | Run dbt CLI commands with correct flags, selectors, and parameter formats | ##### Migration (one-off use) These skills are for migration projects rather than everyday sessions: | Skill | What it does | | ---------------------------------------- | ------------------------------------------ | | `migrating-dbt-core-to-fusion` | Migrate dbt projects from dbt v1 to dbt v2 | | `migrating-dbt-project-across-platforms` | Migrate dbt projects across data platforms | Built-in skills are updated with each dbt Wizard release. Custom skills with the same name take precedence. #### Tips * **Keep `SKILL.md` focused.** One skill per concern (style guide, testing conventions, deployment workflow). Smaller skills are loaded more reliably than large monolithic ones. * **Use `description` to control when dbt Wizard activates the skill.** A precise description ("Use when creating or editing models in models/marts/") means the skill fires when relevant, not on every prompt. * **Start a new session after adding or editing a skill.** Skills are discovered at session start — mid-session changes aren't picked up until you start a new chat. * **Cross-project sharing isn't supported yet.** To reuse a skill in another repo, copy the skill files manually. CLI users can also use `~/.agents/skills/` for skills they want everywhere. #### Related docs * [dbt Wizard in Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) * [Use skills in the dbt platform](https://docs.getdbt.com/docs/dbt-ai/wizard-platform-skills.md) * [How dbt Wizard works](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md) * [dbt Wizard CLI overview](https://docs.getdbt.com/docs/dbt-ai/wizard-cli.md) --- ### Use skills with dbt Wizard in the dbt platform Beta ### Use skills with dbt Wizard in the dbt platform [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt platform | Usage-based Skills are reusable instructions that help dbt Wizard follow your team's SQL conventions, naming rules, modeling patterns, and workflows without repeating them in every prompt. dbt Wizard also includes built-in skills from [dbt Agent skills](https://github.com/dbt-labs/dbt-agent-skills), maintained by dbt Labs. These are always available, no setup needed. For general project context (`AGENTS.md`, `CLAUDE.md`), refer to [Migrate to dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-migrate.md) — dbt Wizard reads those instruction files out of the box. #### How skills work At the start of every session, dbt Wizard scans for repo-level skills in the `.agents/skills/` directory in your dbt project root: ```text .agents/ skills/ your-skill-name/ SKILL.md ``` Custom skills use the [Agent Skills](https://agentskills.io/specification) format, the same format used by [dbt Agent Skills](https://github.com/dbt-labs/dbt-agent-skills) on GitHub. A skill file contains YAML frontmatter (`name`, `description`) and a Markdown body with instructions. If a custom skill and a built-in skill share the same name, the custom skill takes precedence. #### Create a skill In [Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md), use the file explorer to create the skill directory and file: 1. Create a folder at `.agents/skills/SKILL_NAME/` in your project root. 2. Add a `SKILL.md` file inside it. A typical `SKILL.md` includes: * YAML frontmatter with at minimum `name` and `description`. * A Markdown body with sections for when to use the skill, workflow steps, and conventions. * Optional `references/` files for extra detail the agent can load when needed. 3. Optionally add supporting files under `.agents/skills/SKILL_NAME/references/` (for example, `references/naming-conventions.md`) that the agent can read when the skill is active. 4. Start a new dbt Wizard chat — skills are discovered at session start, so mid-session changes won't be picked up until you start a new chat. #### Skill file format A skill file has two required parts: YAML frontmatter and a Markdown body. * **Frontmatter**: Includes at minimum, `name` and `description`. The `description` is especially important: it's how dbt Wizard decides when to apply the skill automatically. * **Body**: Markdown sections explaining when to use the skill, what it does, and any conventions or constraints. * **References** (optional): Supporting `.md` files in a `references/` subfolder for detail that would make `SKILL.md` too long. ##### Example skill file If you're new to skills, start with a small `SKILL.md` like the following example, then grow it over time: ```markdown --- name: my-team-style description: Apply Santi Corp's modeling conventions when editing or creating dbt models. Use when the user asks for refactors, new models, or YAML in this project. --- # My team style ## When to use Use this skill whenever you are changing SQL or YAML under `models/` and the user did not override these rules. ## Conventions - Staging models use prefix `stg_` and live in `models/staging/`. - Facts and dimensions use `fct_` and `dim_` prefixes respectively. - Document new columns in the same PR as the model change. ``` For a full production-style example, refer to dbt's [`adding-dbt-unit-test` skill](https://github.com/dbt-labs/dbt-agent-skills/blob/main/skills/dbt/skills/adding-dbt-unit-test/SKILL.md). #### Use a skill In the dbt Wizard chat panel: 1. Prompt the agent and reference the skill by name — for example: `Use my-team-style to refactor this model and update related YAML`. 2. If needed, use `@` mentions to point to the skill file or supporting `.md` files directly. 3. Review and approve proposed changes as usual in the dbt Wizard panel. #### Built-in dbt skills dbt Wizard ships with skills from [dbt Agent Skills](https://github.com/dbt-labs/dbt-agent-skills), maintained by dbt Labs and the community. These capture analytics engineering knowledge for common workflows and all ship bundled with dbt Wizard. You don't call them directly as the agent loads whichever one matches your prompt. For the latest catalog, refer to the [dbt Agent Skills repository](https://github.com/dbt-labs/dbt-agent-skills). #### Tips * **Keep `SKILL.md` focused.** One skill per concern (style guide, testing conventions, deployment workflow). Smaller skills are loaded more reliably than large monolithic ones. * **Use `description` to control when dbt Wizard activates the skill.** A precise description ("Use when creating or editing models in models/marts/") means the skill fires when relevant, not on every prompt. * **Start a new session after adding or editing a skill.** Skills are discovered at session start — mid-session changes aren't picked up until you start a new chat. * **Cross-project sharing isn't supported yet.** To reuse a skill in another repo, copy the skill files manually. #### Related docs * [dbt Wizard in Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) * [Use skills locally](https://docs.getdbt.com/docs/dbt-ai/wizard-skills.md) * [How dbt Wizard works](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md) --- ### Use subagents with dbt Wizard CLI Beta ### Use subagents with dbt Wizard CLI [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Local development Subagents let dbt Wizard spin up focused, parallel agents to handle parts of a larger task — one to explore your project, one to make changes, one to review them. dbt Wizard orchestrates them and consolidates the results back into your session. Use subagents when you can split a task into independent pieces of work. They're useful for reviewing a large pull request, debugging a failed job, adding tests across multiple models, or researching documentation while another agent inspects your project. For smaller, direct tasks, you usually don't need subagents. Ask a single question, like "What does this model do?" or "Fix this failing test," without splitting the work across agents. Subagents run work in parallel, which uses more tokens than handling the same task in a single-agent session. Use them when the task benefits from dividing the work. See it in action and share your feedback Want to see dbt Wizard in action? Check out the [demo video](https://www.youtube.com/watch?v=-lIzh1xQWMA). We'd love to hear how dbt Wizard is working for you. Share your feedback by either running the `/feedback` slash command in your interactive terminal session or by going to the [#dbt-wizard](https://getdbt.slack.com/archives/C0B6KLW6T26) channel in the [dbt Community Slack](https://docs.getdbt.com/community/join?version=2.0). Thanks so much for your help in improving dbt Wizard and dbt data development! #### Locations and precedence You can define custom agent roles by adding standalone TOML files at the project level or user level. Use project-level agents for roles that should travel with a repo, and user-level agents for roles you want across all local projects. The following table summarizes where dbt Wizard looks for custom agents. Use the intended locations for new agents, and keep compatibility locations only when migrating existing setup. Project-level agent config loads only for trusted projects. Within project-level locations, the current working directory wins over parent directories. Project-level agents win over user-level agents with the same name. | Location | Level | Use for new agents? | Precedence | | -------------------------------- | ------- | ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------- | | `.dbt/wizard/agents/NAME.toml` | Project | Yes. Use this for agents shared with a repo or subdirectory. | Highest project agent location; closer to the current working directory wins over parent directories. | | `~/.dbt/wizard/agents/NAME.toml` | User | Yes. Use this for agents you want across all local projects. | Below project agents, above compatibility imports. | | `.claude/agents/NAME.md` | Project | No. Compatibility import for existing Claude Code agents. | Below `.dbt/wizard/agents/` in the same project location. | | `~/.claude/agents/NAME.md` | User | No. Compatibility import for existing Claude Code agents. | Below `~/.dbt/wizard/agents/`. | If two custom agents use the same name, dbt Wizard uses the higher-precedence location and fills in any missing fields from lower-precedence locations when possible. Avoid duplicate names unless you intentionally want to override an existing role. #### Where you can use subagents Subagents work in the [dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-cli.md). You can define custom agent roles, set display nicknames, and configure global limits through the `config.toml` file. #### How subagents work An agent is a role that describes a type of work, like `explorer`, `worker`, or `test_writer`. A subagent is a running instance of one of those roles. For example, if you start two explorer agents to answer two different questions, you have two subagents that share the same agent role. dbt Wizard handles orchestration for you. It starts subagents, routes work to them, waits for their results, and consolidates their output into your session. You can use subagents in the following ways: | How subagents start | Example | | ---------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | You ask to split up the work. | `Review PR #123. Use one agent to map what changed, one to check tests and downstream impact, and one to look up the relevant dbt docs.` | | dbt Wizard automatically uses a configured agent because your task matches that agent's description. | `Add tests for stg_customers and check whether similar staging models are missing tests.` | dbt Wizard then spawns the agents, lets them work in parallel, and brings their findings back into one response. #### Built-in agents dbt Wizard ships with built-in agents that it can spawn without any configuration. Several are purpose-built for dbt work: | Agent | What it's for | Example | | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | | `explorer` | Answers specific, well-scoped questions about your project. Fast and read-only — spawn several in parallel for independent questions. | `Use explorer to explain what depends on fct_orders.` | | `worker` | Performs execution and production work, such as implementing part of a feature, fixing tests or bugs, or splitting a large refactor into independent chunks. | `Use worker to update these staging models to follow our naming convention.` | | `validation` | Provides dbt validation. After model edits, runs structured validation — SQL check, `dbt run` with `--defer`, prod vs. dev comparison, and impact analysis — to validate changes before you merge. | `Use validation to check whether my changes to int_payments are safe to merge.` | | `test_writer` | Improves dbt test coverage. Analyzes project metadata and warehouse data to find coverage gaps, validates assumptions with queries, and writes `schema.yml` tests for models with low or no coverage. | `Use test_writer to add tests to stg_customers.` | You don't need to declare these — dbt Wizard routes to them automatically when a task fits, or you can ask for one by name. For example, both of these prompts can use the `test_writer` agent: | Prompt style | Example | | ------------------------- | ------------------------------------------------ | | Ask naturally | Add useful tests for stg\_customers. | | Ask for the agent by name | Use test\_writer to add tests to stg\_customers. | To add your own roles, refer to [Custom agents](#custom-agents-cli). #### Manage subagents Use slash commands inside an interactive session to inspect and steer agent threads: | Command | What it does | Example | | ------------ | ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | | `/agent` | Switch the active agent thread to inspect or steer ongoing work. | Use `/agent` when you want to jump from the main session into a specific agent thread to give it more direction. | | `/subagents` | View and switch between running subagent threads. | Use `/subagents` to see which helper agents are active during a delegated task, then select one to inspect its progress. | You can also tell dbt Wizard in plain language to steer, stop, or close an agent thread. For example, `Stop the docs researcher agent and continue with the reviewer findings.` For the full list of session commands, refer to the [slash command reference](https://docs.getdbt.com/docs/dbt-ai/wizard-slash-commands.md). #### Approvals and sandbox Subagents inherit the parent session's [approval and sandbox policy](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md#approval-and-sandboxing). Any runtime overrides you set for a turn (permissions, sandbox mode) apply to the subagents it spawns. When a subagent requests approval, your session shows the request with its source so you know which agent is asking. A custom agent can override sandbox settings for itself — useful when, for example, an exploration agent should stay read-only while a build agent needs workspace write access. #### Custom agents A custom agent role is a reusable role for a particular type of work you want dbt Wizard to perform, such as writing UDFs, exploring a project, or debugging an issue. You can create any role name that fits your workflow; it does not have to be one of the built-in roles. Each file defines one custom agent role. The file name must match the agent's `name` field. For example, an agent with `name = "udf_helper"` must be defined in an `agents/udf_helper.toml` file. dbt Wizard loads each custom agent file as a configuration layer for spawned agent sessions. This means a custom agent can override the same settings as a normal dbt Wizard session config, such as the model, instructions, sandbox mode, and MCP servers. The best custom agents are narrow and opinionated. Give each one a clear job, the tools it needs for that job, and instructions that keep it from drifting into adjacent work. Every custom agent file must define: | Field | Required | Description | | ------------------------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | Yes | Agent name dbt Wizard uses when spawning or referring to this agent. Must match the file name without `.toml`. For example, `name = "udf_helper"` must be in `udf_helper.toml`. | | `description` | Yes | Explains when to use the agent. dbt Wizard reads this description to decide whether the agent fits a task. | | `developer_instructions` | Yes | Self-hosted dbt instructions that define the agent's behavior. | | `nickname_candidates` | No | Display-only labels for spawned instances of this agent in the UI, such as `UDF helper` or `UDF queen`. The nickname does not identify the agent. | | `model` | No | Model this agent should use. Inherits from the parent session when omitted. | | `sandbox_mode` | No | Sandbox mode for this agent. Inherits from the parent session when omitted. | | `mcp_servers` | No | MCP servers available to this agent. Inherits from the parent session when omitted. | You can also include other supported [`config.toml`](https://docs.getdbt.com/docs/dbt-ai/wizard-config.md#configtoml) keys in a custom agent file. Any setting you don't define in the custom agent file inherits from the parent session. ##### Example custom agent Create a standalone custom agent file. The file name and `name` value must match: \~/.dbt/wizard/agents/udf\_helper.toml ```toml name = "udf_helper" # matches the udf_helper.toml file name description = "Helps design and implement dbt UDFs and models that use them." developer_instructions = """ You are an expert in dbt UDFs. Focus on: - choosing the right UDF pattern - writing clear function definitions - configuring function paths - using {{ function() }} references correctly - adding focused models and tests that demonstrate the UDF - keeping changes small and additive Before editing files, explain the proposed UDF design. When editing files, avoid unrelated changes. """ sandbox_mode = "workspace-write" nickname_candidates = ["UDF helper", "UDF queen"] ``` Then ask dbt Wizard to use the agent by name: ```text Use udf_helper to create a UDF model that checks whether customer IDs are positive integers. ``` You can also let dbt Wizard choose the agent automatically by describing the task: ```text Create a dbt UDF that checks whether customer IDs are positive integers, then add a small model that uses it. ``` ##### Display nicknames When several instances of the same agent run at once, use `nickname_candidates` to give each instance a readable label in the UI. For example, two spawned `udf_helper` agents might appear as `UDF helper` and `UDF queen`. Nicknames are display-only. dbt Wizard identifies the agent by its `name` field, not by the nickname shown in the UI. ##### Example with an MCP server Custom agent files can include MCP server configuration when the agent needs additional tools or context. Again, the file name and `name` value must match: \~/.dbt/wizard/agents/dbt\_docs\_helper.toml ```toml name = "dbt_docs_helper" description = "Looks up relevant dbt documentation for patterns referenced in a change." developer_instructions = """ Use the dbt docs MCP server to confirm APIs, options, and version-specific behavior. Return concise answers with links or exact references when available. Do not edit files. """ sandbox_mode = "read-only" [mcp_servers.dbt] command = "uvx" args = ["dbt-mcp"] ``` This setup works well for prompts like: ```text Have udf_helper design the UDF change and dbt_docs_helper verify the dbt function configuration. ``` ##### Global settings Global subagent settings live under the `[agents]` section in `~/.dbt/wizard/config.toml`: \~/.dbt/wizard/config.toml ```toml [agents] max_depth = 2 job_max_runtime_seconds = 1800 interrupt_message = true ``` | Key | Description | | ------------------------- | ------------------------------------------------------------------------------------------------ | | `max_depth` | Maximum nesting depth for spawned agent threads. Root sessions start at depth 0. | | `job_max_runtime_seconds` | Default maximum runtime, in seconds, for agent job workers. | | `interrupt_message` | Whether to record a model-visible message when an agent turn is interrupted. Defaults to `true`. | #### Examples ##### UDF implementation Define a custom agent that specializes in dbt UDF work. Each custom agent is a standalone TOML file under `~/.dbt/wizard/agents/`. \~/.dbt/wizard/agents/udf\_helper.toml ```toml name = "udf_helper" description = "Helps design and implement dbt UDFs and models that use them." developer_instructions = """ You are an expert in dbt UDFs. Focus on: - choosing the right UDF pattern - writing clear function definitions - configuring function paths - using {{ function() }} references correctly - adding focused models and tests that demonstrate the UDF - keeping changes small and additive Before editing files, explain the proposed UDF design. When editing files, avoid unrelated changes. """ sandbox_mode = "workspace-write" nickname_candidates = ["UDF helper", "UDF queen"] ``` Optionally, configure global subagent settings in your main `config.toml`: \~/.dbt/wizard/config.toml ```toml [agents] max_depth = 2 ``` Then prompt dbt Wizard: ```text Use udf_helper to create a UDF that checks whether customer IDs are positive integers, then add a small model that uses it. ``` dbt Wizard starts the requested agent, lets it work on the UDF implementation, and consolidates the result into your session. ##### Debug a failed job run (home app) In the dbt platform home app, ask dbt Wizard to investigate a failed job by delegating to focused agents: ```text The nightly job failed. Use one agent to pull the run error and logs, one to trace the failing model's lineage and find the root cause, and one to propose a fix. Summarize what each found. ``` dbt Wizard spawns the agents, each one works on its part of your connected project, and dbt Wizard consolidates the diagnosis and proposed fix into one response. #### Related docs * [Use MCP servers with the dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-mcp.md) to give agents more tools and context * [Use skills](https://docs.getdbt.com/docs/dbt-ai/wizard-skills.md) for reusable, project-specific instructions * [dbt Wizard slash commands](https://docs.getdbt.com/docs/dbt-ai/wizard-slash-commands.md) for `/agent` and `/subagents` * [dbt Wizard CLI config](https://docs.getdbt.com/docs/dbt-ai/wizard-config.md) for `config.toml` keys and precedence * [How dbt Wizard works](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md) for approvals and sandboxing --- ### Use subagents with dbt Wizard in the dbt platform Beta ### Use subagents with dbt Wizard in the dbt platform [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt platform | Usage-based Subagents let dbt Wizard spin up focused agents to handle parts of a larger task, such as exploring your project, making changes, or reviewing results. dbt Wizard orchestrates them and consolidates the results back into your session. Use subagents when you can split a task into independent pieces of work. They're useful for reviewing a large pull request, debugging a failed job, adding tests across multiple models, or researching documentation while another agent inspects your project. For smaller, direct tasks, you usually don't need subagents. Ask a single question, like "What does this model do?" or "Fix this failing test," without splitting the work across agents. Subagents use more tokens than handling the same task in a single-agent session. Use them when the task benefits from dividing the work. See it in action and share your feedback Want to see dbt Wizard in action? Check out the [demo video](https://www.youtube.com/watch?v=-lIzh1xQWMA). We'd love to hear how dbt Wizard is working for you. Share your feedback by either running the `/feedback` slash command in your interactive terminal session or by going to the [#dbt-wizard](https://getdbt.slack.com/archives/C0B6KLW6T26) channel in the [dbt Community Slack](https://docs.getdbt.com/community/join?version=2.0). Thanks so much for your help in improving dbt Wizard and dbt data development! #### How subagents work An agent is a role that describes a type of work, like `explorer`, `worker`, or `test_writer`. A subagent is a running instance of one of those roles. For example, if you start two explorer agents to answer two different questions, you have two subagents that share the same agent role. dbt Wizard handles orchestration for you. It starts subagents, routes work to them, waits for their results, and consolidates their output into your session. You can use subagents in the following ways: | How subagents start | Example | | ---------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | You ask to split up the work. | `Review PR #123. Use one agent to map what changed, one to check tests and downstream impact, and one to look up the relevant dbt docs.` | | dbt Wizard automatically uses a configured agent because your task matches that agent's description. | `Add tests for stg_customers and check whether similar staging models are missing tests.` | #### Built-in agents dbt Wizard ships with built-in agents that it can spawn without any configuration. Several are purpose-built for dbt work: | Agent | What it's for | Example | | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | | `explorer` | Answers specific, well-scoped questions about your project. Fast and read-only — spawn several in parallel for independent questions. | `Use explorer to explain what depends on fct_orders.` | | `worker` | Performs execution and production work, such as implementing part of a feature, fixing tests or bugs, or splitting a large refactor into independent chunks. | `Use worker to update these staging models to follow our naming convention.` | | `validation` | Provides dbt validation. After model edits, runs structured validation — SQL check, `dbt run` with `--defer`, prod vs. dev comparison, and impact analysis — to validate changes before you merge. | `Use validation to check whether my changes to int_payments are safe to merge.` | | `test_writer` | Improves dbt test coverage. Analyzes project metadata and warehouse data to find coverage gaps, validates assumptions with queries, and writes `schema.yml` tests for models with low or no coverage. | `Use test_writer to add tests to stg_customers.` | You don't need to declare these — dbt Wizard routes to them automatically when a task fits, or you can ask for one by name. #### Related docs * [About dbt Wizard in the dbt platform](https://docs.getdbt.com/docs/platform/wizard-platform.md) * [dbt Wizard in Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) * [Use subagents locally](https://docs.getdbt.com/docs/dbt-ai/wizard-subagents.md) * [How dbt Wizard works](https://docs.getdbt.com/docs/dbt-ai/wizard-how-it-works.md) --- ### User-defined functions User-defined functions (UDFs) enable you to define and register custom functions in your warehouse. Like [macros](https://docs.getdbt.com/docs/build/jinja-macros.md), UDFs promote code reuse, but they are objects in the warehouse so you can reuse the same logic in tools outside dbt, such as BI tools, data science notebooks, and more. UDFs are particularly valuable for sharing logic across multiple tools, standardizing complex business calculations, improving performance for compute-intensive operations (since they're compiled and optimized by your warehouse's query engine), and version controlling custom logic within your dbt project. dbt creates, updates, and renames UDFs as part of DAG execution. The UDF is built in the warehouse before the model that references it. Refer to [listing and building UDFs](https://docs.getdbt.com/docs/build/udfs.md#listing-and-building-udfs) for more info on how to build UDFs in your project. Refer to [Function properties](https://docs.getdbt.com/reference/function-properties.md) or [Function configurations](https://docs.getdbt.com/reference/function-configs.md) for more information on the configs/properties for UDFs. #### Prerequisites * Make sure you're using dbt platform's **v2 Stable** or **v1 Latest** [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) or dbt v1.11+. * Use one of the following adapters: ##### dbt v1 * BigQuery * Snowflake * Redshift * Postgres * Databricks ##### dbt v2 * BigQuery * Snowflake * Redshift * Databricks UDF support JavaScript UDFs are supported in dbt v1.12+ on Snowflake and BigQuery. Additional languages (for example, Java, Scala) aren't currently supported for UDFs. See the [Limitations](#limitations) section below for the full list of currently supported UDF capabilities. #### Defining UDFs in dbt You can define SQL, Python, and JavaScript (available in dbt v1.12+) UDFs in dbt. Python UDFs are supported in Snowflake, BigQuery, and Databricks ([Unity Catalog](https://docs.databricks.com/aws/en/data-governance/unity-catalog/) required). JavaScript UDFs are supported in Snowflake and BigQuery. Follow these steps to define UDFs in dbt: 1. Create a SQL, Python, or JavaScript file under the `functions` directory. For example, this UDF checks if a string represents a positive integer: ##### SQL Define a SQL UDF in a SQL file. functions/is\_positive\_int.sql ```sql # syntax for BigQuery, Snowflake, and Databricks REGEXP_INSTR(a_string, '^[0-9]+$') # syntax for Redshift and Postgres SELECT REGEXP_INSTR(a_string, '^[0-9]+$') ``` ##### Python Define a Python UDF in a Python file. functions/is\_positive\_int.py ```py import re def main(a_string): return 1 if re.search(r'^[0-9]+$', a_string or '') else 0 ``` For Databricks, the contents of the `.py` file become the UDF body verbatim, and Databricks evaluates that body directly instead of calling a named entry point. Write the body so its last statement is a top-level `return` that produces the result. Because of that top-level `return`, the Databricks source is a function *body*, not a runnable `.py` module. For example: functions/is\_positive\_int.py ```py import re def main(a_string): return 1 if re.search(r'^[0-9]+$', a_string or '') else 0 return main(a_string) ``` **Note:** Python UDFs on Databricks require [Unity Catalog](https://docs.databricks.com/aws/en/data-governance/unity-catalog/). ##### JavaScript Define a JavaScript UDF in a JavaScript file. functions/is\_positive\_int.js ```js return /^[0-9]+$/.test(a_string) ? 1 : 0; ``` **Note**: You can specify configs in a config block in the SQL file or in the corresponding properties YAML file in step 2. 2. Specify the function name and define the config, properties, return type, and optional arguments in a corresponding properties YAML file. ##### SQL functions/is\_positive\_int.yml ```yml functions: - name: is_positive_int # required description: My UDF that returns 1 if a string represents a naked positive integer (like "10", "+8" is not allowed). # optional config: schema: udf_schema database: udf_db volatility: deterministic arguments: # optional - name: a_string # required if arguments is specified data_type: string # required if arguments is specified description: The string that I want to check if it's representing a positive integer (like "10") default_value: "'1'" # optional, available in Snowflake and Postgres returns: # required data_type: integer # required ``` ##### Python The following configs are required when defining a Python UDF on Snowflake and BigQuery: * [`runtime_version`](https://docs.getdbt.com/reference/resource-configs/runtime-version.md) — Specify the Python version to run. Supported values are: * [Snowflake](https://docs.snowflake.com/en/developer-guide/udf/python/udf-python-introduction): `3.10`, `3.11`, `3.12`, and `3.13` * [BigQuery](https://cloud.google.com/bigquery/docs/user-defined-functions-python): `3.11` * [`entry_point`](https://docs.getdbt.com/reference/resource-configs/entry-point.md) — Specify the Python function to be called. <br /> On Databricks, `runtime_version` and `entry_point` are accepted for cross-adapter compatibility but have no effect. Databricks manages the Python runtime internally and uses the function body directly, so dbt displays a warning if you set them. You can specify public third-party PyPI packages for your Python UDF with the optional `packages` config. List package names, such as `numpy` and `pandas`, and optionally pin versions, such as `pandas==1.5.0`. The warehouse installs these packages when it creates the UDF, so your UDF can use functionality from external Python libraries. On Snowflake, some packages are installed from the Anaconda repository, and you may need to [accept Anaconda's Terms of Service](https://docs.snowflake.com/en/developer-guide/udf/python/udf-python-packages#using-third-party-packages-from-anaconda) before you can use them. The following example shows a Python UDF with the required configs (`runtime_version`, `entry_point`), the optional `packages` config, and other common configs: functions/is\_positive\_int.yml ```yml functions: - name: is_positive_int # required description: My UDF that returns 1 if a string represents a naked positive integer (like "10", "+8" is not allowed). # optional config: runtime_version: "3.11" # required for Snowflake and BigQuery; optional and ignored on Databricks entry_point: main # required for Snowflake and BigQuery; optional and ignored on Databricks packages: # optional, Python UDFs only - numpy - pandas==1.5.0 schema: udf_schema database: udf_db volatility: deterministic arguments: # optional - name: a_string # required if arguments is specified data_type: string # required if arguments is specified description: The string that I want to check if it's representing a positive integer (like "10") default_value: "'1'" # optional, available in Snowflake and Postgres returns: # required data_type: integer # required ``` ##### JavaScript You can optionally set [`snowflake.quote_args`](https://docs.getdbt.com/reference/resource-configs/quote_args.md) to control whether argument names are quoted when creating a JavaScript UDF on Snowflake. functions/is\_positive\_int.yml ```yml functions: - name: is_positive_int # required description: My UDF that returns 1 if a string represents a naked positive integer (like "10", "+8" is not allowed). # optional config: snowflake: # optional quote_args: true # optional, JavaScript UDFs on Snowflake only arguments: # optional - name: a_string # required if arguments is specified data_type: string # required if arguments is specified description: The string to check # optional returns: # required data_type: integer # required ``` volatility warehouse-specific `volatility` is accepted in dbt for SQL, Python, and JavaScript UDFs, but the handling of it is warehouse-specific. For SQL and Python UDFs on BigQuery, `volatility` is ignored and dbt displays a warning. For JavaScript UDFs on BigQuery, `deterministic` and `non-deterministic` are applied when creating the UDF; `stable` is not supported. In Snowflake, all supported volatility values are applied when creating the UDF. Refer to [volatility](https://docs.getdbt.com/reference/resource-configs/volatility.md) for more information. 3. Run one of the following `dbt build` commands to build your UDFs and create them in the warehouse: Build all UDFs: ```bash dbt build --select "resource_type:function" ``` Or build a specific UDF: ```bash dbt build --select is_positive_int ``` When you run `dbt build`, the property file (`functions/is_positive_int.yml`) and the corresponding SQL, Python, or JavaScript file work together to generate the `CREATE FUNCTION` statement. The rendered `CREATE FUNCTION` statement depends on which adapter you're using. For example: ##### SQL ##### Snowflake ```sql CREATE OR REPLACE FUNCTION udf_db.udf_schema.is_positive_int(a_string STRING DEFAULT '1') RETURNS INTEGER LANGUAGE SQL IMMUTABLE AS $$ REGEXP_INSTR(a_string, '^[0-9]+$') $$; ``` ##### Redshift ```sql CREATE OR REPLACE FUNCTION udf_db.udf_schema.is_positive_int(a_string VARCHAR) RETURNS INTEGER IMMUTABLE AS $$ SELECT REGEXP_INSTR(a_string, '^[0-9]+$') $$ LANGUAGE SQL; ``` ##### BigQuery ```sql CREATE OR REPLACE FUNCTION udf_db.udf_schema.is_positive_int(a_string STRING) RETURNS INT64 AS ( REGEXP_INSTR(a_string, r'^[0-9]+$') ); ``` ##### Databricks ```sql CREATE OR REPLACE FUNCTION udf_db.udf_schema.is_positive_int(a_string STRING) RETURNS INT DETERMINISTIC RETURN REGEXP_INSTR(a_string, '^[0-9]+$'); ``` ##### Postgres ```sql CREATE OR REPLACE FUNCTION udf_schema.is_positive_int(a_string text DEFAULT '1') RETURNS int LANGUAGE sql IMMUTABLE AS $$ SELECT regexp_instr(a_string, '^[0-9]+$') $$; ``` ##### Python ##### Snowflake ```sql CREATE OR REPLACE FUNCTION udf_db.udf_schema.is_positive_int(a_string STRING DEFAULT '1') RETURNS INTEGER LANGUAGE PYTHON RUNTIME_VERSION = '3.11' HANDLER = 'main' PACKAGES = ('numpy', 'pandas==1.5.0') AS $$ import re def main(a_string): return 1 if re.search(r'^[0-9]+$', a_string or '') else 0 $$; ``` ##### BigQuery ```sql CREATE OR REPLACE FUNCTION udf_db.udf_schema.is_positive_int(a_string STRING) RETURNS INT64 LANGUAGE python OPTIONS( runtime_version = "python-3.11", entry_point = "main", packages = ['numpy', 'pandas==1.5.0'] ) AS r''' import re def main(a_string): return 1 if re.search(r'^[0-9]+$', a_string or '') else 0 '''; ``` ##### Databricks ```sql CREATE OR REPLACE FUNCTION udf_db.udf_schema.is_positive_int(a_string STRING) RETURNS INT LANGUAGE PYTHON AS $$ import re def main(a_string): return 1 if re.search(r'^[0-9]+$', a_string or '') else 0 return main(a_string) $$; ``` Databricks omits the `RUNTIME_VERSION` and `HANDLER` clauses. The runtime is managed internally, and the contents of your `.py` file become the function body verbatim — including the trailing `return main(a_string)` that produces the result. ##### JavaScript ##### Snowflake ```sql CREATE OR REPLACE FUNCTION udf_db.udf_schema.is_positive_int("a_string" STRING) RETURNS INTEGER LANGUAGE JAVASCRIPT AS $$ return /^[0-9]+$/.test(a_string) ? 1 : 0; $$; ``` ##### BigQuery ```sql CREATE OR REPLACE FUNCTION udf_db.udf_schema.is_positive_int(a_string STRING) RETURNS INT64 LANGUAGE js AS r''' return /^[0-9]+$/.test(a_string) ? 1 : 0; '''; ``` 4. Reference the UDF in a model using the `{{ function(...) }}` macro. For example: models/my\_model.sql ```sql select maybe_positive_int_column, {{ function('is_positive_int') }}(maybe_positive_int_column) as is_positive_int from {{ ref('a_model_i_like') }} ``` When using [`--defer`](https://docs.getdbt.com/reference/node-selection/defer.md), `function()` resolves to the existing UDF in the deferred environment (for example, production) if the function is not selected or not yet built in your target environment. This requires a state manifest specified using `--state` or an equivalent environment variable (such as `DBT_ENGINE_STATE`), which dbt uses to determine where to defer. This allows models that depend on UDFs to run successfully in [continuous integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md) and development workflows. For more information, refer to [Configure state selection](https://docs.getdbt.com/reference/node-selection/configure-state.md). 5. Run `dbt compile` to see how the UDF is referenced. In the following example, the `{{ function('is_positive_int') }}` is replaced by the UDF name `udf_db.udf_schema.is_positive_int`. models/my\_model.sql ```sql select maybe_positive_int_column, udf_db.udf_schema.is_positive_int(maybe_positive_int_column) as is_positive from analytics.dbt_schema.a_model_i_like ``` In your DAG, a UDF node is created from the SQL/Python and YAML definitions, and there will be a dependency between `is_positive_int` → `my_model`. [![The DAG for the UDF node](/img/docs/building-a-dbt-project/UDF-DAG.png?v=2 "The DAG for the UDF node")](#)The DAG for the UDF node After defining a UDF, your changes are applied to the UDF in the warehouse the next time you run `dbt build` when you update any of the following: * The SQL, Python, or JavaScript file that contains its function body (`is_positive_int.sql`, `is_positive_int.py`, or `is_positive_int.js` in these examples) * Its configurations * Its properties defined in the `.yml` file (such as `arguments` or `returns`) dbt detects all of these changes when using [`state:modified`](https://docs.getdbt.com/reference/node-selection/methods.md#state). ##### Defining overloaded UDFs Use the [`overloads`](https://docs.getdbt.com/reference/resource-properties/overloads.md) property (available in dbt v1.12+) to define multiple argument signatures for the same function. This lets you call the same function name with different input types, without creating separate UDFs for each variant. `overloads` is supported for SQL UDFs in Snowflake and Postgres, and Python and JavaScript UDFs in Snowflake. To define overloaded UDFs: 1. Add an `overloads` list to the function definition in your properties YAML file. Each entry uses `defined_in` to reference a separate file, with optional `arguments` and `returns`: functions/is\_positive\_int.yml ```yml functions: - name: is_positive_int arguments: - name: a_string data_type: string returns: data_type: integer overloads: - defined_in: is_positive_int_numeric # references functions/is_positive_int_numeric.sql arguments: - name: a_num data_type: numeric returns: # optional, inherits from root function if omitted data_type: integer ``` 2. Create a separate file for each overload body. For example, the body for the root function, which accepts a `string` argument: functions/is\_positive\_int.sql ```sql # Snowflake syntax REGEXP_INSTR(a_string, '^[0-9]+$') ``` And the body for the overload, which accepts a `numeric` argument: functions/is\_positive\_int\_numeric.sql ```sql # Snowflake syntax CASE WHEN a_num > 0 THEN 1 ELSE 0 END ``` All overloads are grouped into one DAG node (the root function), so they're built and selected together. On retry, dbt skips overloads that succeeded and reruns only those that failed. When dbt builds the function, it renders a separate `CREATE FUNCTION` statement for each overload using the same function name but different argument types. For more information, refer to [`overloads`](https://docs.getdbt.com/reference/resource-properties/overloads.md). #### Using UDFs in unit tests You can use [unit tests](https://docs.getdbt.com/docs/build/unit-tests.md) to validate models that reference UDFs. Before running unit tests, make sure the function exists in your warehouse. To ensure that the function exists for a unit test, run: ```bash dbt build --select "+my_model_to_test" --empty ``` Following the example in [Defining UDFs in dbt](#defining-udfs-in-dbt), here's an example of a unit test that validates a model that calls a UDF: tests/test\_is\_positive\_int.yml ```yml unit_tests: - name: test_is_positive_int description: "Check my is_positive_int logic captures edge cases" model: my_model given: - input: ref('a_model_i_like') rows: - { maybe_positive_int_column: 10 } - { maybe_positive_int_column: -4 } - { maybe_positive_int_column: +8 } - { maybe_positive_int_column: 1.0 } expect: rows: - { maybe_positive_int_column: 10, is_positive: true } - { maybe_positive_int_column: -4, is_positive: false } - { maybe_positive_int_column: +8, is_positive: true } - { maybe_positive_int_column: 1.0, is_positive: true } ``` #### Listing and building UDFs Use the [`list` command](https://docs.getdbt.com/reference/commands/list.md#listing-functions) to list UDFs in your project: `dbt list --select "resource_type:function"` or `dbt list --resource-type function`. Use the [`build` command](https://docs.getdbt.com/reference/commands/build.md#functions) to select UDFs when building a project: `dbt build --select "resource_type:function"`. For more information about selecting UDFs, see the examples in [Node selector methods](https://docs.getdbt.com/reference/node-selection/methods.md#file). #### Limitations * UDFs in other languages (for example, Java or Scala) are not yet supported. * JavaScript UDFs are supported in Snowflake and BigQuery only. Using JavaScript UDFs on an unsupported adapter raises a parsing error. * Python UDFs are supported in Snowflake, BigQuery, and Databricks only (when using dbt v1 or dbt v2). Other warehouses aren't yet supported for Python UDFs. * Only scalar and aggregate functions are currently supported. For more information, see [Supported function types](https://docs.getdbt.com/reference/resource-configs/type.md#supported-function-types). * The `overloads` property is supported for SQL UDFs in Snowflake and Postgres, and Python and JavaScript UDFs in Snowflake. #### Related FAQs When should I use a UDF instead of a macro? Both user-defined functions (UDFs) and macros let you reuse logic across your dbt project, but they work in fundamentally different ways. Here's when to use each: ###### Use UDFs when:  You need logic accessible outside dbt UDFs are created in your warehouse and can be used by BI tools, data science notebooks, SQL clients, or any other tool that connects to your warehouse. Macros only work within dbt.  You want to standardize warehouse-native functions UDFs let you create reusable warehouse functions for data validation, custom formatting, or business-specific calculations that need to be consistent across all your data tools. Once created, they become part of your warehouse's function catalog.  You want dbt to manage the function lifecycle dbt manages UDFs as part of your DAG execution, ensuring they're created before models that reference them. You can version control UDF definitions alongside your models, test changes in development environments, and deploy them together through CI/CD pipelines.  Jinja compiles at creation time, not on each function call You can use Jinja (loops, conditionals, macros, `ref`, `source`, `var`) inside a UDF configuration. dbt resolves that Jinja **when the UDF is created**, and the resulting SQL body is what gets stored in your warehouse. Jinja influences the function when it’s created, whereas arguments influence it when it runs in the warehouse: * ✅ **Allowed:** Jinja that depends on project or build-time state — for example, `var(“can_do_things”)`, static `ref(‘orders’)`, or environment-specific logic. These are all evaluated once at creation time. * ❌ **Not allowed:** Jinja that depends on **function arguments** passed at runtime. The compiler can’t see those, so dynamic `ref(ref_name)` or conditional Jinja based on argument values won’t work.  You need Python logic that runs in your warehouse A Python UDF creates a Python function directly within your data warehouse, which you can invoke using SQL.<br />This makes it easier to apply complex transformations, calculations, or logic that would be difficult or verbose to express in SQL. Python UDFs support conditionals and looping within the function logic itself (using Python syntax), and execute at runtime, not at compile time like macros. Python UDFs are currently supported in Snowflake and BigQuery. ###### Use macros when:  You need to generate SQL at compile time Macros generate SQL dynamically **before** it's sent to the warehouse (at compile time). This is essential for: * Building different SQL for different warehouses * Generating repetitive SQL patterns (like creating dozens of similar columns) * Creating entire model definitions or DDL statements * Dynamically referencing models based on project structure UDFs execute **at query runtime** in the warehouse. While they can use Jinja templating in their definitions, they don't generate new SQL queries—they're pre-defined functions that get called by your SQL. Expanding UDFs Currently, SQL and Python UDFs are supported. Java and Scala UDFs are planned for future releases.  You want to generate DDL or DML statements Currently, SQL and Python UDFs are supported. Java and Scala UDFs are planned for future releases.  You need to adapt SQL across different warehouses Macros can use Jinja conditional logic to generate warehouse-specific SQL (see [cross-database macros](https://docs.getdbt.com/reference/dbt-jinja-functions/cross-database-macros.md)), making your dbt project portable across platforms. UDFs are warehouse-specific objects. Even though UDFs can include Jinja templating in their definitions, each warehouse has different syntax for creating functions, different supported data types, and different SQL dialects. You would need to define separate UDF files for each warehouse you support.  Your logic needs access to dbt context Both macros and UDFs can use Jinja, which means they can access dbt context variables like `{{ ref() }},` `{{ source() }}`, environment variables, and project configurations. You can even call a macro from within a UDF (and vice versa) to combine dynamic SQL generation with runtime execution. However, the difference between the two is *when* the logic runs: * Macros run at compile time, generating SQL before it’s sent to the warehouse. * UDFs run inside the warehouse at query time.  You want to avoid creating warehouse objects Macros don't create anything in your warehouse; they just generate SQL at compile time. UDFs create actual function objects in your warehouse that need to be managed. ###### Can I use both together? Yes! You can use a macro to call a UDF or call a macro from within a UDF, combining the benefits of both. So the following example shows how to use a macro to define default values for arguments alongside your logic, for your UDF ```sql {% macro cents_to_dollars(column_name, scale=2) %} {{ function('cents_to_dollars') }}({{ column_name }}, {{scale}}) {% endmacro %} ``` ###### Related documentation * [User-defined functions](https://docs.getdbt.com/docs/build/udfs.md) * [Jinja macros](https://docs.getdbt.com/docs/build/jinja-macros.md) --- ### Using catalogs.yml Now that you understand [what an Iceberg catalog is](https://docs.getdbt.com/docs/build/iceberg/about-catalogs.md), let's talk about how to use them within dbt. #### Getting started: Managed catalogs Several data platforms offer their own "managed" catalogs that support the Iceberg table format out-of-the-box. These include Snowflake Horizon, Databricks Unity, AWS Glue (for Athena + Redshift), and BigLake (for BigQuery). If you're using dbt with a data platform that offers a managed Iceberg catalog, then the simplest way to materialize your first dbt model as an Iceberg table is to set the `table_format` configuration: models/hello\_iceberg.sql ```sql {{ config( materialized = 'table', table_format = 'iceberg' ) }} select 'hello_iceberg' as message ``` That's it. This model is materialized as an Iceberg table, with all the default configurations for this adapter, and stored in the default (managed) catalog offered by this data platform. You can now connect to that catalog using another engine (such as DuckDB) to read this table. Congratulations, you're using Iceberg! note Most open source query engines, including DuckDB and Apache Spark, can operate well with external catalogs, but they don't come with a "managed" catalog. Instead, their default behavior for materializing Iceberg tables is to write Parquet files and Iceberg metadata to the local filesystem (wherever the query engine is running). #### Next step: Using catalogs You should start using `catalogs` when: * You want a single place to define custom configurations for how dbt should materialize Iceberg tables * You want to write to multiple catalogs ("external" as well as built-in / managed) * You want to access the same catalog across multiple data platforms / dbt projects dbt defines `catalogs` in a single top-level file, `catalogs.yml`, that lives in the root of your project directory. We first introduced `catalogs.yml` in dbt v1.10; starting in dbt v1.12, we've introduced a new simpler spec (recommended) behind an opt-in behavior flag. ##### New spec (recommended) [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") *Available in dbt v1.12+ (including v2). See GitHub discussion [dbt-labs/dbt#12723](https://github.com/dbt-labs/dbt/discussions/12723) for an explanation of the motivations behind the new spec, and an overview of what's changed.* To use the new spec, first set this behavior flag: dbt\_project.yml ```yml flags: use_catalogs_v2: true ``` Each entry in `catalogs` refers to a specific catalog containing Iceberg tables. Each catalog **should** map to a top-level logical namespace (often called "database" in dbt). Each catalog may be managed or external for this data platform. Each catalog may be accessed (read from and written to) by one or multiple data platforms. For this reason, each catalog's adapter-specific configuration is nested under `ADAPTER` keys (such as `snowflake:` and `databricks:`). If you run the same dbt project, with the same `catalogs.yml`, using different adapters, dbt always uses the catalog configuration for the current active adapter. That said, one "catalog" **must** always point to the same actual data (Iceberg tables in object storage), regardless of whether that catalog is external to or managed by the current active adapter. (Applies to dbt v1.12 and later) catalogs.yml ```yml catalogs: - name: my_iceberg_catalog type: CATALOG_TYPE # see below table_format: iceberg # default, optional config: ADAPTER: # Configuration for a specific adapter to integrate with this catalog. # See available configs for each adapter + catalog combination. ``` ###### Catalog types | catalog `type` | default for | supported by | Notes | | ------------------ | ----------- | ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | horizon | snowflake | snowflake, duckdb | | | glue | athena | athena, snowflake, duckdb | | | biglake\_metastore | bigquery | bigquery, snowflake | Supports BigQuery [Lakehouse Runtime Catalog tables](https://docs.getdbt.com/docs/build/iceberg/adapters/bigquery-iceberg-support.md#lakehouse-runtime-catalog-lrc) through the `lakehouse_catalog` config | | unity | databricks | databricks, snowflake, duckdb | Supports Iceberg (native), Delta, "Uniform" formats | | hive\_metastore | | databricks | Supports Hudi format, in addition to Iceberg + Delta | | ducklake | | duckdb | | | local\_filesystem | | duckdb | | | iceberg\_rest | | snowflake, duckdb | In theory, this option can support any catalog that implements an Iceberg-compatible REST API | ###### Config inheritance Configurations defined in `catalogs.yml` are lowest in the [model-config precedence](https://docs.getdbt.com/reference/model-configs.md). This means that if you set a more-specific config for one model (within its `.sql` or `.py` file), or set a project-level config, those take precedence. For example, you set a default `base_location_root` for all models in the `finance_db` catalog: catalogs.yml ```yml catalogs: - name: finance_db catalog_type: unity config: snowflake: base_location_root: 's3://my-bucket/finance_db' ``` But then you override that config for one particular model: models/finance/my\_special\_model.sql ```sql {{ config( catalog_name = 'finance_db', base_location_root = 's3://my-bucket/somewhere_else' ) }} ``` Some Iceberg-related configurations are only available at the model configuration level, so they can't be set in `catalogs.yml`. For example, the related config `base_location_subpath` determines the exact write path for a single Iceberg table, so it only makes sense to configure per-model, rather than setting a default for all models in the catalog. ###### The `catalog_database` config There is a dedicated configuration for the name of the database mapped to this catalog in each adapter. Unlike other catalog-level configurations, the `catalog_database` applies to *all* models configured with this `catalog_name`, it can't be changed for specific models, and it doesn't follow the usual rules about the `database` config or the `generate_database_name` macro. This means that if a model has both a `database` config and a `catalog_name` config, the catalog’s `catalog_database` takes precedence over the model’s database config, to tell dbt where to materialize this model. Why? We strongly recommend a 1:1 mapping between each Iceberg catalog and the top-level namespace (logical "database") to which it is linked/synced. The namespaces don't have to be identical, but if you can make them the same everywhere, it's simpler to reason about and debug. If you don't specify a `catalog_database`, then dbt materializes models based on their `database` config. In this case, the catalog serves as a collection of shared Iceberg configs, but it doesn't map to a consistent namespace containing all its Iceberg tables. ##### Old spec *Available in dbt v1.10+* Each catalog configures one or more `write_integrations`, and then specifies an "active" write integration to use for the current invocation / data warehouse. catalogs.yml ```yml catalogs: - name: my_glue_catalog active_write_integration: glue_rest write_integrations: - name: glue_rest catalog_type: iceberg_rest table_format: iceberg adapter_properties: catalog_linked_database: catalog_linked_db_glue catalog_linked_database_type: glue ``` --- ### Validations Validations refer to the process of checking whether a system or configuration meets the expected requirements or constraints. In the case of the Semantic Layer, powered by MetricFlow, there are three built-in validations — [parsing](#parsing), [semantic](#semantic), and [data platform](#data-platform). These validations ensure that configuration files follow the expected schema, the semantic graph doesn't violate any constraints, and semantic definitions in the graph exist in the physical table — providing effective data governance support. These three validation steps occur sequentially and must succeed before proceeding to the next step. The code that handles validation [can be found here](https://github.com/dbt-labs/dbt-semantic-interfaces/tree/main/dbt_semantic_interfaces/validations) for those who want to dive deeper into this topic. #### Validations command You can run validations from the dbt platform or the command line with the following [MetricFlow commands](https://docs.getdbt.com/docs/build/metricflow-commands.md). In dbt, you need user credentials configured in your account settings to run `dbt sl validate` in the IDE or CLI, and deployment credentials to run it in CI. * For dbt v2 and dbt users in the dbt platform CLI or locally with a valid [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md): ```bash dbt sl validate ``` This runs parsing, semantic, and (where supported) data platform validations. When using `dbt sl validate` locally, the command validates your local semantic manifest, and not the platform's manifest. This means your uncommitted local changes are included in the validation. * For dbt v1 (open source) users or v2 CLI users not connected to dbt platform and using local MetricFlow: ```bash mf validate-configs ``` This runs parsing and semantic validations. #### Availability by environment Validation behavior and availability differ depending on your environment and setup: | Environment | Who can use | Parsing | Semantic syntax | Data platform | How to run | | ---------------- | --------------------------------------------------- | ------- | --------------- | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | dbt v2 | dbt platform users for full Semantic Layer features | ✅ | ✅ \* | ✅ | - Parsing validations run automatically while generating the semantic manifest.<br />- When running in development, semantic syntax validations run automatically on dbt platform if [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md) is configured. If not, run manually using `mf validate-configs`.<br />- Data platform validations don't run automatically for dbt v2. You must run `dbt sl validate` to run data platform validations. | | dbt platform CLI | dbt platform users | ✅ | ✅ | ✅ | Run any dbt platform CLI command; validations execute automatically except data platform validations. You must run `dbt sl validate` to run data platform validations. | | dbt v1 | Open source users | ✅ | ✅ | ❌ | Use dbt v1 for parsing/builds. Run additional validation manually with the MetricFlow CLI. | | MetricFlow CLI | Open source users | ✅ | ✅ | ✅ | Run `mf validate-configs` locally to validate and test metrics. | \*Jobs run in **Orchestration** or **Studio IDE** run this validation automatically. #### Parsing In this validation step, we ensure your config files follow the defined schema for each semantic graph object and can be parsed successfully. It validates the schema for the following core objects: (Applies to dbt v1.12 and later) * Semantic models * Identifiers * Simple metrics * Dimensions * Metrics #### Semantic syntax This syntactic validation step occurs after we've built your semantic graph. The Semantic Layer, powered by MetricFlow, runs a suite of tests to ensure that your semantic graph doesn't violate any constraints. For example, we check to see if (Applies to dbt v1.12 and later) simple metric names are unique, or if metrics referenced in materialization exist. The current semantic rules we check for are: (Applies to dbt v1.12 and later) 1. Check those semantic models with simple metrics have a valid time dimension 2. Check that there is only one primary identifier defined in each semantic model 3. Dimension consistency 4. Unique simple metrics in semantic models 5. Simple metrics are valid 6. Cumulative metrics are configured properly #### Data platform This type of validation checks to see if the semantic definitions in your semantic graph exist in the underlying physical table. To test this, we run queries against your data platform to ensure the generated SQL for semantic models, dimensions, and metrics will execute. We run the following checks: (Applies to dbt v1.12 and later) * Simple metrics and dimensions exist * Underlying tables for data sources exist * Generated SQL for metrics will execute You can run semantic validations (against your semantic layer) in a CI job to guarantee any code changes made to dbt models don't break these metrics. For more information, refer to [semantic validation in CI](https://docs.getdbt.com/docs/deploy/ci-jobs.md#semantic-validations-in-ci). --- ### Version compatibility Preview ### Version compatibility [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") dbt v2, its language server (LSP), and the dbt VS Code extension work together to power local development. This page explains how their versions relate, which combinations are compatible, and how to verify you have matching binaries. #### How versioning works dbt v2 ships as a single binary that provides both the command-line interface (CLI) and the language server (LSP). Because the CLI and LSP come from the *same* binary, they always share the same version and can't be mismatched. The dbt VS Code extension keeps its own release version because it follows the VS Code Marketplace release lifecycle. To make compatibility explicit, each extension release declares the range of dbt v2 versions it supports. When the extension starts the LSP, it checks the installed dbt v2 version against that range and surfaces an [actionable error](#troubleshooting-version-issues) if they don't match. | Component | Example versioning | | ------------------------- | ----------------------------------------------------------------------------------- | | dbt v2 binary (CLI + LSP) | `2.0.0-preview.N` | | dbt VS Code extension | The extension version (for example, `0.36.0`), with compatible dbt v2 version range | #### Compatibility matrix The following table maps each dbt VS Code extension version to the dbt versions it supports. If you let the extension download and manage dbt v2 for you, it always installs a compatible version automatically — you only need this table if you install or pin binaries manually. | dbt VS Code extension | Min dbt v2 version | Max dbt v2 version | Checksums | Notes | | --------------------- | ------------------ | ------------------ | -------------------- | ------- | | 0.36.x | 2.0.0-preview.90 | — | `checksums-0.36.txt` | Current | | 0.35.x | 2.0.0-preview.80 | 2.0.0-preview.89 | `checksums-0.35.txt` | - | A dash (-) in the **Max** column means there is no upper bound yet. Any dbt v2 release at or above the minimum is supported. #### Check your installed versions To confirm your setup is within the supported range, check the version of dbt v2 you have installed: ```shell dbt --version ``` For a machine-readable summary you can paste into a bug report or parse in tooling, add the `--format json` flag: ```shell dbt --version --format json ``` For details on the full output, refer to [About dbt --version](https://docs.getdbt.com/reference/commands/version.md). #### Verify binaries for manual and air-gapped installs If you install dbt v2 manually in an air-gapped or firewall-restricted environment where the extension can't download binaries for you, use the [compatibility matrix](#compatibility-matrix) to pick a version that matches your dbt VS Code extension, then verify the download before you distribute it. 1. From the compatibility matrix, note the dbt v2 version range for your extension version. 2. Download the matching dbt v2 binary and its checksum file. 3. Verify the checksum before installing or distributing the binary: ##### Mac / Linux ```shell shasum -a 256 -c checksums-0.36.txt ``` ##### Windows (PowerShell) ```powershell Get-FileHash .\dbt.exe -Algorithm SHA256 ``` Compare the printed hash against the value in the checksum file. 4. Point the dbt VS Code extension at the verified binary using the `dbt.fusionPath` setting. Refer to [dbt extension settings](https://docs.getdbt.com/docs/configure-dbt-extension.md#dbt-extension-settings). For network and proxy requirements, and how to pre-build an offline bundle, refer to [Networking requirements](https://docs.getdbt.com/docs/local/dbt-networking-requirements.md#restricted-network-installation). #### Known-bad releases If a shipped dbt v2 release is later found to contain a regression, dbt Labs flags it as a known-bad release. When you have a flagged version installed, the dbt VS Code extension shows a warning notification telling you which version to update to, even if that version is otherwise within the supported range. Air-gapped users who don't have outbound network access can distribute the known-bad releases manifest alongside their binary bundle and point the extension at the local copy with the `dbt.badReleasesManifestPath` setting. Refer to [dbt extension settings](https://docs.getdbt.com/docs/configure-dbt-extension.md#dbt-extension-settings). #### Troubleshooting version issues If the extension can't find, start, or verify a compatible dbt v2 binary, it surfaces an actionable error with a link or button to resolve it. Refer to the [dbt VS Code extension troubleshooting](https://docs.getdbt.com/docs/sign-in-dbt-extension.md#troubleshooting) section for the full list of messages and fixes. --- ### Version upgrade guides #### [📄️ v2 readiness checklist](https://docs.getdbt.com/docs/dbt/dbt-readiness.md) [Your to-do list for preparing for dbt v2](https://docs.getdbt.com/docs/dbt/dbt-readiness.md) --- ### View documentation dbt provides intuitive and scalable tools for viewing your dbt documentation. Detailed documentation is essential for your developers and other stakeholders to gain shared context for your dbt project. You can view documentation in three complementary ways, depending on your needs: | Option | Description | Availability | | ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------- | | [**dbt Docs (Legacy)**](#dbt-docs) | Generates a static website with model lineage, metadata, and documentation that can be hosted on your web server (like S3 or Netlify). | dbt v1 or dbt Developer plans | | [**dbt Docs v2**](#dbt-docs-v2) | A modern, performant open-source catalog built for data consumers. Includes a redesigned UI, large-project performance, Semantic Layer metadata, and column-level lineage (dbt v2), served as a static site you can host anywhere. | dbt v2 | | [**Catalog**](https://docs.getdbt.com/docs/explore/explore-projects.md) | The premier documentation experience in dbt. Builds on dbt Docs to provide a dynamic, real-time interface with rich [metadata](https://docs.getdbt.com/docs/explore/explore-projects.md#generate-metadata), customizable views, deep insight into your project and resources, and collaborative tools. | dbt Starter, Enterprise, or Enterprise+ plans | #### Navigating your documentation The following sections describe how to navigate your documentation in Catalog and dbt Docs. ##### Catalog [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) offers a dynamic, interactive way to explore your models, sources, and lineage. To access Catalog, navigate to the **Catalog** option in the dbt navigation menu. [![Example of Catalog's resource details page and its lineage.](/img/docs/collaborate/dbt-explorer/example-model-details.png?v=2 "Example of Catalog's resource details page and its lineage.")](#)Example of Catalog's resource details page and its lineage. [![Access Catalog from the lineage tab in a job by double-clicking a lineage node.](/img/docs/collaborate/dbt-explorer/explorer-main-page.gif?v=2 "Access Catalog from the lineage tab in a job by double-clicking a lineage node.")](#)Access Catalog from the lineage tab in a job by double-clicking a lineage node. Catalog offers users a comprehensive suite of features to enhance data project navigation and understanding, like: * Interactive lineage visualization for your project's DAG to understand relationships between resources. * Resource search bar with comprehensive filters to help find project resources efficiently and quickly. * Model performance insights to access metadata on dbt runs for in-depth analysis of model performance and quality. * Project recommendations with suggestions to improve test coverage and documentation across your data estate. * Data health signals to monitor the health and performance of each resource through data health indicators. * Model query history to track consumption queries on your models to gain deeper insights into data usage. * Downstream exposures to automatically expose relevant data models from tools like Tableau to enhance visibility. For additional details and instructions on how to explore your lineage, navigate your resources, view model query history and data health signals, feature availability, and more — refer to [Discover data with Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md). ##### dbt Docs v2 dbt Docs v2 is the next-generation open-source catalog experience, available when using dbt v2. It is designed for data consumers (analysts, BI users, data scientists, and stakeholders) who need to understand what data exists, how it was built, and whether they can trust it. Key improvements over dbt Docs: * **Performance:** Better handling for large dbt projects. The browser queries compact, pre-built index files with DuckDB-WASM (WebAssembly) instead of loading the full `manifest.json`. * **Modernized UI:** Visually aligned with the dbt platform, with better navigation and resource discovery. * **Semantic Layer metadata:** Surfaces compiled SQL logic, queryable dimensions, and metric definitions from your dbt Semantic Layer. * **Column-level lineage:** Available when using dbt v2. * **Statically hostable:** `dbt docs generate` writes a self-contained static site — no server or live warehouse connection — that you can host on any file host, such as S3, GitHub Pages, or Netlify. To generate and serve dbt Docs v2 with dbt v2, run `dbt docs generate` to build the site, then `dbt docs serve` to preview it locally. `dbt docs generate` compiles your project and writes the index for you in a single command. To include column-level lineage and richer column metadata, first produce the artifacts with [`--static-analysis strict`](https://docs.getdbt.com/docs/build/about-static-analysis?version=1.13) using `dbt compile` or `dbt build`, then export the site: ```shell dbt compile --write-index --static-analysis strict # or: dbt build --write-index --static-analysis strict dbt docs generate --no-compile ``` Refer to [dbt docs commands](https://docs.getdbt.com/reference/commands/cmd-docs.md) for full usage. ###### Self-hosting dbt Docs v2 Because `dbt docs generate` writes a self-contained static site — the single-page app plus the Parquet index files, with no server-side query engine — you can host it on any static file host instead of running `dbt docs serve` locally. The browser loads DuckDB-WASM from a content delivery network (CDN) at runtime and queries the Parquet directly, so viewers only need a static file server and outbound internet access to reach the CDN. Generate a self-contained directory and publish it to your host of choice: ```shell dbt docs generate --output-dir site ``` This writes a `site/` directory (the app, hashed assets, and a copy of the index) that you can host on S3, GitHub Pages, Netlify, GitLab Pages, or any similar static file host. To refresh the docs, regenerate the site and re-publish. ##### dbt Docs (Legacy) dbt Docs provides valuable insights into your dbt v1 or dbt Developer plan projects. The interface enables you to navigate to the documentation for specific models. That might look something like this: [![Auto-generated documentation for a dbt model](/img/docs/building-a-dbt-project/testing-and-documentation/f2221dc-Screen_Shot_2018-08-14_at_6.29.55_PM.png?v=2 "Auto-generated documentation for a dbt model")](#)Auto-generated documentation for a dbt model Here, you can see a representation of the project structure, a markdown description for a model, and a list of all of the columns (with documentation) in the model. From the dbt Docs page, click the green button in the bottom-right corner of the webpage to expand a "mini-map" of your DAG. This pane displays the immediate parents and children of the model that you're exploring. [![Opening the DAG mini-map](/img/docs/building-a-dbt-project/testing-and-documentation/ec77c45-Screen_Shot_2018-08-14_at_6.31.56_PM.png?v=2 "Opening the DAG mini-map")](#)Opening the DAG mini-map In this example, the `fct_subscription_transactions` model only has one direct parent. By clicking the "Expand" button in the top-right corner of the window, we can pivot the graph horizontally and view the full lineage for our model. This lineage is filterable using the `--select` and `--exclude` flags, which are consistent with the semantics of [model selection syntax](https://docs.getdbt.com/reference/node-selection/syntax.md). Further, you can right-click to interact with the DAG, jump to documentation, or share links to your graph visualization with your coworkers. [![The full lineage for a dbt model](/img/docs/building-a-dbt-project/testing-and-documentation/ac97fba-Screen_Shot_2018-08-14_at_6.35.14_PM.png?v=2 "The full lineage for a dbt model")](#)The full lineage for a dbt model #### Deploy the documentation site Effortlessly deploy documentation in Catalog or dbt Docs to make it available to your teams. Security The `dbt docs serve` command is only intended for local/development hosting of the documentation site. Please use one of the methods listed in the next section (or similar) to ensure that your documentation site is hosted securely! ##### Catalog [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") Catalog automatically updates documentation after each production or staging job run using the metadata generated. This means it always has the latest results for your project with no manual deployment required. For details on how Catalog uses metadata to automatically update documentation, refer to [Generate metadata](https://docs.getdbt.com/docs/explore/explore-projects.md#generate-metadata). To learn how to deploy your documentation site, see [Build and view your docs with dbt](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md). ##### dbt Docs (Legacy) dbt Docs was built to make it easy to host on the web. The site is "static," meaning you don't need any "dynamic" servers to serve the docs. You can host your documentation in several ways: * Host on [Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosting.html) (optionally [with IP access restrictions](https://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html#example-bucket-policies-use-case-3)) * Publish with [Netlify](https://discourse.getdbt.com/t/publishing-dbt-docs-to-netlify/121) * Use your own web server like Apache/Nginx * If you're on a dbt Developer plan, see [Build and view your docs with dbt](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md#dbt-docs) to learn how to deploy your documentation site. Interested in using Catalog for the complete dbt documentation experience, sign up for a free [dbt trial](https://www.getdbt.com/signup) or [contact us](https://www.getdbt.com/contact). --- ### Visualize and orchestrate downstream exposures dbt platform | Enterprise, Enterprise+ The following table summarizes the differences between visualizing and orchestrating downstream exposures: | Info | Set up and visualize downstream exposures | Orchestrate downstream exposures [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") | | ----------------- | ---------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Purpose | Automatically brings downstream assets into your dbt lineage. | Proactively refreshes the underlying data sources during scheduled dbt jobs. | | Benefits | Provides visibility into data flow and dependencies. | Ensures BI tools always have up-to-date data without manual intervention. | | Location | Exposed in dbt [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) | Exposed in [dbt scheduler](https://docs.getdbt.com/docs/deploy/deployments.md) | | Supported BI tool | Tableau | Tableau | | Use case | Helps users understand how models are used and reduces incidents. | Optimizes timeliness and reduces costs by running models when needed. | Check out the following sections for more information on visualizing and orchestrating downstream exposures: [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md) ###### [Set up and visualize downstream exposures](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md) [Set up downstream exposures automatically from dashboards to understand how models are used in downstream tools for a richer downstream lineage.](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/platform-integrations/orchestrate-exposures.md) ###### [Orchestrate downstream exposures](https://docs.getdbt.com/docs/platform-integrations/orchestrate-exposures.md) [Proactively refreshes the underlying data sources (like Tableau extracts) using the dbt scheduler during scheduled dbt jobs.](https://docs.getdbt.com/docs/platform-integrations/orchestrate-exposures.md) --- ### Visualize and orchestrate downstream exposures dbt platform | Enterprise, Enterprise+ Visualize and orchestrate downstream exposures in dbt to automatically generate exposures from dashboards and proactively refresh the underlying data sources (like Tableau extracts) during scheduled dbt jobs. The following table summarizes the differences between visualizing and orchestrating downstream exposures: | Info | Set up and visualize downstream exposures | Orchestrate downstream exposures [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") | | ----------------- | ---------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Purpose | Automatically brings downstream assets into your dbt lineage. | Proactively refreshes the underlying data sources during scheduled dbt jobs. | | Benefits | Provides visibility into data flow and dependencies. | Ensures BI tools always have up-to-date data without manual intervention. | | Location | Exposed in dbt [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) | Exposed in [dbt scheduler](https://docs.getdbt.com/docs/deploy/deployments.md) | | Supported BI tool | Tableau | Tableau | | Use case | Helps users understand how models are used and reduces incidents. | Optimizes timeliness and reduces costs by running models when needed. | Check out the following sections for more information on visualizing and orchestrating downstream exposures: [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md) ###### [Set up and visualize downstream exposures](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md) [Set up downstream exposures automatically from dashboards to understand how models are used in downstream tools for a richer downstream lineage.](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/platform-integrations/orchestrate-exposures.md) ###### [Orchestrate downstream exposures](https://docs.getdbt.com/docs/platform-integrations/orchestrate-exposures.md) [Proactively refreshes the underlying data sources (like Tableau extracts) using the dbt scheduler during scheduled dbt jobs.](https://docs.getdbt.com/docs/platform-integrations/orchestrate-exposures.md) --- ### Visualize downstream exposures dbt platform | Enterprise, Enterprise+ Downstream exposures integrate natively with Tableau (Power BI coming soon) and auto-generate downstream lineage in Catalog for a richer experience. As a data team, it’s critical that you have context into the downstream use cases and users of your data products. By leveraging downstream [exposures](https://docs.getdbt.com/docs/build/exposures.md) automatically, data teams can: * Gain a better understanding of how models are used in downstream analytics, improving governance and decision-making. * Reduce incidents and optimize workflows by linking upstream models to downstream dependencies. * Automate exposure tracking for supported BI tools, ensuring lineage is always up to date. * [Orchestrate exposures](https://docs.getdbt.com/docs/platform-integrations/orchestrate-exposures.md) to refresh the underlying data sources during scheduled dbt jobs, improving timeliness and reducing costs. Orchestrating exposures is essentially a way to ensure that your BI tools are updated regularly by using the [dbt job scheduler](https://docs.getdbt.com/docs/deploy/deployments.md). * For more info on the differences between visualizing and orchestrating exposures, see [Visualize and orchestrate downstream exposures](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures.md). To configure downstream exposures automatically from dashboards in Tableau, prerequisites, and more — refer to [Configure downstream exposures](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md). ##### Supported plans Downstream exposures is available on all dbt [Enterprise-tier plans](https://www.getdbt.com/pricing/). Currently, you can only connect to a single Tableau site on the same server. Tableau Server If you're using Tableau Server, you need to [allowlist dbt's IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for your dbt region. #### View downstream exposures After setting up downstream exposures in dbt, you can view them in [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) for a richer experience. Navigate to Catalog by selecting **Catalog** from the top-level navigation. You can view downstream exposures from a couple of places: * [Exposures menu](#exposures-menu) * [Exposure detail page](#exposure-detail-page) * [Project lineage](#project-lineage) ##### Exposures menu View all downstream exposures for a project from the Catalog sidebar: 1. In the sidebar, select your project. 2. Under the project, select **Exposure**. You will only see this option if you set up downstream exposures in [Tableau](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md#set-up-in-tableau) and [dbt](https://docs.getdbt.com/docs/platform-integrations/downstream-exposures-tableau.md#set-up-in-dbt-cloud). The **Exposures** table lists every exposure in the project so you can quickly access and manage them: * **Name**: The name of the exposure. * **Health**: The [data health signal](https://docs.getdbt.com/docs/explore/data-health-signals.md) of the exposure. * **Type**: The type of exposure, such as `dashboard` or `notebook`. * **Owner**: The owner of the exposure. * **Owner email**: The email address of the owner of the exposure. * **Integration**: The BI tool that the exposure is integrated with. * **Exposure mode**: The type of exposure defined: **Auto** or **Manual**. Auto exposures from Tableau appear alongside manual exposures [![View the Exposures table from the Catalog sidebar.](/img/docs/platform-integrations/auto-exposures/explorer-view-resources.png?v=2 "View the Exposures table from the Catalog sidebar.")](#)View the Exposures table from the Catalog sidebar. ##### Exposure detail page After you open the **Exposures** table ([previous section](#exposures-menu)), select an exposure name to open its detail page. On the exposure detail page, you can review metadata, [data health signals](https://docs.getdbt.com/docs/explore/data-health-signals.md), description, and lineage. For Tableau auto exposures, use **Open in Dashboard** to open the workbook in Tableau, or **Modify integration** to update your Tableau connection settings. [![View an exposure detail page in Catalog.](/img/docs/platform-integrations/auto-exposures/explorer-view-exposure-detail.png?v=2 "View an exposure detail page in Catalog.")](#)View an exposure detail page in Catalog. ##### Project lineage You can also view exposures from the **Project lineage** view, separate from the **Exposures** table: 1. In the sidebar, select your project. 2. Click **View lineage**. 3. Select an exposure node with the Tableau icon to view its details in the side panel. This view visualizes the dependencies and relationships in your project. For Tableau auto exposures, use **View in Tableau** or **Modify integration** from the side panel. [![View from the dbt Catalog in your Project lineage view, displayed with the Tableau icon.](/img/docs/platform-integrations/auto-exposures/explorer-lineage2.png?v=2 "View from the dbt Catalog in your Project lineage view, displayed with the Tableau icon.")](#)View from the dbt Catalog in your Project lineage view, displayed with the Tableau icon. [![View from the dbt Catalog in your Project lineage view, displayed with the Tableau icon.](/img/docs/platform-integrations/auto-exposures/explorer-lineage.png?v=2 "View from the dbt Catalog in your Project lineage view, displayed with the Tableau icon.")](#)View from the dbt Catalog in your Project lineage view, displayed with the Tableau icon. --- ### Webhooks for your jobs dbt platform | Starter, Enterprise, Enterprise+ With dbt, you can create outbound webhooks to send events (notifications) about your dbt jobs to your other systems. Your other systems can listen for (subscribe to) these events to further automate your workflows or to help trigger automation flows you have set up. A webhook is an HTTP-based callback function that allows event-driven communication between two different web applications. This allows you to get the latest information on your dbt jobs in real time. Without it, you would need to make API calls repeatedly to check if there are any updates that you need to account for (polling). Because of this, webhooks are also called *push APIs* or *reverse APIs* and are often used for infrastructure development. dbt sends a JSON payload to your application's endpoint URL when your webhook is triggered. You can send a [Slack](https://docs.getdbt.com/guides/zapier-slack.md) notification, a [Microsoft Teams](https://docs.getdbt.com/guides/zapier-ms-teams.md) notification, [open a PagerDuty incident](https://docs.getdbt.com/guides/serverless-pagerduty.md) when a dbt job fails. You can create webhooks for these events from the [dbt web-based UI](#create-a-webhook-subscription) and by using the [dbt API](#api-for-webhooks): * `job.run.started` — Run started. * `job.run.completed` — Run completed. This can be a run that has failed or succeeded. * `job.run.errored` — Run errored. dbt retries sending each event five times. dbt keeps a log of each webhook delivery for 7 days. Every webhook has its own **Recent Deliveries** section, which lists whether a delivery was successful or failed at a glance. A webhook in dbt has a timeout of 10 seconds. This means that if the endpoint doesn't respond within 10 seconds, the webhook processor will time out. This can result in a situation where the client responds successfully after the 10 second timeout and records a success status while the dbt webhooks system will interpret this as a failure. Videos If you're interested in course learning with videos, check out the [Webhooks on-demand course](https://learn.getdbt.com/courses/webhooks) from dbt Labs. You can also check out the free [dbt Fundamentals course](https://learn.getdbt.com/courses/dbt-fundamentals). #### Prerequisites * You have a dbt account that is on the [Starter or Enterprise-tier](https://www.getdbt.com/pricing/) plan. * For `write` access to webhooks: * **Enterprise-tier plans** — Permission sets are the same for both API service tokens and the dbt UI. You, or the API service token, must have the Account Admin, Admin, or Developer [permission set](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md). * **Starter plan accounts** — For the dbt user interface (UI), you need to have a [Developer license](https://docs.getdbt.com/docs/platform/manage-access/self-service-permissions.md). * You have a multi-tenant or an AWS single-tenant deployment model in dbt. For more information, refer to [Tenancy](https://docs.getdbt.com/docs/platform/about-platform/tenancy.md). * Your destination system supports [Authorization headers](#troubleshooting). #### Create a webhook subscription 1. Navigate to **Account settings** in dbt (by clicking your account name from the left side panel) 2. Go to the **Webhooks** section and click **Create webhook**. 3. To configure your new webhook: * **Webhook name** — Enter a name for your outbound webhook. * **Description** — Enter a description of the webhook. * **Events** — Choose the event you want to trigger this webhook. You can subscribe to more than one event. * **Jobs** — Specify the job(s) you want the webhook to trigger on. Or, you can leave this field empty for the webhook to trigger on all jobs in your account. By default, dbt configures your webhook at the account level. * **Endpoint** — Enter your application's endpoint URL, where dbt can send the event(s) to. 4. When done, click **Save**. dbt provides a secret token that you can use to [check for the authenticity of a webhook](#validate-a-webhook). It’s strongly recommended that you perform this check on your server to protect yourself from fake (spoofed) requests. #### Archived webhooks dbt automatically archives a webhook after 1,000 consecutive failed deliveries or 7 consecutive days of failed deliveries, whichever comes first. * On the **Account settings → Webhooks** page in dbt platform, a dismissible warning banner appears when one or more subscriptions are archived. * Archived subscriptions show an **Archived** badge in the status column instead of an HTTP status. * Hover over the badge for details on why dbt archived it and how to reactivate it. [![Webhooks page showing the archived subscription warning banner with the 'Archived; status badge](/img/docs/deploy/webhooks-archived.png?v=2 "Webhooks page showing the archived subscription warning banner with the 'Archived; status badge")](#)Webhooks page showing the archived subscription warning banner with the 'Archived; status badge To reactivate a webhook, use one of the following methods: * **dbt platform UI**: Update the webhook's endpoint URL. dbt automatically reactivates the webhook when the URL changes. * **REST API**: Send a `PUT` request to [Update a webhook](#update-a-webhook) and set `active` to `true`, or update `client_url` to a new endpoint URL. * **Terraform provider**: Set `active = true` and update `client_url` in your webhook resource. To find the appropriate dbt access URL for your region and plan, refer to [Regions & IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). ##### Differences between completed and errored webhook events The `job.run.errored` event is a subset of the `job.run.completed` events. If you subscribe to both, you will receive two notifications when your job encounters an error. However, dbt triggers the two events at different times: * `job.run.completed` — This event only fires once the job’s metadata and artifacts have been ingested and are available from the dbt Admin and Discovery APIs. * `job.run.errored` — This event fires immediately so the job’s metadata and artifacts might not have been ingested. This means that information might not be available for you to use. If your integration depends on data from the Admin API (such as accessing the logs from the run) or Discovery API (accessing model-by-model statuses), use the `job.run.completed` event and filter on `runStatus` or `runStatusCode`. If your integration doesn’t depend on additional data or if improved delivery performance is more important for you, use `job.run.errored` and build your integration to handle API calls that might not return data a short period at first. #### Validate a webhook You can use the secret token provided by dbt to validate that webhooks received by your endpoint were actually sent by dbt. Official webhooks will include the `Authorization` header that contains a SHA256 hash of the request body and uses the secret token as a key. An example for verifying the authenticity of the webhook in Python: ```python auth_header = request.headers.get('authorization', None) app_secret = os.environ['MY_DBT_CLOUD_AUTH_TOKEN'].encode('utf-8') signature = hmac.new(app_secret, request_body, hashlib.sha256).hexdigest() return signature == auth_header ``` Note that the destination system must support [Authorization headers](#troubleshooting) for the webhook to work correctly. You can test your endpoint's support by sending a request with curl and an Authorization header, like this: ```shell curl -H 'Authorization: 123' -X POST https://<your-webhook-endpoint> ``` #### Inspect HTTP requests When working with webhooks, it’s good practice to use tools like [RequestBin](https://requestbin.com/) and [Requestly](https://requestly.io/). These tools allow you to inspect your HTML requests, response payloads, and response headers so you can debug and test webhooks before incorporating them into your systems. #### Examples of JSON payloads An example of a webhook payload for a run that's started: ```json { "accountId": 1, "webhookId": "wsu_12345abcde", "eventId": "wev_2L6Z3l8uPedXKPq9D2nWbPIip7Z", "timestamp": "2023-01-31T19:28:15.742843678Z", "eventType": "job.run.started", "webhookName": "test", "data": { "jobId": "123", "jobName": "Daily Job (dbt build)", "runId": "12345", "environmentId": "1234", "environmentName": "Production", "dbtVersion": "1.0.0", "projectName": "Snowflake Github Demo", "projectId": "167194", "runStatus": "Running", "runStatusCode": 3, "runStatusMessage": "None", "runReason": "Kicked off from the UI by test@test.com", "runStartedAt": "2023-01-31T19:28:07Z" } } ``` An example of a webhook payload for a completed run: ```json { "accountId": 1, "webhookId": "wsu_12345abcde", "eventId": "wev_2L6ZDoilyiWzKkSA59Gmc2d7FDD", "timestamp": "2023-01-31T19:29:35.789265936Z", "eventType": "job.run.completed", "webhookName": "test", "data": { "jobId": "123", "jobName": "Daily Job (dbt build)", "runId": "12345", "environmentId": "1234", "environmentName": "Production", "dbtVersion": "1.0.0", "projectName": "Snowflake Github Demo", "projectId": "167194", "runStatus": "Success", "runStatusCode": 10, "runStatusMessage": "None", "runReason": "Kicked off from the UI by test@test.com", "runStartedAt": "2023-01-31T19:28:07Z", "runFinishedAt": "2023-01-31T19:29:32Z" } } ``` An example of a webhook payload for an errored run: ```json { "accountId": 1, "webhookId": "wsu_12345abcde", "eventId": "wev_2L6m5BggBw9uPNuSmtg4MUiW4Re", "timestamp": "2023-01-31T21:15:20.419714619Z", "eventType": "job.run.errored", "webhookName": "test", "data": { "jobId": "123", "jobName": "dbt Vault", "runId": "12345", "environmentId": "1234", "environmentName": "dbt Vault Demo", "dbtVersion": "1.0.0", "projectName": "Snowflake Github Demo", "projectId": "167194", "runStatus": "Errored", "runStatusCode": 20, "runStatusMessage": "None", "runReason": "Kicked off from the UI by test@test.com", "runStartedAt": "2023-01-31T21:14:41Z", "runErroredAt": "2023-01-31T21:15:20Z" } } ``` #### API for webhooks You can use the dbt API to create new webhooks that you want to subscribe to, get detailed information about your webhooks, and to manage the webhooks that are associated with your account. The following sections describe the API endpoints you can use for this. Access URLs dbt is hosted in multiple regions in the world and each region has a different access URL. People on Enterprise-tier plans can choose to have their account hosted in any one of these regions. For a complete list of available dbt access URLs, refer to [Regions & IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md). ##### List all webhook subscriptions List all webhooks that are available from a specific dbt account. ###### Request ```shell GET https://{your access URL}/api/v3/accounts/{account_id}/webhooks/subscriptions ``` ###### Path parameters | Name | Description | | ----------------- | ------------------------------------------------- | | `your access URL` | The login URL for your dbt account. | | `account_id` | The dbt account the webhooks are associated with. | ###### Response sample ```json { "data": [ { "id": "wsu_12345abcde", "account_identifier": "act_12345abcde", "name": "Webhook for jobs", "description": "A webhook for when jobs are started", "job_ids": [ "123", "321" ], "event_types": [ "job.run.started" ], "client_url": "https://test.com", "active": true, "created_at": "1675735768491774", "updated_at": "1675787482826757", "account_id": "123", "http_status_code": "0" }, { "id": "wsu_12345abcde", "account_identifier": "act_12345abcde", "name": "Notification Webhook", "description": "Webhook used to trigger notifications in Slack", "job_ids": [], "event_types": [ "job.run.completed", "job.run.started", "job.run.errored" ], "client_url": "https://test.com", "active": true, "created_at": "1674645300282836", "updated_at": "1675786085557224", "http_status_code": "410", "dispatched_at": "1675786085548538", "account_id": "123" } ], "status": { "code": 200 }, "extra": { "pagination": { "total_count": 2, "count": 2 }, "filters": { "offset": 0, "limit": 10 } } } ``` ###### Response schema | Name | Description | Possible Values | | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `data` | List of available webhooks for the specified dbt account ID. | | | `id` | The webhook ID. This is a universally unique identifier (UUID) that's unique across all regions, including multi-tenant and single-tenant | | | `account_identifier` | The unique identifier for *your* dbt account. | | | `name` | Name of the outbound webhook. | | | `description` | Description of the webhook. | | | `job_ids` | The specific jobs the webhook is set to trigger for. When the list is empty, the webhook is set to trigger for all jobs in your account; by default, dbt configures webhooks at the account level. | - Empty list<br />- List of job IDs | | `event_types` | The event type(s) the webhook is set to trigger on. | One or more of these: - `job.run.started`<br />- `job.run.completed`<br />- `job.run.errored` | | `client_url` | The endpoint URL for an application where dbt can send event(s) to. | | | `active` | A Boolean value indicating whether the webhook is active or not. | One of these: - `true`<br />- `false` | | `created_at` | Timestamp of when the webhook was created. | | | `updated_at` | Timestamp of when the webhook was last updated. | | | `http_status_code` | The latest HTTP status of the webhook. | Can be any [HTTP response status code](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status). If the value is `0`, that means the webhook has never been triggered. | | `dispatched_at` | Timestamp of when the webhook was last dispatched to the specified endpoint URL. | | | `account_id` | The dbt account ID. | | ##### Get details about a webhook Get detailed information about a specific webhook. ###### Request ```shell GET https://{your access URL}/api/v3/accounts/{account_id}/webhooks/subscription/{webhook_id} ``` ###### Path parameters | Name | Description | | ----------------- | ----------------------------------------------- | | `your access URL` | The login URL for your dbt account. | | `account_id` | The dbt account the webhook is associated with. | | `webhook_id` | The webhook you want detailed information on. | ###### Response sample ```json { "data": { "id": "wsu_12345abcde", "account_identifier": "act_12345abcde", "name": "Webhook for jobs", "description": "A webhook for when jobs are started", "event_types": [ "job.run.started" ], "client_url": "https://test.com", "active": true, "created_at": "1675789619690830", "updated_at": "1675793192536729", "dispatched_at": "1675793192533160", "account_id": "123", "job_ids": [], "http_status_code": "0" }, "status": { "code": 200 } } ``` ###### Response schema | Name | Description | Possible Values | | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `id` | The webhook ID. | | | `account_identifier` | The unique identifier for *your* dbt account. | | | `name` | Name of the outbound webhook. | | | `description` | Complete description of the webhook. | | | `event_types` | The event type the webhook is set to trigger on. | One or more of these: - `job.run.started`<br />- `job.run.completed`<br />- `job.run.errored` | | `client_url` | The endpoint URL for an application where dbt can send event(s) to. | | | `active` | A Boolean value indicating whether the webhook is active or not. | One of these: - `true`<br />- `false` | | `created_at` | Timestamp of when the webhook was created. | | | `updated_at` | Timestamp of when the webhook was last updated. | | | `dispatched_at` | Timestamp of when the webhook was last dispatched to the specified endpoint URL. | | | `account_id` | The dbt account ID. | | | `job_ids` | The specific jobs the webhook is set to trigger for. When the list is empty, the webhook is set to trigger for all jobs in your account; by default, dbt configures webhooks at the account level. | One of these: - Empty list<br />- List of job IDs | | `http_status_code` | The latest HTTP status of the webhook. | Can be any [HTTP response status code](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status). If the value is `0`, that means the webhook has never been triggered. | ##### Create a new webhook subscription Create a new outbound webhook and specify the endpoint URL that will be subscribing (listening) to the webhook's events. ###### Request sample ```shell POST https://{your access URL}/api/v3/accounts/{account_id}/webhooks/subscriptions ``` ```json { "event_types": [ "job.run.started" ], "name": "Webhook for jobs", "client_url": "https://test.com", "active": true, "description": "A webhook for when jobs are started", "job_ids": [ 123, 321 ] } ``` ###### Path parameters | Name | Description | | ----------------- | ----------------------------------------------- | | `your access URL` | The login URL for your dbt account. | | `account_id` | The dbt account the webhook is associated with. | ###### Request parameters | Name | Description | Possible Values | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------- | | `event_types` | Enter the event you want to trigger this webhook. You can subscribe to more than one event. | One or more of these: - `job.run.started`<br />- `job.run.completed`<br />- `job.run.errored` | | `name` | Enter the name of your webhook. | | | `client_url` | Enter your application's endpoint URL, where dbt can send the event(s) to. | | | `active` | Enter a Boolean value to indicate whether your webhook is active or not. | One of these: - `true`<br />- `false` | | `description` | Enter a description of your webhook. | | | `job_ids` | Enter the specific jobs you want the webhook to trigger on or you can leave this parameter as an empty list. If this is an empty list, the webhook is set to trigger for all jobs in your account; by default, dbt configures webhooks at the account level. | One of these: - Empty list<br />- List of job IDs | ###### Response sample ```json { "data": { "id": "wsu_12345abcde", "account_identifier": "act_12345abcde", "name": "Webhook for jobs", "description": "A webhook for when jobs are started", "job_ids": [ "123", "321" ], "event_types": [ "job.run.started" ], "client_url": "https://test.com", "hmac_secret": "12345abcde", "active": true, "created_at": "1675795644808877", "updated_at": "1675795644808877", "account_id": "123", "http_status_code": "0" }, "status": { "code": 201 } } ``` ###### Response schema | Name | Description | Possible Values | | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `id` | The webhook ID. | | | `account_identifier` | The unique identifier for *your* dbt account. | | | `name` | Name of the outbound webhook. | | | `description` | Complete description of the webhook. | | | `job_ids` | The specific jobs the webhook is set to trigger for. When the list is empty, the webhook is set to trigger for all jobs in your account; by default, dbt configures webhooks at the account level. | One of these: - Empty list<br />- List of job IDs | | `event_types` | The event type the webhook is set to trigger on. | One or more of these: - `job.run.started`<br />- `job.run.completed`<br />- `job.run.errored` | | `client_url` | The endpoint URL for an application where dbt can send event(s) to. | | | `hmac_secret` | The secret key for your new webhook. You can use this key to [validate the authenticity of this webhook](#validate-a-webhook). | | | `active` | A Boolean value indicating whether the webhook is active or not. | One of these: - `true`<br />- `false` | | `created_at` | Timestamp of when the webhook was created. | | | `updated_at` | Timestamp of when the webhook was last updated. | | | `account_id` | The dbt account ID. | | | `http_status_code` | The latest HTTP status of the webhook. | Can be any [HTTP response status code](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status). If the value is `0`, that means the webhook has never been triggered. | ##### Update a webhook Update the configuration details for a specific webhook. ###### Request sample ```shell PUT https://{your access URL}/api/v3/accounts/{account_id}/webhooks/subscription/{webhook_id} ``` ```json { "event_types": [ "job.run.started" ], "name": "Webhook for jobs", "client_url": "https://test.com", "active": true, "description": "A webhook for when jobs are started", "job_ids": [ 123, 321 ] } ``` ###### Path parameters | Name | Description | | ----------------- | ----------------------------------------------- | | `your access URL` | The login URL for your dbt account. | | `account_id` | The dbt account the webhook is associated with. | | `webhook_id` | The webhook you want to update. | ###### Request parameters | Name | Description | Possible Values | | ------------- | -------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | | `event_types` | Update the event type the webhook is set to trigger on. You can subscribe to more than one. | One or more of these: - `job.run.started`<br />- `job.run.completed`<br />- `job.run.errored` | | `name` | Change the name of your webhook. | | | `client_url` | Update the endpoint URL for an application where dbt can send event(s) to. | | | `active` | Change the Boolean value indicating whether the webhook is active or not. | One of these: - `true`<br />- `false` | | `description` | Update the webhook's description. | | | `job_ids` | Change which jobs you want the webhook to trigger for. Or, you can use an empty list to trigger it for all jobs in your account. | One of these: - Empty list<br />- List of job IDs | ###### Response sample ```json { "data": { "id": "wsu_12345abcde", "account_identifier": "act_12345abcde", "name": "Webhook for jobs", "description": "A webhook for when jobs are started", "job_ids": [ "123" ], "event_types": [ "job.run.started" ], "client_url": "https://test.com", "active": true, "created_at": "1675798888416144", "updated_at": "1675804719037018", "http_status_code": "200", "account_id": "123" }, "status": { "code": 200 } } ``` ###### Response schema | Name | Description | Possible Values | | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `id` | The webhook ID. | | | `account_identifier` | The unique identifier for *your* dbt account. | | | `name` | Name of the outbound webhook. | | | `description` | Complete description of the webhook. | | | `job_ids` | The specific jobs the webhook is set to trigger for. When the list is empty, the webhook is set to trigger for all jobs in your account; by default, dbt configures webhooks at the account level. | One of these: - Empty list<br />- List of job IDs | | `event_types` | The event type the webhook is set to trigger on. | One or more of these: - `job.run.started`<br />- `job.run.completed`<br />- `job.run.errored` | | `client_url` | The endpoint URL for an application where dbt can send event(s) to. | | | `active` | A Boolean value indicating whether the webhook is active or not. | One of these: - `true`<br />- `false` | | `created_at` | Timestamp of when the webhook was created. | | | `updated_at` | Timestamp of when the webhook was last updated. | | | `http_status_code` | The latest HTTP status of the webhook. | Can be any [HTTP response status code](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status). If the value is `0`, that means the webhook has never been triggered. | | `account_id` | The dbt account ID. | | ##### Test a webhook Test a specific webhook. ###### Request ```shell GET https://{your access URL}/api/v3/accounts/{account_id}/webhooks/subscription/{webhook_id}/test ``` ###### Path parameters | Name | Description | | ----------------- | ----------------------------------------------- | | `your access URL` | The login URL for your dbt account. | | `account_id` | The dbt account the webhook is associated with. | | `webhook_id` | The webhook you want to test. | ###### Response sample ```json { "data": { "verification_error": null, "verification_status_code": "200" }, "status": { "code": 200 } } ``` ##### Delete a webhook Delete a specific webhook. ###### Request ```shell DELETE https://{your access URL}/api/v3/accounts/{account_id}/webhooks/subscription/{webhook_id} ``` ###### Path parameters | Name | Description | | ----------------- | ----------------------------------------------- | | `your access URL` | The login URL for your dbt account. | | `account_id` | The dbt account the webhook is associated with. | | `webhook_id` | The webhook you want to delete. | ###### Response sample ```json { "data": { "id": "wsu_12345abcde" }, "status": { "code": 200, "is_success": true } } ``` #### Related docs * [dbt CI](https://docs.getdbt.com/docs/deploy/continuous-integration.md) * [Use dbt's webhooks with other SaaS apps](https://docs.getdbt.com/guides.md?tags=Webhooks) #### Troubleshooting If your destination system isn't receiving dbt webhooks, ensure it allows Authorization headers. dbt webhooks send an Authorization header, and if your endpoint doesn't support this, it may be incompatible. Services like Azure Logic Apps and Power Automate may not accept Authorization headers, so they won't work with dbt webhooks. You can test your endpoint's support by sending a request with curl and an Authorization header, like this: ```shell curl -H 'Authorization: 123' -X POST https://<your-webhook-endpoint> ``` --- ### Weekly dbt single-tenant release notes dbt platform | Free dbt Single-tenant release notes for weekly updates. Release notes fall into one of these categories: * **New:** New products and features * **Enhancement:** Performance improvements and feature enhancements * **Fix:** Bug and security fixes * **Behavior change:** A change to existing behavior that doesn't fit into the other categories, such as feature deprecations or changes to default settings Release notes are grouped by date for single-tenant environments. ![RSS](/img/fontawesome/rss.svg)Subscribe to release note updates via [RSS](https://docs.getdbt.com/assets/files/release-notes-st-rss-29e8fb8f6ac267c1314933ff61ba962e.xml), [Atom](https://docs.getdbt.com/assets/files/release-notes-st-atom-d2de630ac79be192b2b299ce34038a69.xml), or [JSON Feed](https://docs.getdbt.com/assets/files/release-notes-st-rss-5c0699620f3e7df021c631289f0d1443.json). #### September 2, 2026 #### Enhancements ##### APIs, Identity, and Administration * **Analyst Read permission set available to all accounts**: The [Analyst Read](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#analyst-read) permission set is now available to all accounts without requiring a feature flag. You can assign it to groups so read-only users can view Catalog and project configuration such as connections, environments, and Semantic Layer settings. ##### Semantic Layer * **Clearer Snowflake authentication and permission errors**: When you connect to Snowflake through the Semantic Layer, authentication failures and permission errors now return distinct messages prefixed with `[WAREHOUSE_AUTHENTICATION_FAILED]` or `[WAREHOUSE_PERMISSION_DENIED]`, so you can tell credential issues apart from missing grants. #### Fixes ##### Orchestration and run status * **Clearer invalid cron expression errors**: Saving a job with an invalid day-of-month value such as `*,L` now shows a validation error and prevents the broken schedule from being saved. Use either `*` or `L` in the day-of-month field, not both. #### Behavior change ##### APIs, Identity, and Administration * **Unscoped group and service token permissions rejected**: When you assign a project-scoped permission set to a group or service token, you must now specify either all projects or a specific project. Requests that leave project scope unset return a `400` error. Account-level permission sets such as Billing Admin and Notification Manager are not affected. Existing legacy assignments continue to work until you change them. #### August 26, 2026 #### New ##### Orchestration and run status * **dbt State explain tab on run details**: A new "State explain" tab on the run details page shows dbt State's decision for each model in a run (rebuilt, reused, or cloned), with expandable details, search, and Comma-Separated Values (CSV) download. You can use this tab to investigate why each model was rebuilt or reused. Contact your account manager to enable. #### Enhancements ##### dbt platform * **Clearer billing admin guidance for non-admins**: Non-admin users now see an "Ask an admin to enable" message on the dbt Wizard and dbt State cards in Billing & Usage when a trial is available but they lack permission to start it, instead of a blank space. The same message appears if a non-admin tries to start a trial from a dbt State or dbt Wizard link. ##### Orchestration and run status * **dbt State available on more release tracks**: dbt State is now available for jobs running on the Compatible, Fusion Extended, and Fusion Fallback release tracks, in addition to previously supported tracks. * **Corrected Fusion release track names**: The display names for dbt Fusion release tracks are now "Fusion Stable" and "Fusion Nightly" instead of the previous reversed labels "Stable Fusion" and "Nightly Fusion". * **Large compare results in pull request comments**: When compare results are larger than 50 MB, pull request comments now show a "too large to summarize" notice with a link to the full compare report, instead of failing with no message. ##### Studio IDE * **Correct browser tab title**: The Studio IDE browser tab now displays "dbt Studio" instead of a generic editor title. ##### Integrations * **More reliable MCP OAuth sign-in**: When you connect an MCP client with OAuth, more clients can now complete sign-in successfully. #### August 19, 2026 #### Enhancements ##### Studio IDE * **Console tab persists across sessions**: New sessions open on the Wizard tab when available, and the Studio IDE remembers your last-used tab for each project so you can pick up where you left off. ##### Catalog * **Exact model relation name in the Discovery API**: A new `relationName` field on the `ModelAppliedStateNode` and `ModelAppliedStateNestedNode` GraphQL types exposes the fully-qualified, adapter-rendered relation name (for example, `"database"."schema"."model_name"`) from the last successful model build. #### August 12, 2026 #### Enhancements ##### dbt AI and agents * **Auto-expanding Wizard chat input**: The Wizard chat input grows vertically as you type or paste text, and shrinks back when content is removed. ##### APIs, Identity, and Administration * **Job read access is now included in the `account:read` OAuth scope**: Applications authorized with `account:read` can now read job data without also requesting the `jobs:run` scope. #### August 5, 2026 #### New ##### dbt AI and agents * **dbt Wizard home tab**: [The dbt Wizard home tab in dbt platform](https://docs.getdbt.com/docs/platform/wizard-home.md) is now available in public preview. You can build and change dbt projects through natural language, with inline diffs, DAG previews, and validation built in. #### Enhancements ##### dbt AI and agents * **Auto-open diff preview in edit-automatically mode**: When dbt Wizard edits files automatically, the diff preview side pane now opens immediately so you can review changes without an extra click. When the side pane is not available, for example in non-fullscreen Studio IDE, the diff falls back to an inline card. ##### Orchestration and run status * **Run history refreshes automatically after a trigger**: After you trigger a run or rerun, the run history list now polls every 2.5 seconds until the new run appears, eliminating the need for a manual page reload. Polling stops automatically once the run is visible or after 45 seconds. #### Behavior change ##### Catalog * **30-day model staleness removed from health criteria**: Models are no longer flagged as unhealthy solely because they have not been rebuilt in the past 30 days. The "Stale state" warning banner no longer appears on resource detail pages, and models whose only health issue was staleness now show as healthy in the Trust Signals badge. Source staleness is unchanged and continues to surface as a health issue for sources. #### July 29, 2026 #### New ##### Semantic Layer * **External OAuth for Redshift with AWS IAM Identity Center**: Semantic Layer development connections to Redshift now support external OAuth using Okta or Microsoft Entra with AWS IAM Identity Center, in addition to username and password. #### Enhancements ##### APIs, Identity, and Administration * **Clearer SCIM error messages**: System for Cross-domain Identity Management (SCIM) API errors now include the user email addresses that caused seat or license failures, so you can identify which users blocked provisioning. #### Behavior change ##### APIs, Identity, and Administration * **GraphQL complexity limit now enforced**: Semantic Layer GraphQL queries that exceed the complexity limit of 200,000 now return an error instead of completing with a warning. If you hit this error, request fewer fields, use pagination, narrow your filters, or split the query into smaller ones. #### July 22, 2026 #### New ##### dbt AI and agents * **Unified `get_node_details` MCP tool**: The [dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/mcp-available-tools.md#discovery) now uses one `get_node_details` tool for all resource types. The older type-specific tools are deprecated and will be removed in a future release. ##### Orchestration and run status * **Job deactivation reason in banners**: When a job is deactivated, the banner now shows a specific reason — repeated run failures, account inactivity, or a generic fallback — with tailored reactivation instructions for each case. #### Enhancements ##### dbt AI and agents * **Reliability improvements for wizard platform**: The agent now automatically retries transient LLM provider failures — network timeouts, rate limits, and server errors — with exponential backoff, so brief provider blips are less likely to surface as errors during your session. * **Unlimited client tool loop iterations**: Client tool loops now run until the agent finishes rather than stopping after 50 iterations, eliminating premature termination of long-running agentic workflows. * **Copy button on code blocks in dbt Wizard**: Code blocks in dbt Wizard responses now include a Copy button on hover, so you can reuse generated SQL or YAML more easily. * **Clearer error for Bring-Your-Own-Key models that don't support embeddings**: When a Bring-Your-Own-Key (BYOK) OpenAI model is configured with a deployment that does not support embeddings (for example, a `gpt-4o` Azure deployment), the similar models feature now returns an actionable error message prompting you to use a text-embedding model instead of a generic internal error. ##### Catalog * **Directional lineage filtering in `get_lineage`**: The `get_lineage` tool now accepts a `direction` parameter (`upstream`, `downstream`, or `both`) to narrow results to only ancestors or only descendants of a target node, reducing response size for large graphs. The response also now includes a `description` field on each returned node. * **Metric filtering by metadata in `list_metrics`**: The `list_metrics` tool now accepts a `meta_filter` parameter to restrict results to metrics whose `config.meta` contains specified key-value pairs (for example, `{"agent_accessible": true}`), keeping result sets small enough to preserve description and metadata in the response. * **Health status filtering for applied models**: The `ModelAppliedFilter` input type now includes a `health` field, letting you filter applied models by health status (`unknown`, `degraded`, `caution`, or `healthy`) directly in the Discovery API. * **`warn` run status surfaced in model execution info**: The `RunStatus` enum and the `lastRunStatus` field on model execution information now include `warn`, so models whose last run completed with warnings correctly reflect that status. * **Warn run status available as a filter**: The run status filter panel now includes a **Warn** option alongside **Success**, **Error**, **Skipped**, and **Reused**. * **Default search environment matches page context**: When searching from within a project environment route (for example, a Staging page), the Catalog search now defaults the environment filter to that environment type rather than always defaulting to Production. * **Redesigned search result cards**: A redesigned search result card replaces tooltip-based match pills with inline expandable snippets for columns, tags, descriptions, and code matches. Please contact your account manager to enable. ##### Insights * **Snowflake Adaptive warehouse cost support**: Cost Insights can now attribute query costs to models run on Snowflake Adaptive warehouses using the `QUERY_METERING_HISTORY` table. Without access to this table, Adaptive warehouse queries were previously recorded as $0. The connection test now also checks and reports on `QUERY_METERING_HISTORY` access so you can diagnose missing attribution before it affects cost data. Please contact your account manager to enable. ##### Studio IDE * **Fusion Stable is now the default track**: The `latest-fusion` release track is now Fusion Stable across all settings. Existing configurations have been updated automatically. No action is needed. * **Environments already on Fusion no longer see upgrade checkbox**: On the Enable Fusion Environments page, environments already running Fusion now show a disabled checkbox, preventing unnecessary saves. * **More specific error messages on failed Fusion environment upgrades**: When saving a Fusion upgrade fails, the platform now displays the top-level user message from the API instead of internal field-level error details. * **Faster command status updates in Studio IDE**: The command panel now shows live status updates as commands run, so you see progress sooner without waiting for a refresh. ##### dbt platform * **Consumption pool card renamed and repositioned**: The "Committed spend" card is now labeled "Consumption pool" with copy explaining that usage-based features like dbt State draw from it. The card now appears between the current plan metric tiles and the product-specific sections on billing Overview and usage tab pages. * **dbt State DATT chart shows billable and free usage separately**: The Daily Active Target Tables (DATTs) chart now stacks billable and free series, so trial users whose usage is entirely free see real bars instead of an empty chart. ##### Orchestration and run status * **Reduced out-of-memory rates in Fusion**: Memory-tuning optimizations are now applied automatically to all Fusion runs, reducing out-of-memory kill rates and improving overall uptime. #### Fixes * **More reliable Claude responses**: Claude-backed agents can return longer answers and handle some previously broken interactions more reliably. #### July 15, 2026 #### New ##### Orchestration and run status * **Hybrid job type**: You can now create hybrid jobs to track runs triggered by an external orchestrator. Hybrid jobs have a simplified setup that omits execution steps, triggers, advanced settings, and cost-optimization controls. They display **Externally triggered** as their next-run schedule and are available only for projects configured as [Hybrid projects](https://docs.getdbt.com/docs/deploy/hybrid-projects.md). #### Enhancements ##### Orchestration and run status * **Faster linting for Fusion-version runs**: Runs using a Fusion dbt version now invoke the built-in [`dbt lint`](https://docs.getdbt.com/reference/commands/lint.md?version=2.0) command instead of SQLFluff. Fusion virtual environments do not include SQLFluff, so linting now works for all Fusion-version runs and runs faster. ##### dbt AI and agents * **Compaction indicator during context optimization**: When the agent compresses conversation context in the background, a spinner labeled **Optimizing conversation context…** now appears in the chat area. Submitting new messages and stopping the agent are disabled while compaction is in progress to prevent conflicts. * **Wizard unavailable screen replaces generic "not enabled" message**: When [dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-platform.md) is unavailable (not activated, trial expired, or spend limit reached), Studio IDE now shows a dedicated screen with the specific reason and an appropriate action instead of a generic message. ##### APIs, Identity, and Administration * **Server-side user search and pagination in account settings**: The users table, group member lists, and user edit drawer now search, filter, sort, and paginate server-side. On large accounts, all users are findable by name, email, or license type, group member search no longer misses results beyond the first page, and users beyond the first page can be opened and edited in the user edit drawer. ##### dbt platform * **Global account discovery generally available**: The **Enable global account discovery** setting on the **Account settings** page is now visible to all entitled accounts without requiring a feature flag. You can allow or restrict account discovery from [Account settings](https://docs.getdbt.com/docs/platform/account-settings.md#enable-global-account-discovery). * **Connection overrides visible in profile view mode**: Credential-level [connection overrides](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) (such as Databricks catalog, Snowflake warehouse, role, and database) are now surfaced as a read-only **Connection overrides** section in the profile details view, without requiring you to open the edit form. #### Fixes ##### Orchestration and run status * **Runs no longer stuck in "running" after OOM kill**: When a run pod is Out of Memory (OOM)-killed and restarted, the platform now passes the correct status code and message to the config API so the run transitions to a failed state in the dbt platform UI instead of remaining **running** indefinitely. ##### Semantic Layer * **More reliable SSH tunnel connections**: The Secure Shell (SSH) connection and authentication timeouts for Semantic Layer data platform connections are now 30 seconds (previously 1 second). If your bastion host or network path has higher latency, you will no longer experience deterministic connection failures. Refer to [Set up the Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md) for more information. * **Compile SQL rejects oversized query shapes**: Some types of Compile SQL queries are now rejected if they are too complex. If a request fails with a validation error, try reducing the number of metrics or group-by dimensions in the query. #### Behavior change ##### APIs, Identity, and Administration * **Service token creation now requires service token authentication**: You can no longer create a [service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) using an account-scoped [personal access token](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) (PAT). Requests authenticated with a PAT now return a `400` error. Use a service token to create new ones instead. #### July 13, 2026 #### Behavior change ##### Billing * **Improved billing email alerts**: You’ll now receive email alerts when you’re drawing down on your consumption — whether that’s from models built or newer consumption features like dbt State. #### July 8, 2026 #### Enhancements ##### dbt AI and agents * **User list search and group filtering**: The user listing API now supports a `search` parameter (case-insensitive substring match across email and name) and a `group_id` parameter to retrieve only members of a specific group, enabling paginated group-member lookups for large accounts. * **Expanded Analyst Read permission set**: The [Analyst Read](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#analyst-read) permission set now includes the project and account read permissions analysts need to browse catalog and configuration without write access. #### Fixes ##### Orchestration and run status * **Clearer errors for malformed dbt commands**: A dbt step with invalid command syntax, such as an unclosed quote, now returns a user-facing syntax error instead of a generic unhandled exception. #### July 1, 2026 #### Enhancements ##### APIs, Identity, and Administration * **SCIM user responses include standard group fields**: User responses from SCIM endpoints now include the standard `value` (group ID) and `display` (group display name) fields alongside the existing `id` and `displayName` fields, improving compatibility with SCIM spec consumers. * **Credentials page accessible with `user_credential_write` permission**: The credentials list and detail pages are now accessible to users who have `user_credential_write` permission even without `credentials_read`, supporting credential self-service flows. #### Fixes ##### dbt platform * **Email notifications accept addresses with special characters**: External email notification addresses with valid but non-standard local-part characters (for example, ampersands in `ops&alerts@example.com`) are now accepted instead of being rejected by the validator. ##### Orchestration and run status * **Clearer errors for oversized environment variable payloads**: Oversized custom environment variables now fail at run start with a clear error that names the largest offender, instead of crashing mid-run. Reduce the variable size and retry. ##### dbt AI and agents * **More reliable dbt Wizard conversations**: dbt Wizard now recovers automatically from transient errors that could previously interrupt a conversation, so long threads keep working. * **Accurate job and run scoping in dbt Wizard**: dbt Wizard now correctly scopes job and run investigations to your current project instead of returning results across your entire account. You no longer see unrelated jobs from other projects when asking dbt Wizard to investigate a run. #### Behavior change ##### dbt platform * **"Development credentials" renamed to "User credentials"**: All user-facing labels, section headings, tooltip text, and in-app messages that previously referred to "development credentials" now use "user credentials." * **Versionless migration banner removed**: The banner notifying users about migration to versionless dbt has been removed from the notification stack. #### June 24, 2026 #### New ##### Insights * **Cost breakdown by job**: Cost Insights now includes a Jobs table view alongside the existing all-models view. Use the **All**/**Jobs** toggle to switch between a per-model breakdown and a per-job summary, and select **Download** to export the active view as a comma-separated values (CSV) file. ##### Run Logs * **Download OpenTelemetry logs**: You can download OpenTelemetry (OTel) logs for individual dbt command steps in Fusion job runs. ##### APIs, Identity, and Administration * **Run history now scoped to projects**: You can now view live run, step, and log data scoped to a specific project. Results support filtering by step status and log type, with consistent pagination across all views. * **Presigned log download URLs**: Logs for completed run steps are now downloaded directly from storage rather than streamed through the service, improving download reliability and performance. Download links expire after 15 minutes. #### Enhancements ##### Studio IDE * **Find in files**: The [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/ide-user-interface.md#search-your-project) now includes search and replace functionality and a command palette, enabling you to quickly find and replace text across your project, navigate files, jump to symbols, and run IDE configuration commands. ##### Catalog * **Filter assets by run status**: You can now filter Catalog search results by the most recent run status of an asset — success, error, or skipped — making it easier to spot and investigate assets that may need attention. * **Model metadata preserved after failed or skipped runs:** Lineage, tests, descriptions, and other model metadata now persist correctly even when a run fails, is skipped, or reuses a prior result. Previously, these runs could overwrite stored metadata, causing tests to detach from their models and lineage to disappear. ##### dbt State * **dbt State now works on CI and merge jobs**: You can now enable the dbt State cost-optimization option on CI and merge jobs, not just deploy jobs. Previously this returned a validation error. * **dbt State credential management simplified**: Studio IDE now uses account-level dbt State credentials, removing the per-user provisioning step that previously ran on first use. ##### Semantic Layer * **Longer query timeout for Semantic Layer**: The query timeout has been doubled from 10 minutes to 20 minutes, reducing timeout errors for long-running queries. ##### APIs, Identity, and Administration * **[Administrative API v3](https://docs.getdbt.com/dbt-cloud/api-v3) now supports private endpoints**: [`list`](https://docs.getdbt.com/dbt-cloud/api-v3?version=2.0#/operations/List%20Private%20Endpoints), [`create`](https://docs.getdbt.com/dbt-cloud/api-v3?version=2.0#/operations/Create%20Private%20Endpoint), [`retrieve`](https://docs.getdbt.com/dbt-cloud/api-v3?version=2.0#/operations/Retrieve%20Private%20Endpoint), [`update`](https://docs.getdbt.com/dbt-cloud/api-v3?version=2.0#/operations/Update%20Private%20Endpoint), and [`delete`](https://docs.getdbt.com/dbt-cloud/api-v3?version=2.0#/operations/Delete%20Private%20Endpoint). Use these endpoints to manage private connectivity programmatically. * **Clearer error messages for service outages**: When a third-party service, such as a data warehouse, is temporarily unavailable, the dbt platform now returns a descriptive error message instead of a generic one, making it easier to diagnose connection issues. ##### Integrations * **BigQuery Workload Identity Federation falls back to connection impersonation**: When a BigQuery Workload Identity Federation (WIF) credential has no explicit `service_account_impersonation_url`, the platform now derives the URL from the connection-level `impersonate_service_account` field. This supports the Terraform `dbtcloud_global_connection` configuration pattern where impersonation is defined on the connection rather than the credential. * **Snowflake connection hostname normalized for TLS**: Snowflake account identifiers containing underscores (for example, `fdr_apac_dev`) are now hyphenated when building connection hostnames and OAuth URLs, preventing Transport Layer Security (TLS) hostname verification failures against Snowflake's wildcard certificate. * **Google SSO retries on transient token endpoint errors**: Google Workspace Single Sign-On (SSO) group refresh now retries once on 5xx responses from the Google token endpoint before surfacing an authentication error, reducing sign-in failures caused by transient Google API outages. * **Databricks OAuth retries on transient token endpoint errors**: The Databricks OAuth token refresh path now retries once on 5xx responses before surfacing a retryable error, making profile generation for Databricks OAuth connections more resilient to short Databricks outages. ##### dbt AI and agents * **dbt Model Context Protocol (MCP) semantic search for related models**: The `get_related_models` tool is now available in multi-project agent contexts, allowing the agent to search for semantically similar models across projects by resolving each project's production environment automatically. #### Fixes ##### Studio IDE * **Clearer errors for unconfigured development credentials**: Studio IDE now returns an actionable error when development credentials are not configured for an environment. * **More precise error responses for development environment setup**: The development environment endpoint now returns distinct HTTP status codes for missing project configuration (400), unconfigured development credentials (412), permission errors (401), and retrieval timeouts (504), rather than mapping multiple failure modes to the same error response. ##### APIs, Identity, and Administration * **Connection test restricted to authorized environments**: The connection test endpoint now validates that the environment ID in the request belongs to the account and project, returning a 404 for unrecognized environment IDs instead of silently proceeding. * **SSO redirect preserved after session expiry on logout**: When an unauthenticated user visits the logout endpoint on a multi-tenant cell, the platform now resolves the correct Single Sign-On (SSO)-mandatory provider from the request host and redirects to `/enterprise-login/<slug>` instead of stranding them on `/login/`. * **Publications handler handles cloud-config timeouts gracefully**: When a `GetCrossProjectEnvironment` or `ListEnvironments` call to cloud-config exceeds its deadline, the publications handler now returns an `UNAVAILABLE` status to the caller instead of propagating an unhandled error. You can retry the request after a short delay. ##### Semantic Layer * **More reliable Semantic Layer job result retrieval**: Semantic Layer job and paginated query results now deserialize more reliably, reducing failures when retrieving query results. * **Fixed database write errors for cache timestamps**: Resolves an issue where the cache engine could fail to persist timestamps because timezone-aware datetimes cannot be encoded into `TIMESTAMP WITHOUT TIME ZONE` columns by `asyncpg`. The fix ensures timezone-naive UTC timestamps are used for all database writes. * **Result-too-large Semantic Layer errors now return HTTP 400**: Semantic Layer queries that exceed the result size limit now return a bad request error instead of an internal server error. ##### dbt Copilot and agents * **Safer handling of non-JSON OpenAI error responses**: Error handling for OpenAI `BadRequestError` now gracefully handles responses with non-JSON bodies, preventing an unhandled exception when parsing the error code. You should see a proper error rather than an internal server error in these cases. #### Behavior change ##### APIs, Identity, and Administration * **Account-scoped credential reads enforced for Personal Access Tokens**: The user credentials endpoints now enforce account scoping consistently when a request uses an account-scoped Personal Access Token (PAT). Requests that previously returned credentials outside the token's account now return a 404. #### June 17, 2026 #### Enhancements ##### dbt AI and agents * **Live streaming for Wizard dbt command output**: [dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-platform.md) tool calls for dbt command invocations now stream their output live in chat, in both the Studio IDE and [Wizard home](https://docs.getdbt.com/docs/platform/wizard-home.md). * **OAuth scopes declared in Model Context Protocol resource metadata**: The Model Context Protocol (MCP) protected resource metadata endpoint now advertises the OAuth scopes it supports (`offline_access`, `account:read`, `projects:query`, `catalog:read`, `projects:develop`, and `jobs:run`). MCP clients that perform dynamic capability discovery can now request the correct scopes automatically. ##### dbt platform * **dbt State in development**: [Enable dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-setup.md#enabling-dbt-state-in-studio) in development environments, or override it per user in **User development settings**. Requires dbt State to be enabled on the account by an admin. * **dbt State disabled for unsupported warehouse adapters on jobs**: The dbt State checkbox in job settings is now disabled when the job's environment uses an unsupported warehouse adapter. A help icon displays a tooltip explaining the limitation, and the feature is automatically removed from the job payload on save. #### June 10, 2026 #### New ##### dbt Copilot and agents * **dbt documentation search and retrieval tools**: The [remote dbt MCP server](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-remote.md) now includes `search_product_docs` and `get_product_doc_pages`. Available in and in agent clients connected to remote dbt MCP. Refer to the [product docs toolset](https://docs.getdbt.com/docs/dbt-ai/mcp-available-tools.md?version=2.0#product-docs). * **Dimension values lookup tool**: A new `get_dimension_values` MCP tool lets agents retrieve distinct values for a given Semantic Layer dimension, optionally scoped to specific metrics. Use this to discover valid filter values (for example, available regions or order statuses) before building a `where` clause in a `query_metrics` call. #### Enhancements ##### Studio IDE * **Workspace file operations API**: Adds public Studio file operation endpoints for `stat`, `get`, `put`, `list directory`, `delete`, `mkdir`, and `rename` under `/api/ide/v3/{environment_id}/files/`. File paths are passed as query parameters to avoid user paths appearing in traces. * **Environment status endpoint**: Adds a `/api/ide/v3/{environment_id}/status` endpoint that returns the dbt version and Fusion status for a development environment, allowing Studio to display version information without additional API calls. #### Behavior change ##### dbt platform * **Password management updates**: The in-page password change form has been removed from profile security settings, and the "Forgot password?" link no longer appears on the sign-in page. Password resets are now handled through the email-based reset flow, which can be initiated from your profile settings. #### June 3, 2026 #### New ##### dbt Copilot and agents * **Debug with Copilot from run and job surfaces**: A new "Debug with Copilot" button appears on failed run detail pages, runs lists, job details, environment runs, and the project home activity feed. Clicking it opens dbt Copilot or the full-page Wizard to investigate and debug the failed run. Please contact your account manager to enable. ##### dbt platform * **dbt State trial progress, stats, and usage on settings page**: The dbt State settings page now shows a trial progress bar (days elapsed of 30), monthly model reuse stats (models reused, build reduction percentage, and query run time reduction), and a model build chart. These sections appear when an account has an active dbt State subscription. * **Delete private link endpoint**: You can now delete private link endpoints from the endpoint details page. A confirmation modal requires you to type `DELETE` before the deletion proceeds. Please contact your account manager to enable. ##### APIs, Identity, and Administration * **OAuth consent improvements**: OAuth consent now recognizes the `identity:read` scope, displaying a "Read user details" label and description. Scopes limited to `identity:read` and `offline_access` no longer show the project access selector. #### Enhancements ##### dbt platform * **AI providers settings page consolidated**: The Copilot and Wizard settings pages are unified under a single "AI providers" page at `/settings/accounts/{id}/pages/ai`. The previous `/pages/copilot` URL redirects automatically, and the sidebar item and page title now use "AI providers." * **"Enable dbt State" checked by default on job create**: When creating a new job, the **Enable dbt State** checkbox is now checked by default on all environments when dbt State is available and an active subscription is present. * **dbt State model build chart adds "Reused (cloned)" series**: The dbt State model build chart now tracks three series — Built, Reused (no-op), and Reused (cloned) — giving a more detailed breakdown of model reuse. * **Teams notifications generally available**: Microsoft Teams notifications no longer require a feature flag. The Teams integration now appears in the OAuth integrations card and notification settings for all accounts. * **Private endpoints page shows Beta badge and updated info banner**: The private endpoints list and create pages now display a "Beta" badge in the header. The info banner on the create page is no longer dismissible and has updated copy clarifying that self-serve creation is available only for Snowflake AWS. ##### Orchestration and run status * **Reused node status in run results**: Studio IDE now recognizes and surfaces the `reused` node status in run results and metadata counts, giving you a more accurate picture of what ran during a dbt invocation. #### Fixes ##### Orchestration and Run Status * **Run list action buttons fixed and clickable**: Action buttons on the runs list (for example, "Debug with Copilot") no longer silently navigate to the run detail page instead of triggering the intended action. ##### Catalog * **Accurate health status filtering for stale assets**: The Catalog health filter now correctly classifies assets with a healthy bitmask but a last successful run older than 30 days as "Caution" instead of "Healthy." Assets whose last run was marked `reused` continue to be treated as healthy. #### Behavior change ##### dbt Copilot and agents * **Agent validates autofix with `dbt build` instead of `dbt compile`**: dbt Wizard autofix and model change validation is improved by defaulting to broader `dbt build` commands. #### May 27, 2026 #### New ##### Webhooks * **Disabled webhook subscriptions banner**: The webhooks settings page now shows a dismissible warning banner when one or more webhook subscriptions have been automatically disabled due to repeated failures. Disabled subscriptions now display an "Archived" badge with a tooltip explaining how to reactivate them. ##### Studio IDE * **Directory listing API**: The file browser now supports streaming directory listings in Newline-Delimited JSON (NDJSON) format, returning each file's name and type. The endpoint supports an optional `limit` parameter and `ETag` and `Last-Modified` headers to avoid re-fetching unchanged directory contents. * **File and directory deletion API**: You can now delete individual files or entire directory trees from the workspace. Recursive deletion requires `recursive=true` to be set explicitly, preventing accidental data loss. * **File and directory rename and move API**: You can now move files and directories within the workspace. An `overwrite` parameter controls whether an existing destination is replaced. The endpoint surfaces clear errors for missing sources, path traversal, name-too-long conditions, and directory conflicts. #### Enhancements ##### Webhooks * **Bounded webhook delivery history fetches**: Webhook delivery history reads are now capped at 1,000 records and limited to a 7-day lookback window, preventing unbounded memory growth from high-volume subscriptions. You should see more consistent performance for webhook history lookups on active subscriptions. ##### Integrations * **Smoother token refresh flow in OAuth consent page**: To provide a smoother experience with fewer steps, you only see the project selector for scopes that require project-level access. ##### APIs, Identity, and Administration * **Create account button shown for existing users**: The **Create account** button in the account switcher is now also shown to users who already have accounts, making it easier to create additional accounts. Please contact your account manager to enable. * **Private endpoint edit**: The private endpoint detail page now supports editing the endpoint name and port. An "Edit" button opens an inline form with validation, a confirmation modal, and clear error messaging. Please contact your account manager to enable. * **Private endpoint status badges with icons**: Connectivity status and endpoint state badges on the private endpoint list and detail pages now include status icon variants (success, error, in-progress, waiting, canceled, and health-unknown) for clearer at-a-glance status. #### Fixes ##### dbt Copilot and agents * **Protected edits to generated files**: dbt Copilot is now instructed not to edit files in `dbt_packages/`, `target/`, or `logs/`, directing fixes to source-controlled files instead. * **Build validation after column changes**: dbt Copilot now runs `dbt build` (not just `dbt compile`) after edits that add, rename, or alias columns, or change `ref` or `source` references, catching runtime errors that compilation alone would miss. ##### Webhooks * **Skipped invalid email addresses in model notifications**: Model-level notifications now validates email addresses before attempting delivery, skipping any invalid entries with a warning rather than proceeding with an invalid address. This ensures dispatching model notifications to correct email addresses. #### May 20, 2026 #### New ##### APIs, Identity, and Administration * **Notification Manager permission set**: A new `notification_manager` account-level role grants read and write access to job notification settings, Slack integration status, and member listings without requiring broader admin permissions. Refer to [Notification Manger](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#notification-manager) for more information. #### Enhancements ##### APIs, Identity, and Administration * **Cursor-based pagination for Discovery API job queries**: Job-based Discovery API queries now support cursor pagination with `first` and `after` arguments. Supported queries include `models`, `sources`, `seeds`, `snapshots`, `tests`, `macros`, `metrics`, and `exposures`. Use the returned `paginationCursor` value as `after` to fetch the next page. Page size defaults to and caps at 100. * **Support for the`lastKnownResults` filter accepting `null` values**: The `lastKnownResults` filter on `TestAppliedFilter` now accepts `null` as a value, allowing you to match tests that have no recorded result status. * **OAuth client audit log entries include registration type**: Audit log entries for OAuth client registration now append `(manual)` or `(dynamic)` to the description, making it clear whether a client was registered via dynamic client registration or manually — for example, "ChatGPT was registered for this account (manual)". ##### Orchestration and run status * **Clearer Fusion eligibility message for migrated jobs**: The Fusion eligibility surface now maps the `job_on_fusion` reason code to "This job is already running on Fusion." with no call-to-action, giving clearer feedback for jobs that are already migrated. * **Fusion job warning notifications enabled by default**: Warning notifications for Fusion runs are now always active and no longer require a feature flag, so you will see run-level warning statuses without any additional configuration. * **OpenTelemetry log format always on for Fusion runs**: Fusion runs now always use the OpenTelemetry (OTel) structured log format (except in shadow mode), removing the feature-flag requirement and ensuring consistent log output. ##### Insights * **Builds-only view when Cost Insights connection is disabled**: When your deployment environment's warehouse connection has Cost Insights disabled, the Insights page now shows a "Cost Insights is not enabled for this connection" banner and a Builds tab with the model build chart and environment and period selectors, instead of no chart content. ##### Catalog * **Connection-aware Cost Insights enablement**: Cost Insights now checks the environment connection and platform metadata credentials to confirm `cost_insights_enabled` is configured. If credentials are missing or disabled, cost metrics and tabs are hidden and a banner prompts you to configure platform metadata credentials. ##### Studio IDE * **Faster command history loading**: Command history now loads more quickly because the API returns only the data needed for the history view. ##### Deployment and configuration * **Environment permission elevation for the `job creator` role**: The `job_creator` role is now included in the set of roles that allow environment-level permission elevation, consistent with other development roles such as `developer` and `git_admin`. Refer to [Job creator](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#job-creator) for more information. #### May 13, 2026 #### New ##### APIs, Identity, and Administration * **Job creator permission set**: Adds a new [Job creator permission set](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#job-creator) for users who need to create, edit, and run jobs within assigned projects and environments. Job creators have read-only access to environments and environment variables and cannot edit environment settings. ##### Studio IDE * **Hidden file support in file search**: You can optionally include dotfiles in file search results; search remains limited to your project tree. #### Enhancements ##### dbt Copilot and agents * **Admin API tools enabled by default**: The admin API toolset (job management and run operations) is now always available in the dbt Copilot dev agent and no longer requires a feature flag. You no longer need to contact your account manager to enable these tools. * **Improved Studio IDE agent prompt**: The Developer agent in Studio now uses dbt commands more efficiently (including fewer redundant `dbt ls calls` and correct `dbt show` limits), runs independent reads in parallel when appropriate, surfaces clear next-step choices instead of ending mid-action, stays within local git capabilities (no PR/GitHub promises), and formats SQL and YAML in copy-ready code blocks. ##### Orchestration and run status * **Fusion upgrade available banner on environment settings**: A new banner appears on the environment settings page when Fusion is available for a project but not yet enabled. Admins can click **Enable Fusion access** directly from the banner to open the upgrade confirmation modal. Users without the required permissions see a read-only message directing them to contact an admin. ##### Deployment and configuration * **Snowflake password authentication blocked for new credentials**: Password authentication is no longer available when creating new Snowflake development or deployment credentials. New credentials default to key-pair authentication. Existing credentials that still use password authentication display a warning banner and cannot be saved until you migrate to key-pair or OAuth. ##### Integrations * **Paginated OAuth client registration tables**: The App Integrations settings card now loads manually registered and dynamically registered OAuth clients from separate paginated queries (10 rows per page). Each table has independent pagination controls, and deleting the last row on a non-first page automatically returns you to the previous page. * **MCP endpoint URL displayed in account settings and App Integrations**: When the dbt Model Context Protocol (MCP) feature is enabled, your account's MCP endpoint URL is now shown in the Account settings page and in the App Integrations card with a copy button, making it easier to connect external AI tools to your dbt account. Contact your account manager to enable. ##### Catalog * **Multi-value test result status filtering**: The `TestAppliedFilter` input now supports a `lastKnownResults` field that accepts an array of test statuses, allowing you to filter applied tests by multiple result states in a single query. You can include `null` in the array to match tests with no known result. The previous scalar field `lastKnownResult` is deprecated; use `lastKnownResults` instead. For more information, refer to [Filter tests by last-known result](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-environment-applied-tests.md#filter-tests-by-last-known-result) #### Fixes ##### Orchestration and run status * **Repo cache hard reset applied unconditionally**: The git hard reset performed after restoring a repository cache is now always executed, removing a feature flag that previously controlled this behavior and ensuring a clean working directory after every cache restoration. * **Clearer error message for missing repository URL**: When a job cannot clone its repository because no remote URL is configured, the error message now explains the most likely causes — an invalid Git remote URL, a Git provider outage, or a deprecated HTTPS connection — and directs you to verify the URL, confirm your provider is operational, and ensure the repository uses SSH with deploy keys before retrying. ##### Studio IDE * **Accurate node status during runs**: During dbt runs in Studio, node status in run logs now updates correctly when evaluation events arrive, so in-progress state matches what's actually happening. * **Clearer git ref lock errors**: A few failure paths that used to surface as unclear errors now show messages that explain what happened and what to try next. ##### Deployment and configuration * **Clearer connection test error messages**: The connection test failure message now prompts you to verify your connection details and confirm that your credentials have access to the data warehouse, rather than showing a generic failure message. ##### Run logs * **Structured run logs enabled by default**: Structured run step logs are now always used when available, without requiring a feature flag. Runs that already have structured logs continue to display them; runs with only plaintext logs fall back to the legacy view automatically. #### May 6, 2026 #### New ##### dbt Copilot and agents * **Project instruction files in agent system prompt**: dbt Copilot now reads `AGENTS.md` and `CLAUDE.md` files at the root of your dbt project and injects their contents into the agent system prompt on every turn. Combined content must not exceed 64 KB. * **Subdirectory instruction file discovery**: The agent now surfaces a manifest of `AGENTS.md` and `CLAUDE.md` files found in project subdirectories and reads them on demand when working in the relevant subtree, keeping context loading efficient for large projects. ##### dbt platform * **Create account for unlicensed users**: Users with zero accounts can now create a new account directly from the regional account switcher. When enabled and a `create_account_url` is available, a "Create account" button appears on the account switcher for users with no accounts. Contact your account manager to enable. #### Enhancements ##### Packages * **New:** [Native private packages](https://docs.getdbt.com/docs/build/packages.md#native-private-packages) are now generally available (GA). ##### Discovery API * **Enhancement:** In the Discovery API [Tests object schema](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-environment-applied-tests.md), you can now filter `environment.applied.tests` by multiple test result statuses in a single query using the new `lastKnownResults: [TestStatus]` filter field on `TestAppliedFilter`. The single-value `lastKnownResult` filter field is still supported but deprecated. Update your queries to use `lastKnownResults` going forward. ##### dbt Copilot and agents * **Preview**: The [Developer agent](https://docs.getdbt.com/docs/platform/wizard-platform.md) is now in preview. Use natural language prompts to build or refactor models, and generate SQL, tests, documentation, and semantic models from scratch. For more information, refer to the [Developer agent](https://docs.getdbt.com/docs/platform/wizard-platform.md). * **Enhancement:** Delete individual [chat conversations](https://docs.getdbt.com/docs/platform/wizard-platform.md#availability-and-considerations) from the conversation list (**More actions** menu (three dots) > **Delete**). Deleting the open conversation clears the panel. * **Enhancement:** Commands run by and the [](https://docs.getdbt.com/docs/platform/wizard-platform.md)now appear in the Studio IDE **Commands** tab with a icon and **Run by Copilot** tooltip, so you can tell agent-run commands apart from manually run ones. ##### dbt platform * **Clearer message when account creation is disabled**: The `/accounts/new` page now renders an inline "Account creation is disabled" message instead of silently redirecting to the home page, preventing a redirect loop for users with zero accounts. * **Account feature changes emit audit log events**: When an account feature opt-in is toggled (for example, enabling catalog ingestion or advanced CI), an audit log event is now published recording the previous and new values. ##### APIs, Identity, and Administration * **Notification Manager permission set**: A new account-level **Notification Manager** [permission set](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md) is now available for Enterprise accounts. Assign it to users who need to manage Slack, Microsoft Teams, and email job notifications across all projects without requiring full Account Admin access. * **Credentials page access with `user_credential_write` permission**: Users with the `user_credential_write` permission on any project can now access the Credentials settings page and edit their user credentials, even without a Developer-tier license or `develop_access`. #### Fixes ##### dbt Copilot and agents * **Correct handling of `tool_call_chunk` content blocks**: Fixed a bug in single tenant environments that would occasionally block conversations from being able to be continued. #### Behavior change ##### APIs, Identity, and Administration * **Single Sign-On configuration uses auto-generated slugs only**: The Single Sign-On (SSO) settings card now always uses server-assigned slugs. The manual slug input has been removed. All SSO configurations are created as drafts with an auto-generated slug before you fill in connection details. ##### Deployment and Configuration * **Fusion version detection uses strict catalog membership**: `isFusionDbtVersion` now only returns `true` for version strings that are explicit members of the known Fusion versions catalog (`latest-fusion`, `fusion-stable`, and so on). Previously, any string containing the word "fusion" was treated as a Fusion version. Custom or unknown version strings that happen to include "fusion" are no longer treated as Fusion versions. #### April 29, 2026 #### Enhancements ##### dbt Copilot and agents * **Job investigation support in Studio agent**: The Studio IDE dev agent can now help you investigate and troubleshoot dbt job and run failures using the `troubleshooting-dbt-job-errors` skill. The agent notes when your local project state may differ from the job (for example, a different branch or uncommitted changes). This feature is currently in beta. Refer to [Debug job failures](https://docs.getdbt.com/docs/platform/wizard-platform.md?version=2.0#debug-job-failures) for more information. ##### Semantic Layer * **Semantic Layer MCP request size limit**: Semantic Layer requests through MCP are now capped at 10 MiB (previously unlimited) to improve infrastructure stability. ##### Orchestration and run status * **Finer-grained permissions on the Debug on Fusion menu**: The "Debug in Studio" and "Run once on Fusion" menu items are now independently disabled based on your permissions. If you lack the required permission for an action, that item shows a tooltip explaining why, while the other item remains available. * **Automatic dbt version override before "Debug in Studio"**: When you click "Debug in Studio," dbt platform now automatically sets your user-level `DBT_DEVELOP_CORE_VERSION` environment variable to `latest-fusion` before opening the Studio IDE, so you no longer need to configure this manually. * **Clearer failure state for "Debug in Studio"**: If the preparation step before opening the Studio IDE fails, the button temporarily shows "Debug failed" in a red state for 5 seconds before resetting, so you know to try again rather than seeing a silent failure. * **Fusion upgrade eligibility for projects with no jobs**: Projects with no jobs configured are now treated as eligible for Fusion. The upgrade card no longer requires a successful job run for such projects and instead shows "No jobs configured yet" under job eligibility. * **Confirmation pop-up before overriding Fusion eligibility**: Clicking "Override eligibility status" on the run details page now opens a confirmation pop-up before applying the override, preventing accidental changes. * **Fusion eligibility review button visible to all users**: The review button on the jobs list for jobs with unknown Fusion eligibility is now shown to all users regardless of run write permissions, so anyone can view the eligibility details modal. * **Model execution notifications for dbt Fusion runs**: The orchestrator now publishes model execution events for grouped models and tests when running with dbt Fusion's OpenTelemetry (OTel) log format, extending model notification support to Fusion-based runs. * **Structured logs available for externally ingested runs**: The run ingestion pipeline now detects and signals when structured dbt logs are present for a step, so logs appear correctly in the dbt platform for runs ingested from external executors. ##### Deployment and configuration * **Sticky submit bar on the create private endpoint form**: The Cancel and Save buttons on the create private endpoint page now stick to the bottom of the viewport so they remain accessible when scrolling through the form. * **Docs generation deprecation scoped to Fusion jobs only**: The "Generate docs on run" deprecation notice is now only shown for jobs running on a Fusion dbt version. Non-Fusion jobs continue to show the standard checkbox. #### Fixes ##### Semantic Layer * **Custom metric granularities no longer rejected**: Metric manifest fields `granularity` and `offset_to_grain` now accept arbitrary string values instead of only a fixed enum. Projects using custom granularities such as `fiscal_year` will no longer fail ingestion. ##### Catalog * **Tag search field**: Tags are now a searchable field in the advanced search panel. You can filter results by tag matches. Filtering uses OR logic, returning assets that match any of the specified tags rather than requiring all tags to be present. #### April 22, 2026 #### New ##### Catalog * **Health and run status filters in catalog search**: The catalog search sidebar now includes Health and Last Run Status filter sections. You can filter dbt resources (models, sources, and exposures) by health status (healthy, caution, degraded, unknown) and by last run status (success, error, skipped, reused). * **Tag search field**: Tag is now a searchable field in the advanced search panel. You can filter results by tag matches #### Enhancements ##### Studio IDE * **More reliable dark mode on initial load**: Added additional layers of theme preference fallbacks, including the user's OS theme preferences, to aid in incorrect theming when user-preferences is slow to respond. * **Deep-linking to console tabs**: You can now navigate directly to a specific Studio IDE console tab (for example, Commands or Lineage) using a `consoleTab` URL query parameter. Invalid tab identifiers are removed from the URL automatically. * **Compile button after deprecation autofix in Fusion**: After the deprecation autofix workflow completes in Fusion environments, a **Compile** button now appears in the autofix results panel so you can immediately verify the updated project without manually triggering a compile. ##### Orchestration and run status * **Fusion eligibility toggle replaces dropdown filter**: The jobs list Fusion eligibility dropdown filter has been replaced with a toggle and help icon. When enabled, each job displays its current Fusion eligibility badge, and a persistent info banner explains how eligibility is recalculated. The toggle state is saved per-project in your browser. * **Debug on Fusion menu**: The single "Run once on Fusion" button on the job details page and job list has been replaced with a "Debug on Fusion" menu that offers "Debug in Studio," "Run once on Fusion," and (when dbt Copilot is enabled) "Debug in Studio with Copilot" options. * **Simplified Fusion run error banner**: The Fusion run error banner on run details now uses the same "Debug on Fusion" menu as the jobs page. The banner no longer requires setting a personal dbt version override before navigating to Studio. ##### Webhooks * **Webhook test flow uses receipt polling**: Testing a webhook subscription now triggers a test event and polls for the delivery receipt, showing the actual HTTP status code and error from the endpoint response. A 60-second timeout is applied, with a clear timeout message if the endpoint does not respond in time. * **Webhook receipt endpoint returns 404 for pending events**: The webhook event receipt endpoint now returns a `404` response when a delivery record has not yet been written (for example, when the notification system has not yet processed the event), rather than returning an incomplete record. * **Corrected status code for timed-out webhook deliveries**: Webhook delivery history records now show `504` as the HTTP status code when a delivery timed out (previously stored as `0`), improving accuracy in the delivery history view. * **Webhook event history note always visible**: The note that event history is limited to the past 7 days now appears on the webhook events history page unconditionally. ##### Integrations * **Slack notification settings migration banner**: A migration banner now appears on the Slack notification settings page when you have notification settings from a previous Slack integration. You can migrate them to the new Slack app in one click or dismiss the banner. After migration, you are shown which private channels need the dbt Cloud bot invited for notifications to be delivered. Contact your account manager to enable. ##### dbt platform * **`account:read` scope on OAuth consent page**: The OAuth consent page now displays a "View account information" scope option, which grants view-only access to account details including project and environment information. * **PrivateLink endpoint pending status**: A new `pending` connectivity status is available for PrivateLink endpoints, in addition to the existing `success` and `failed` states. * **`fusion_readiness_read` permission added to Member role**: The Member permission set now includes `fusion_readiness_read`, allowing members to view Fusion readiness information for projects without requiring elevated permissions. #### Fixes ##### dbt Copilot and agents * **Correct model routing for Azure OpenAI Responses API (BYOK customers)**: Azure OpenAI deployments now correctly pass the deployment name as the `model` field when using the Responses API, preventing misrouted requests when the deployment name differs from the model name. #### April 15, 2026 #### Enhancements ##### Catalog * **Health and run status search filters**: The `AccountSearchQueryFilter` input now accepts `health` and `runStatus` filter arrays. Use `health` to narrow results by health status (`healthy`, `caution`, `degraded`, or `unknown`) and `runStatus` to filter by last run outcome (`success`, `error`, `skipped`, or `reused`). Multiple values within each filter are combined with `OR` logic. * **Health-aware search ranking**: Healthy dbt resources (those with no detected issues) now rank higher in search results than resources with unresolved issues when text relevance is otherwise equivalent. ##### Studio IDE * **Keyboard shortcut to open Commands tab**: Press \`Ctrl+\`\` to open the Commands tab directly from the editor. ##### Orchestration and run status * **Clearer Fusion job eligibility messages**: Fusion eligibility reason messages are rewritten to be shorter and more actionable. For example, unsupported adapters now read "This job uses an adapter that's not currently available on the Fusion engine" and jobs not on Latest now read "This job uses a dbt version that's not tested for Fusion eligibility." * **Fusion eligibility confirmation modal**: Clicking "Run once on Fusion" on a job now opens a confirmation modal before triggering the run, showing the environment name and a warning that job commands will execute in that environment. * **Improved `dbt ls` and `dbt list` run log status (dbt Fusion engine only):**: Run steps that execute `dbt ls` or `dbt list` now show node results with a no-op status instead of "unknown," reducing confusion in run logs for list operations. ##### dbt platform * **More descriptive Fusion readiness toggle**: The account-level setting to enable Fusion readiness and upgrade features now has an updated label ("Enable Fusion readiness & upgrade features") and a more detailed description explaining what the setting allows administrators and developers to do. * **Debug on Fusion navigates with version override**: The "Debug on Fusion" button (previously "Debug manually") on failed Fusion run banners now sets your personal `DBT_DEVELOP_CORE_VERSION` override to `latest-fusion` before opening Studio IDE, ensuring you open the IDE on the Fusion engine. A loading state is shown while the override saves, and an inline error is displayed if the save fails. ##### Deployment and configuration * **Private endpoint connectivity status column renamed**: The "Status" column in the private endpoints list is renamed to "Connectivity status" for clarity. * **Snowflake private endpoint validation shows specific missing fields**: When pasting Snowflake Private Link configuration output, the validation error now lists the specific required fields that are missing (for example, `privatelink-account-url`) rather than a generic message. Valid output now also shows a success indicator. * **YAML credential fields now accept array values**: Environment credential and connection forms that accept YAML Extended Attributes (for example, Redshift `db_groups`) now correctly validate arrays as values. Previously, array values were incorrectly rejected during client-side validation. ##### Integrations * **Snowflake PrivateLink supports reusing existing interface endpoints**: When creating a Snowflake PrivateLink connection, you can now supply an optional `interface_endpoint_id` to attach a new profile to an existing interface endpoint rather than always creating a new one. The endpoint must be in `Available` status; a `409 Conflict` is returned otherwise. Contact your account manager to enable. #### Fixes ##### Studio IDE * **New folders in Git Controls now expand correctly**: Files inside a newly created folder are now listed individually in the Git Controls panel. Previously, a new folder appeared as a single unexpanded entry rather than showing the files it contained. * **Parent folder hint shown for all new files**: Files created inside a new folder now always display the parent folder name as a hint in the Git Controls panel, even when the file name is unique across all changed files. #### April 8, 2026 #### New ##### dbt Copilot and agents * **Admin API tools for MCP remote server**: The dbt MCP remote server now includes Admin API tools, including `list_jobs`, `list_projects`, `get_job_details`, `trigger_job_run`, `cancel_job_run`, `retry_job_run`, `get_job_run_details`, `get_job_run_error`, `list_job_run_artifacts`, and `get_job_run_artifact`. These tools let MCP clients list, inspect, trigger, cancel, and retry dbt jobs and runs directly from connected AI assistants. Contact your account manager to enable. ##### dbt platform * **OAuth consent page**: A new OAuth consent page lets you authorize third-party applications (for example, dbt MCP) to access your dbt platform account. You can select which permissions and projects to grant, then approve or deny the request. ##### Catalog * **Performance tab on test and snapshot detail pages**: Test and snapshot detail pages now include a Performance tab showing cost insights data — including cost, usage, build time, and build count charts — matching the existing model performance experience. ##### Orchestration and run status #### Enhancements ##### dbt Copilot and agents * **Smarter validation after file edits**: The Studio DevAgent now selects the lightest appropriate validation check after each change — for example, skipping compilation for description-only edits and running `dbt parse` for project config changes — instead of always running a full `dbt compile`. This reduces unnecessary round-trips and keeps iteration faster. ##### Studio IDE * **Deferral environment selector**: Replaces the simple defer-to-production toggle with a popover that lets you choose between your development environment, dbt's default deferral behavior (staging if available, otherwise production), or a specific custom environment. A badge in the command bar shows your current deferral target at a glance. * **Revert personal dbt version override**: Adds an "Edit / Revert" action to the version override option in the environment popover. Clicking "Revert" opens a confirmation modal that removes your personal dbt version override and restarts the session. * **Improved file context pill in dbt Copilot**: Moves the active-file context pill to above the text input for greater visibility. When you remove the file context, a "Use current file as context" affordance appears so you can restore it without switching tabs. ##### Catalog * **Reused test status in DAG lens**: State-Aware Orchestration (SAO) test runs that reuse prior results now display with a "reused" icon in the DAG test status lens, matching the existing model run status behavior. * **Function resource type support in selectors**: The `function` resource type is now recognized in dbt selectors and the resource node type map, enabling correct filtering and navigation for function resources in Catalog. ##### Insights * **Fusion status column in account insights table**: Look for a "Fusion status" column in your account insights table when the Fusion readiness flow is available for your account. You'll see one of four states: On Fusion, Start upgrade, Partial-Fusion, or Non-Fusion — based on each project's readiness and migration progress. Projects that are ready to upgrade show a "Start upgrade" button that navigates directly to the project home page. Contact your account manager to enable. #### Fixes ##### Studio IDE * **Parse status no longer shows error badge during Fusion compilation**: In Fusion mode, the parse status badge no longer switches to an error state solely because diagnostic errors are present. The badge now correctly reflects compilation progress and completion independent of diagnostic counts. * **Clearer authentication errors for rejected git connections**: Adds "remote rejected authentication" as a recognized, non-retryable git authentication error. You will now see a clear authentication failure message instead of a misleading retry loop when your git provider rejects your credentials. ##### Catalog * **Reused models no longer flagged as stale**: Models with a `last_run_status` of `reused` are no longer marked stale even when their last execution date exceeds 30 days. This prevents false health issue warnings for models that were intentionally reused rather than re-executed. * **Resource counts refresh on environment switch**: Fixes a bug where resource counts on the project landing page were not updated when switching environments. #### April 1, 2026 #### New ##### Studio IDE * **Fuzzy file path search**: Studio IDE now supports fuzzy file path search that finds files in your project using partial name matching. You can filter by glob patterns, set a result limit, and receive ordered results with a total match count, making it faster to navigate large projects. ##### dbt platform * **OAuth consent endpoint for Connected Auth**: A new `/oauth/consent` endpoint enables the Connected Auth OAuth flow, supporting user consent decisions (approve and deny), project-level resource boundaries, and authorization code issuance. This feature is in private beta. To request access, contact your account manager. #### Enhancements ##### dbt Copilot and agents * **Persistent agent mode across sessions**: The Studio agent now remembers your last-used mode (Ask or Code) across browser sessions, so you no longer need to reselect it each time you open the IDE. ##### Studio IDE * **More accurate file search results**: File search now validates each result against the filesystem before returning matches. Files that have been deleted locally but not yet staged are no longer included in search results. * **No unexpected git pulls on the primary branch**: Removes behavior where the IDE server automatically pulled changes from your primary branch during git status checks, which could cause unintended overwrites for projects using trunk-based development. ##### Orchestration and run status * **Teradata column-level lineage support**: Adds Teradata to the SQL dialect adapter map, enabling column-level lineage parsing for dbt projects using the Teradata adapter. ##### APIs, Identity, and Administration * **Fusion status includes readiness and migration availability**: Adds fields indicating availability of readiness and migration features. * **Faster account feature flag propagation**: Account feature flag changes now take effect within 60 seconds instead of up to one hour. You should see feature toggles apply more promptly across your account. ##### Webhooks * **Notification delivery reliability improvements**: Reduced the likelihood of delayed notifications (webhooks, email, Slack, and Teams) in certain third-party/system disruption scenarios. #### Fixes ##### APIs, Identity, and Administration * **GitHub webhook secret null check before signature validation**: The GitHub webhook endpoint now correctly checks for a null webhook secret before attempting to validate the request signature, preventing a crash when a repository's webhook secret is not set. * **`github_installation_id` and `github_webhook_id` support large values**: These repository fields have been promoted from 32-bit to 64-bit integers (`BigIntegerField`) to accommodate GitHub installation and webhook IDs that exceed the 32-bit integer range. #### Behavior change ##### APIs, Identity, and Administration * **Fusion migration gated by API availability**: The Fusion migration checklist, the Enable Fusion Environments page, and the "Enable Fusion" button in Studio IDE now use the `is_migration_available` field from the Fusion status API instead of the legacy `orc2609ShowFusionToggle` feature flag. Fusion migration UI is shown only when the backend has marked the project as ready for migration. #### March 25, 2026 #### New ##### Orchestration and Run Status * **Fusion run error banner**: When a run using the dbt Fusion engine fails, a banner now appears on the run details page with options to debug the failure in Studio IDE. If dbt Copilot is enabled, you can also open a guided fix-with-Copilot workflow directly from the banner. Contact your account manager to enable. #### Enhancements ##### Studio IDE * **Consistent console pane default size**: The bottom console pane now opens at a preferred size of 33% of the available space, providing a more consistent default layout. * **Faster text search results**: File search now reports results incrementally on a per-file basis rather than per-match line, reducing memory pressure and improving perceived responsiveness during large searches. * **Smarter bulk-edit file handling**: When Studio IDE applies multi-file edits (for example, from dbt Copilot agent tasks), it now only updates editor models for files that are already open. Previously, every edited file was opened in a new tab, which cluttered the editor. ##### Deployment and Configuration * **Fusion migration enablement via project API**: You can now set `fusion_migration_enabled` on a project via the project update API. Enabling it requires the `fusion_readiness_write` permission, and the project must meet all readiness prerequisites (supported adapter, supported dbt version, a successful run, and eligible jobs). * **Filter jobs by Fusion readiness**: The jobs list endpoint (`GET /api/v2/accounts/{account_id}/jobs/`) now accepts an `is_fusion_ready` boolean query parameter. When `true`, it returns only conformant or override-ready jobs; when `false`, it returns only non-ready jobs. You can also include `fusion_readiness` in the `include_related` parameter to surface Fusion readiness details alongside the job response. * **Platform metadata credentials form opens immediately**: When adding platform metadata credentials for a connection, the credential form is now shown immediately instead of requiring you to click an "Add credentials" button first. #### Fixes ##### Catalog * **Accurate resource counts on environment switch**: Fixes a bug where resource counts in the navigation tree were not refreshed when switching environments. You should now see up-to-date counts after changing the active environment. ##### Orchestration and Run Status * **Stuck runs are now cancelled**: A new cleanup job detects runs and run steps that have exceeded the maximum allowed duration and marks them as `CANCELLED`, preventing stale in-progress states from accumulating ##### Semantic Layer * **More reliable Snowflake connections after warehouse auto-resume**: The Semantic Layer Gateway now retries the initial connection when a Snowflake warehouse is waking up from auto-suspend, instead of failing immediately. You should see fewer connection errors when querying the Semantic Layer after a period of inactivity. ##### APIs, Identity, and Administration * **Large group permission sync no longer silently truncated**: Fixed an issue where group permission sync could miss updates for groups with many permissions. #### Behavior change ##### Studio IDE * **Fusion OpenTelemetry log rendering always enabled**: Studio IDE now enables Fusion OpenTelemetry (OTel) log rendering for all invocations running on a Fusion core version, removing the previous feature flag requirement. If you are running a Fusion core version, you automatically receive OTel-based log output without any additional configuration. #### March 18, 2026 #### Enhancements ##### Studio IDE * **Faster file search:** Studio IDE now reuses its file-search index across searches, so repeated searches return results faster. * **More responsive Git status decorations:** Studio IDE debounces rapid file change events and avoids applying stale responses, so Git status badges update more reliably during bulk edits and saves. * **Clearer server status details:** The server status popover uses a clearer grouped layout and action buttons to help you troubleshoot user credentials and server health. Please contact your account manager to enable. ##### dbt Copilot and agents * **More accurate product guidance:** Copilot and agents can use a product documentation toolset to answer product and workflow questions more reliably. * **Full-screen Copilot view:** You can open Copilot in a dedicated full-screen view for a more focused chat and coding workflow. Please contact your account manager to enable. * **Lighter default file context:** Copilot now references your active file by path instead of automatically attaching the file contents, which reduces message size and improves chat reliability. * **Run `dbt-autofix` from Copilot and agents:** Copilot can run `dbt-autofix` commands (with confirmation) and stream the output into chat, and Studio IDE agents can run `dbt-autofix` using `run_autofix` for bulk deprecation fixes and migrations. ##### Catalog * **Custom materialization filter:** Catalog search now groups non-standard materializations under a single "Custom" filter, so you can narrow results without picking each materialization type. ##### Insights * **More complete Redshift query attribution:** Insights can resolve missing Redshift query IDs from warehouse query history when artifacts do not include them, improving cost coverage for runs with executions. * **Copilot entry stays available during lockouts:** If dbt Copilot is temporarily locked for your account, you can still open Copilot from Insights to see lock details. ##### Orchestration and Run Status * **Run metadata includes triggering and canceling actors:** Run details now include who triggered or canceled a run (user or service token), which helps you audit run activity. * **Custom branch preserved for runs and reruns:** When an environment uses a custom branch, dbt platform now carries that branch through run triggers, retries, and reruns more consistently. * **Fusion readiness metadata for jobs and environments:** You can now retrieve Fusion readiness signals for projects, environments, and jobs to support Fusion migration planning. Please contact your account manager to enable. * **More accurate command names for dbt Fusion runs:** Orchestration now reads the invocation name from `run_results.json` using `command` when `invocation_command` is missing, so you see the correct dbt command in run details. ##### Run Logs * **More resilient run step history ingestion:** Run step history ingestion now drops invalid events and de-duplicates redundant step-start events before writing step data, improving step-level accuracy. Please contact your account manager to enable. ##### Deployment and Configuration * **Longer project descriptions:** You can now add project descriptions of up to 1,024 characters. * **Connection links in profiles:** You can now open a connection directly from the connection profile table in a new tab. * **Clearer YAML validation for extended attributes:** You now get more consistent validation and clearer error messages for invalid YAML syntax, null values, and non-object YAML content when you edit extended attributes. ##### Semantic Layer * **Improved filtered-query cache matching:** Cached query results can now be matched and reused more reliably when your query includes filters, which can reduce repeated compilation and improve response times. #### Fixes ##### APIs, Identity, and Administration * **Fewer transient Cloud Config failures:** Retries now only apply to transient errors during Cloud Config lookups, so you should see fewer intermittent failures without added delay for permission, authentication, or not-found responses. * **More reliable sign in redirects from `current_email`:** If you are already authenticated and land on `/login` with `current_email`, dbt platform now redirects you to `/api/auth/auth-login/` so the email is forwarded during sign in. * **IP restrictions toggle saves reliably:** Turning IP restrictions on or off now updates form state correctly, so your changes save as expected. * **More consistent audit log date filtering:** The audit log date range defaults no longer shift during re-renders, so your filters stay stable while you review results. * **More reliable Single Sign-On (SSO) migration domain updates:** Domain updates during Single Sign-On (SSO) migration no longer rely on mutating existing provider data, which improves save reliability. ##### Deployment and Configuration * **Clearer Bring Your Own Key (BYOK) credential errors:** If your OpenAI credentials include invalid characters, you now get a clearer error message so you can correct the configuration. * **More reliable credential edits:** Encrypted credential fields now stay optional when you edit credentials, which reduces unexpected validation failures. * **Correct connection details while editing environments:** You now see the correct connection details more consistently when you edit an environment that uses global connections and connection profiles. ##### Orchestration and Run Status * **Run steps are available for ingestion runs:** You can now open and review run steps for ingestion-triggered runs. * **Cleaner run error fields:** Run results no longer populate an error string with `None` when dbt does not provide a message or failure count, so you see clearer run error details. * **Clearer errors for invalid dbt projects:** When Orchestration cannot restore the repository cache because the dbt project is missing or malformed, it now returns an invalid project error so you get a more actionable message in run results. ##### Catalog * **Skipped snapshots show as skipped:** Snapshots selected but not executed in multi-step runs now appear with a skipped status instead of missing run status fields. ##### Insights * **Copilot chat no longer gets stuck loading:** Insights now clears the Copilot chat loading state reliably after responses complete or error, so you can keep chatting without refreshing the page. * **More reliable Copilot handoff starts:** When you arrive in Insights with a Copilot handoff message, Insights now starts the handoff once and clears stale handoff state when you navigate directly. ##### Integrations * **More consistent JDBC typing for Tableau and Power BI:** Semantic Layer now derives explicit string conversions from returned result metadata, so categorical dimensions and entities are more consistently typed as strings in Tableau and Power BI queries. ##### Semantic Layer * **More reliable cache key deletion:** Cache invalidation no longer fails when an in-memory cache key is already missing, which reduces intermittent errors during cache cleanup. * **More accurate run ID validation:** Semantic Layer now requests and caches run details scoped to your account, which reduces incorrect run validation results. ##### dbt Copilot and agents * **More reliable cancellations during tool use:** If you cancel a request while an agent is running tools, the agent now recovers cleanly instead of getting stuck on incomplete tool-call history. * **Cleaner AI diff overlays:** Studio IDE now removes the accept and reject overlay when you leave an artificial intelligence (AI) diff view to prevent stale UI controls. #### Behavior change ##### Studio IDE * **Updated file search and command shortcuts:** Studio IDE now uses VS Code Quick Open for file search (`Cmd+P` or `Ctrl+P`) and the VS Code Command Palette (`Cmd+Shift+P` or `Ctrl+Shift+P`) instead of the legacy Studio dialogs. ##### Integrations * **Disallowed `MIN()` and `MAX()` for metrics and dimensions:** Tableau and Power BI queries can no longer request `MIN()` or `MAX()` for a metric or dimension (except time min-max queries), and you now receive a clear error if you attempt it. #### March 11, 2026 #### New ##### Deployment and Configuration * **Self-serve Snowflake private endpoint requests:** You can request a new Snowflake private endpoint from account settings by pasting the output from `SELECT SYSTEM$GET_PRIVATELINK_CONFIG();`, then track request status in the private endpoints table. This is available for Enterprise Business Critical accounts only, and please contact your account manager to enable. For other connection types, contact <support@dbtlabs.com>. #### Enhancements ##### Orchestration and Run Status * **Run retries support dbt Fusion runs:** You can now retry failed runs as long as your environment is on dbt Core version `1.6` or higher or dbt Fusion. ##### Integrations * **More reliable Slack notifications:** Slack channel discovery and notifications now retry on Slack rate limits to reduce dropped messages during busy periods. ##### APIs, Identity, and Administration * **Improved OpenAPI typing for large integers:** OpenAPI schemas now mark 64-bit integer fields as `format: int64` to improve generated client types. * **Clearer credentials schemas:** Credentials OpenAPI docs now use a `type` discriminator (`postgres`, `redshift`, `snowflake`, `bigquery`, and `adapter`) to improve code generation and request validation. #### Fixes ##### Orchestration and Run Status * **More reliable job search:** Searching jobs with numeric terms (for example, `12`) no longer triggers API validation errors, so you can load job lists reliably. * **Clearer cross-project publication errors:** When dbt platform cannot fetch a publication artifact for an upstream project declared in `dependencies.yml`, you now see which project is missing an artifact and guidance to run the upstream environment at least once. ##### Integrations * **More accurate Microsoft Teams notification triggers:** Microsoft Teams notifications now use the correct trigger event type for each notification, so you see the expected run outcome context in the message. ##### APIs, Identity, and Administration * **More accurate error responses during permission checks:** You now receive more accurate errors from permission checks, and underlying service errors surface instead of being reported as authorization failures. ##### Deployment and Configuration * **Clearer private endpoint validation errors:** Creating a private endpoint now returns a `400` error with a clear message when `snowflake_output` is malformed or not valid JSON. #### Behavior change ##### Orchestration and Run Status * **Model timing unavailable for dbt Fusion runs:** You now see an informational notice instead of the Model timing chart for dbt Fusion runs because dbt Fusion handles threading differently. ##### APIs, Identity, and Administration * **System for Cross-domain Identity Management (SCIM) `id` fields are now strings:** SCIM schema discovery now reports `id` fields as strings for users and groups. #### March 4, 2026 #### Enhancements ##### Orchestration and Run Status * **Clearer SAO description**: Job settings now describe state-aware orchestration (SAO) as only building models when data or code changes are detected. * **Direct links for cost optimization setup**: Fusion cost optimization settings now link to account-level Cost Insights settings and setup documentation so you can validate cost data and savings. ##### APIs, Identity, and Administration * **Confirmation when enabling manual SCIM updates**: When you enable manual updates for System for Cross-domain Identity Management (SCIM), dbt platform now asks you to confirm so you do not accidentally allow changes outside your identity provider. * **More reliable SCIM group provisioning**: SCIM has been updated so that when a SCIM-provisioned user with an expired invite is added to a SCIM-managed group through a SCIM request, the invite is automatically resent during group assignment. This helps prevent errors caused by unaccepted invites. ##### dbt platform * **Project names and descriptions handle empty values better**: Projects with missing names now show as "Untitled Project," and you can save project descriptions as empty. ##### Studio IDE * **Removed non-functional "Open Settings" actions**: Studio IDE no longer shows "Open Settings" buttons in editor notifications because Studio IDE does not expose VS Code settings, and the action would not help you resolve issues. #### Fixes ##### Catalog * **More reliable file tree loading**: Catalog no longer gets stuck loading the file tree on initial page load. * **Clearer trust signals**: Trust signals now suppress less-severe upstream-source issues when a more severe issue is present, so badges and messages are easier to interpret. ##### Integrations * **Clearer deploy key decryption errors**: When dbt platform cannot decrypt a deploy key, you now get a clearer failure instead of a generic git credentials error. ##### Studio IDE * **Cleaner LSP disconnects**: If authentication fails when you connect to the Language Server Protocol (LSP) WebSocket, the connection now closes cleanly instead of failing with an internal server error, so you should see fewer unexpected disconnects. * **Improved timeout handling and authentication stability**: Reduced environment setup timeouts and resolved intermittent authentication failures during busy periods. * **Clearer invalid credentials error**: If your development connection credentials are invalid, you now see a clearer error message to help you diagnose the issue faster. #### Behavior change ##### Orchestration and Run Status * **`versionless` dbt version is no longer accepted**: dbt platform now treats `versionless` as deprecated and updates existing environments and jobs to use `latest`. If you set `dbt_version` in an API integration or automation, update it to send `latest` instead. ##### Webhooks * **Account identifier required for run-based notifications**: If you send events that include a `run_id`, you must also provide an `account_identifier` so the service can validate and resolve the correct account before dispatch. If `account_identifier` is missing, the event fails instead of falling back to a `run_id`-only lookup. #### February 25, 2026 #### New ##### Catalog * **Saved queries now ingested for lineage and governance**: Saved query definitions (including tags, exports, parameters, and lineage relationships) are now captured during ingestion so they can participate in Catalog lineage and governance workflows. #### Enhancements ##### dbt platform * **System logs now surface warnings and errors**: Run step structured logs now show an indicator when system warnings or errors are present, making issues easier to spot during run triage. * **Region labels now use backend display names**: Account Settings now shows the backend-provided region display name for clearer, more accurate region labeling. * **SCIM create group UI change**: Changes to our UI to improve the experience of managing groups with SCIM enabled. * **Updated the post-invite message for SSO accounts**: After a user accepts an invite, the UI now explains that they must log in using SSO to fully redeem the invite and access the account. This replaces the previous "Joined successfully" message and helps avoid confusion when users accept an invite but do not complete the SSO login flow. ##### Studio IDE and Copilot * **Improved crash recovery and not-found routing**: Studio IDE now catches unexpected render failures with a top-level error boundary and shows Not Found more reliably for unknown in-project routes. * **Improved navigation accessibility and semantics in Studio IDE**: The main navigation trigger area is now a navigation element with improved focus and labeling. * **Reduced shortcut conflicts with VS Code search**: When Visual Studio Code (VS Code) search is enabled, Studio IDE avoids unregistering Quick Open and suppresses conflicting command palette shortcuts. ##### Catalog and Insights Data * **More accurate source freshness outdated status in Catalog**: Source freshness Outdated status can now be computed at query time, improving freshness status filtering consistency. * **Improved search and lineage usability in Catalog**: Search results better support column-level navigation and very long queries show a clear validation error, and lineage visuals have improved alignment and reduced edge clutter. * **Improved cross-project lineage and function awareness in Catalog**: Lineage graph building now includes cross-project dependencies and supports function nodes as first-class lineage entities. ##### APIs, Identity, and Administration * **Project deletion now supported in Admin v2 and v3 Projects APIs**: Projects APIs now explicitly support DELETE with stricter permission checks. #### Behavior change ##### Webhooks * **Updated job run event field presence and status normalization**: Webhook payloads now include `runFinishedAt` only for completed events and `runErroredAt` only for errored events; canceled runs no longer include `runCanceledAt`, and run status is normalized from Cancelled to Canceled. Also note that enabling JSON preserve order can change key ordering, so consumers should parse JSON rather than string-compare payloads. ##### Insights APIs * **Optional source freshness expiration windows**: Source freshness expiration windows can optionally derive from each source's freshness criteria rather than a fixed window. You must enable in your deployment. ##### Deployment and Configuration * **Source ingestion may skip sources for extremely large manifests in Catalog**: For very large `manifest.json` files, ingestion may strip sources above a configurable threshold to prevent out of memory failures. Set `SOURCE_INGESTION_THRESHOLD=0` if you must always ingest sources regardless of size. * **Removed deprecated object storage settings in Studio IDE**: Deprecated settings `project_storage_bucket_name` and `project_storage_object_prefix` have been removed. Migrate to `object_storage_bucket_name` and `object_storage_object_prefix`. #### February 18, 2026 #### New ##### Cost Insights * **Estimated warehouse compute costs**: Cost Insights shows estimated warehouse compute costs and run times for your dbt projects and models, directly in the dbt platform. It highlights cost reductions and efficiency gains from optimizations like state-aware orchestration across your project dashboard, model pages, and job details. This feature is in private beta. To request access, contact your account manager. #### Enhancements ##### Studio IDE * **Reduced conflicts across multiple tabs**: Studio IDE can pause the Language Server Protocol (LSP) in background tabs and resume on return to improve stability when the editor is open in more than one tab. * **More informative header and more editor space**: Adds a Visual Studio Code-style header showing a dbt badge and current project name, with an option to hide surrounding chrome for more editor space. Please contact your account manager to enable. * **Clearer file and folder creation errors**: Surfaces more actionable filesystem errors (for example, name too long and file-is-a-directory) instead of generic failures. * **Copy relative path**: Adds a Copy Relative Path action that respects `dbt_project_subdirectory` for quicker navigation and sharing. * **Friendlier lineage error messages**: Improves user-facing errors for lineage failures (including server errors and cases where upstream returns HTML instead of JSON). * **More reliable private connectivity selection**: Improves private endpoint filtering by adapter type and updates Studio IDE to use the correct version 3 private endpoints endpoint. ##### Canvas * **More reliable Add Sources CSV uploads**: Improves Comma-Separated Values (CSV) upload progress, resume behavior, and common error handling during Add Sources. ##### Catalog * **Faster and more usable lineage for large projects**: Improves directed acyclic graph (DAG) performance by rendering only visible elements and improving layout for disconnected nodes. * **Safer search result interactions**: Improves keyboard and hover behavior in the search dropdown and avoids showing stale results while searches are loading. ##### dbt platform * **More informative user invite statuses**: This change shows clearer invite status (invitation sent and invitation accepted) and supports accepted, login pending for Single Sign-On (SSO). * **Unpaid billing banner enabled by default**: The unpaid billing banner is no longer feature-flagged and will display when applicable, while billing link visibility remains permission-based. * **System for cross-domain identity management (SCIM)**: Bug fixes and improvements related to managed invites for easier processing. ##### dbt Copilot and agents * **Streaming control for server-sent events**: Adds Server-Sent Events (SSE) streaming control so clients can choose chunk streaming or message streaming. This enables more responsive Copilot experiences in environments that support streaming. * **More reliable similar models requests**: Improves responsiveness for AI Similar Models and Similar Sources requests by enforcing tighter embedding and database timeouts aligned to request deadlines. Users should see faster, more consistent results when exploring related models. * **dbt Copilot: Improved bring your own key error handling**: Categorizes OpenAI failures with Bring Your Own Key (BYOK) awareness so BYOK failures return the expected 424-class behavior instead of generic 500-series errors. This makes it easier to diagnose and resolve key or configuration issues. * **Expanded dbt Model Context Protocol tooling**: Updates dbt Model Context Protocol (MCP) tooling, including adding `get_all_macros` and improving error categorization, enabling more accurate responses. #### Fixes ##### Studio IDE and Catalog * **More reliable search and replace**: Ensures bulk edits stay in sync after server-side edits to prevent stale content from overwriting changes. * **Correct search preview highlighting**: Fixes preview and match highlighting assembly so match ranges align correctly in multi-line previews. * **Improved startup failure experience**: Shows a proper error layout and notification on unrecoverable initialization failures. ##### Canvas * **Fewer Add Sources UI interruptions**: Prevents incorrect tab closing after uploads complete and avoids showing the floating node panel when not on a file tab. ##### Catalog * **Public model lineage across environments**: Fixes lineage resolution for public model parents when the producer model lives in a non-default environment. ##### dbt Copilot And Agents * **Reduced resource growth under load**: Fixes an OpenAI connection pool leak that could lead to out-of-memory (OOM) conditions under sustained load. Users should see fewer slowdowns during high-traffic periods. * **Fewer related models timeouts**: Reduces intermittent failures when attaching related models by increasing internal timeouts for related-model fetching. Users should experience fewer timeout errors when working with related models. #### Behavior change ##### Studio IDE * **Prevent destructive root operations**: Prevents rename and delete operations on the repository root and shows clearer warnings. * **Resumable dbt command log streaming**: Improves dbt command log streaming reliability by resuming from the last known Command Line Interface (CLI) event offset. Contact your account manager to enable. ##### Admin And APIs * **Job Admin gains write access in Profiles API**: Job Admin now includes `profiles_write`, which can change what Job Admin users can do where Profiles are enabled. * **Search parameter renamed**: Version 3 Private Endpoints query parameter `name_search` is renamed to `search`, and search matches endpoint name and endpoint value. * **Connections: Postgres database name required**: Postgres connection validation now requires a non-empty database name. * **User credentials: Prevent sharing credentials across users**: Prevents associating the same active credentials object to multiple users, returning a conflict instead of silently duplicating associations. ##### Integrations * **GitHub: More flexible repository URL schemes**: GitHub shared webhooks now accept repository URLs using https, git, and Secure Shell (SSH) formats. * **Slack: Tighter permission gating for settings**: Slack linking and notification settings are more strictly gated by the relevant permissions. * **Slack: Permission check aligned to job notification access**: Slack integration listing now uses job notifications read permission, reducing incorrect permission-denied scenarios. ##### CLI Runtime * **Shorter default request timeouts**: Reduces default timeouts from 60 seconds to 5 seconds for Cloud Config and Cloud Artifact calls, causing requests to fail faster in high-latency environments unless overridden. * **OpenTelemetry logs: Corrected JSON field name**: Corrects the OpenTelemetry (OTel) log payload field name to `additional_message` (from the misspelled `addtional_message`), which may require updates to downstream parsing. #### February 11, 2026 #### Enhancements ##### Catalog * **Faster model graph rendering for large projects**: Improved model graph layout performance to reduce load time in larger projects. * **Faster similar models results**: Similar Models lookup now uses an optimized vector search strategy to reduce timeouts on large projects. ##### Studio IDE * **Clearer project root in Catalog file tree**: When your dbt project is in a subdirectory, the project root is highlighted in the Catalog file tree. * **More native rename and delete in Catalog file tree**: Rename and delete actions now use native editor behaviors when using the Catalog file tree. * **More reliable in-browser formatting**: Formatting updates now apply directly to the active editor buffer to reduce prompts and inconsistent results. * **Cleaner code generation workflow**: Code generation no longer creates a temporary file in your repository during generation. ##### dbt platform * **Fusion compatibility validation on environments**: Environment settings now prevent saving a Fusion dbt version with an incompatible connection and surface field level validation errors. * **Smarter Fusion defaults during connection setup**: When setting up a new connection, Fusion eligible adapters now default to the latest Fusion version to reduce misconfiguration during setup. * **Improved Private Link endpoint management**: Private Endpoints can be sorted by status and connections, and endpoint details now show associated connections and environments. ##### Run Logs * **More reliable invocation event streaming**: Invocation event streaming is more reliable for long running jobs by deriving totals from the latest stream event identifier. * **Reduced Redis usage after log streams complete**: Log streaming now cleans up Redis keys after a stream completes, reducing stale keys and Redis memory pressure for high volume runs. #### Fixes ##### dbt Copilot * **Consistent usage limit messaging in Insights and Studio IDE**: When users hit the usage limit, dbt disables Copilot and shows a clear message, including the reset date when available. ##### Studio IDE * **Git status decorations registered once**: Fixed duplicate Git status decorations in the file tree that could cause visual issues and performance impact. * **Avoid automatic pull on primary branch**: Studio IDE no longer runs an automatic pull on the primary branch to reduce unexpected changes during development. * **Clearer file operation validation errors**: File operations now return structured validation errors and explicitly reject names that exceed operating system limits. * **More reliable command log refresh and finalization**: Command logs for the dbt Cloud Command Line Interface (CLI) are refreshed and finalized more reliably. ##### Run Automation * **Correct account attribution for automatically triggered runs**: Scheduler triggered runs now include account context, improving run attribution and preventing some downstream triggers from running without proper context. * **Reject malformed account identifiers for exposure events**: Exposure generated events now validate that account identifiers are numeric before triggering follow on automation. ##### Webhooks * **More compatible run completion payload for canceled and errored runs**: Webhook payloads now include consistent completion and error timestamps, and canceled runs include a canceled timestamp and normalized status. * **Restored dual dispatch for some failure and completion triggers**: When both failure and completion triggers are configured, errored runs may generate two webhook deliveries to match legacy behavior. ##### dbt Project Metadata * **Manifest Ingestion: Accept functions section in manifest.json**: Ingestion now accepts the `functions` section (for example, Snowflake user defined functions (UDF)) to prevent parse failures on newer manifest schemas. * **Macro Metadata: More consistent timestamps and argument comparison**: Macro metadata persistence now uses more consistent Coordinated Universal Time (UTC) timestamps and improves argument comparison to reduce noisy or incorrect macro updates. #### Behavior change ##### dbt platform APIs * **Removed credential configuration fields from responses**: Profiles API responses no longer include credential configuration and extended attributes; use the appropriate credentials and configuration endpoints instead. * **Filter connections by Private Endpoint**: Account Connections list supports filtering by Private Endpoint identifier for easier management. * **Additional ordering options**: Private Endpoints list now supports ordering by endpoint state and connection count. * **Private Link: Updated license permission defaults**: User licenses now include read access for Private Link resources, which may change who can view Private Link related settings. ##### Studio IDE * **Metric generation writes directly to active file**: Generated metrics are now written directly into the active model file instead of using an accept and reject diff flow. #### February 4, 2026 #### New ##### Studio IDE * **Studio IDE: Copilot link in console toolbar**: Adds a link that opens Copilot from the console toolbar. You can use Copilot to read files and list directories for better context. * **Studio IDE: Copy repo-relative path command**: Adds a command to copy a file path relative to your dbt project subdirectory, making it easier to share paths in runbooks and support tickets. #### Enhancements ##### dbt platform * **dbt platform: Fusion eligibility and compatibility indicators in setup flows**: Improves Fusion setup by showing "Fusion compatible" indicators during connection setup. * **dbt platform: Compare Changes shows partial success warnings**: When Compare Changes subqueries fail, the experience now surfaces a partial success state with expandable warning details to make troubleshooting faster. * **dbt platform: In-progress run logs preserve text selection**: Improves log usability during in-progress runs by preserving text selection while logs auto-refresh and rerender. * **dbt platform: Job completion trigger job picker search**: Adds server-side search and clearer loading and empty states to the job picker for job-completion triggers. * **dbt platform: Job artifacts content types and downloads**: Improves artifact handling for job documentation and run artifacts by strengthening HTML detection, defaulting empty paths to `index.html`, and returning clearer `Content-Type` and download filenames. * **dbt platform: Private Endpoints API listing and pagination improvements**: Improves Private Endpoints API v3 list behavior with validated query parameters, filtering, limit and offset pagination, and `connection_count` in responses. ##### Studio IDE * **Studio IDE: Format file more reliable in subdirectories**: Improves formatting reliability by consistently using the active editor content and a stable repo-relative path when invoking formatting. * **Studio IDE: Better stability for tabs and Git operations**: Reduces errors when working with non-file tabs and improves robustness around tab-close and Git checkout flows. * **Studio IDE: Sidebar layout improvements for embedded panels**: Improves embedded panel sizing to reduce clipping and scrolling issues in the sidebar. * **Studio IDE: Fusion prompts reflect actual eligibility**: Improves Fusion banners and prompts by checking project eligibility via a Fusion status endpoint to reduce confusing prompts for ineligible projects. ##### Catalog and Discovery * **Catalog: Improved cross-project lineage for dbt Mesh**: Improves cross-project lineage ("public ancestors") computation to better match expected external lineage boundaries in dbt Mesh experiences. ##### Insights * **Insights: More reliable Copilot Agent requests and context handoff**: Standardizes Copilot Agent requests to the API and includes active tab content as context to improve reliability of agent runs and handoff. #### Fixes ##### dbt platform * **dbt platform: Webhook form editing more resilient**: Improves webhook subscription editing reliability with asynchronous data and fixes a multiselect focus issue that could cause accidental option selection. * **dbt platform: Run warning emails render correctly**: Fixes HTML email markup that could break rendering for run warning notifications. * **dbt platform: Profiles URLs moved under project dashboard**: Profile create and view routes now live under `/dashboard/:accountId/projects/:projectId/profiles/...`, which may affect bookmarks and direct links. ##### Studio IDE * **Studio IDE: Cleaner command history list**: Removes hidden background commands (such as listing and parsing commands) from command history to reduce noise for users. * **Studio IDE: More reliable inline compile and show output**: Improves robustness of inline compile and show output attachment, including cases with tricky quoting and newlines, reducing missing results during interactive use. * **Studio IDE: More reliable log downloads for dbt commands**: Fixes log download behavior so downloads correctly serve either the active `dbt.log` or the finalized compressed log. * **Studio IDE: More reliable artifact uploads to Microsoft Azure Blob Storage**: Fixes edge cases where gzipped artifacts (such as manifests) could fail to upload due to upload stream handling, improving upload reliability. * **Studio IDE: More stable language server protocol (LSP) sessions in workers**: Reduces noisy disconnect and cleanup errors when multiple websocket connections and processes map to the same invocation, improving session stability. ##### Catalog * **Catalog: Search highlighting displays correctly with multiple matches**: Fixes search result highlighting when the backend returns multiple highlights per field, improving readability of matches. Updates search highlights to display as compact badges with counts for easier scanning of results. * **Catalog: Environment filtering more accurate in search results**: Improves environment-scoped Catalog search filtering by using merged environment identifiers and preserving warehouse-only assets via a dedicated sentinel value. * **Catalog: Public models return empty list when none exist**: Improves behavior for environments with no public models by returning an empty list instead of falling into follow-on query logic. ##### Copilot * **Copilot: More reliable model context protocol (MCP) connections during long tool calls**: Improves keep-alive behavior so connections shut down cleanly when the client disconnects, reducing noisy failures. * **Copilot: Semantic Layer tools only offered when available**: Prevents failing tool calls by hiding Semantic Layer tools when the Semantic Layer is not available for the user or environment. * **Copilot: More accurate HTTP error responses**: Improves error reporting by walking wrapped exceptions and exception groups to return the most specific status code and detail available. * **Copilot: Empty Tool Outputs No Longer Cause Failures**: Treats empty tool outputs as valid results (for example, "no matches") to reduce unnecessary "tool call failed" errors. #### Behavior change ##### dbt platform * **dbt platform: Fusion default dbt version selection more restrictive**: During connection setup, the default dbt version now only defaults to `latest-fusion` when the selected adapter is Fusion-compatible and the project and account are eligible. * **dbt platform: dbt version enforcement now project-aware**: dbt version "allowed version" checks now account for `project_id` across jobs and environments, including Application Programming Interface (API)-triggered runs, improving correctness for overrides and automatic mapping to allowed equivalents when possible. * **dbt platform: Connected app refresh tokens now last 7 days**: Refresh token expiration for connected app OAuth flows increased from 8 hours to 7 days, reducing re-authorization frequency. ##### Studio IDE * **Studio IDE: File stat timestamps now milliseconds**: File stat responses now return modified time and created time as integer milliseconds since epoch instead of float seconds; integrations consuming these endpoints may need to adjust. * **Studio IDE: Language Server Protocol deferral controls expanded**: The Language Server Protocol (LSP) websocket now supports `defer_env_id` to defer against a specific environment and `no_defer=true` to explicitly disable deferral. * **Studio IDE: Deferral toggle applied more consistently to Language Server Protocol connections**: When "defer to production" is turned off, the Studio Integrated Development Environment (IDE) now passes `no_defer=true` to align editor intelligence with the selected deferral behavior. (Language Server Protocol (LSP)) ##### Catalog * **Catalog: Source freshness outdated status removed**: The freshness status value `outdated` was removed; unconfigured freshness is now handled explicitly as `unconfigured`, and sources will no longer report `outdated`. * **Catalog: Rows per page selector removed from tables**: The rows-per-page selector was removed, and pagination now uses a fixed page size. ##### Orchestration and Run Status * **Orchestration: Cached and stale outcome status mapping updated**: Cached nodes are now consistently surfaced as Reused with clearer reasons, and stale outcomes are treated as errors, which can change the statuses operators see in run output and telemetry. #### January 28, 2026 ##### New * **Canvas** * **New two-step "upload source" API for more resilient uploads**: Use `POST /v1/workspaces/{workspace_id}/upload-source` to create an upload, then `PATCH /v1/workspaces/{workspace_id}/upload-source/{file_id}/process` to stream processing progress (SSE). ##### Enhancements * **Catalog & Search** * **Improved search relevance and highlighting**: Ranking now boosts results by modeling layer, and highlighting is more consistent (including support for multiple highlight snippets per field). * **dbt platform** * **Private endpoints details page**: The dbt platform now includes a Private Endpoint details view with endpoint properties, connectivity status, and associated projects. * **Fusion-aware default dbt version during setup**: Connection setup and environment creation can now default to `latest-fusion` for eligible projects. * **Studio IDE** * **Search and replace in files**: Adds a dedicated sidebar search experience. Please contact your account manager to enable. * **Autofix now includes package upgrades**: Upgrade flows can proceed from fixing deprecations into package upgrades in the same guided run. * **Editor UI polish**: Fixed multiple layout/styling issues for a more consistent editor experience. ##### Fixes * **dbt platform** * **Run logs render ANSI/structured output more reliably**: Improved rendering and cleanup of escape sequences in step logs. * **More accurate source freshness status in multi-job environments**: Freshness status is preserved when a run lacks freshness results but freshness remains configured. * **More robust seed artifact ingestion**: Ingestion now tolerates missing/null `schema` fields in the manifest to avoid failures. * **Studio IDE** * **CLI project sync no longer fails on broken symlinks**: Sync skips missing symlink targets instead of failing the whole sync. * **IDE abort is clearer when a command is missing**: Aborting a command that no longer exists returns a specific "no-command-found" response. * **More robust inline command results**: Malformed inline commands no longer break result processing; `show --inline` with an empty result returns an empty preview table. * **Canvas** * **Clearer errors for duplicate uploaded-source names**: Creating an uploaded-source model with a duplicate name now returns HTTP 409 with an actionable message. * **Failed uploads are now visible via file state**: Uploaded-source processing records failure state instead of deleting the file record, improving retry/resume workflows. * **Invocation status streaming reliability**: The invocation status SSE endpoint now correctly awaits the status stream. ##### Behavior changes * **Catalog & Search** * **Search highlight fields deprecated and highlights shape expanded**: `AccountSearchHit.highlight` and `AccountSearchHit.matchedField` are deprecated. `AccountSearchHit.highlights` now supports multiple highlight snippets per field (arrays). * **dbt platform** * **Deprecations**: The "Adaptive" job type is deprecated. `last_checked_at` is deprecated and no longer populated in run responses. * **Canvas** * **Existing CSV upload SSE endpoint deprecated**: Migrate to the new two-step [upload source](https://docs.getdbt.com/docs/platform/use-canvas.md#upload-data-to-canvas) flow. #### January 21, 2026 ##### New * **dbt platform** * **Favorites are now available in Catalog**: Add resources to favorites and organize your frequently accessed resources in the Catalog navigation. * **Connectivity / private networking** * **New v3 API endpoint to fetch a specific PrivateLink endpoint**: You can now retrieve individual PrivateLink endpoints by ID, enabling better automation and troubleshooting workflows. ##### Enhancements * **dbt platform** * **Run artifacts are now searchable**: Find specific artifacts faster in run history with the new artifacts search box and improved empty states. * **Webhooks editor is more stable**: The webhook form no longer resets while job options are loading, and server-generated fields now display reliably after creation. * **Fusion onboarding completion card can be dismissed**: After completing the Fusion onboarding checklist, you can now dismiss the card and it will stay dismissed. * **Cross-project lineage is now generally available**: Cross-project lineage is now enabled for all applicable accounts. * **Catalog & Search** * **Improved Catalog search relevance and performance**: Enhanced search scoring and matching provides more accurate results, with better column matching and highlighting for large catalogs. * **Search results are refreshed when column metadata changes**: Column name and description updates now automatically trigger re-indexing, ensuring search results stay current. * **Search typeahead includes "View all results"**: Quickly access full search results from the typeahead dropdown with the new footer link. * **Cleaner environment dropdown behavior**: The environment selector now only shows "Staging" when your account has projects with a staging environment configured. * **Studio IDE** * **Clearer error messages when fetching dev credentials and defer state**: IDE-related endpoints now return more specific and helpful error messages for common configuration issues and timeouts. * **Studio console and command log viewer improvements**: Enhanced command log viewer with improved download capabilities and more consistent error log viewing. ##### Fixes * **AI-assisted workflows** * **Enhancement:** [dbt ](https://docs.getdbt.com/docs/platform/wizard-platform.md)adds missing column descriptions more accurately. generated documentation now correctly detects column names across various `schema.yml` files, adds only missing descriptions, and preserves existing ones. * **Catalog & lineage** * **Fixes missing auto-generated exposures in model lineage**: Auto-generated exposures now appear correctly in lineage views. * **Catalog search no longer errors when a warehouse connection name is missing**: Search now handles missing connection names gracefully without causing errors. * **Improved security: malformed identity headers are rejected cleanly**: Requests with invalid authentication tokens now fail safely with clear error messages. * **Studio IDE** * **Command status is more reliable when Cloud CLI invocation data expires**: Commands that can't be fetched are now properly marked as failed instead of staying in a "running" state. * **APIs** * **Jobs API deferral validation is stricter and clearer**: Job deferral settings are now validated to ensure the deferring job and environment exist within the same account, with improved error messages. ##### Behavior changes * **dbt platform** * **Account Insights default page size changed to 5 rows**: Tables in Account Insights now display 5 rows per page by default (previously 10). * **Webhooks** * **Webhook timestamps are now consistently UTC RFC3339 with `Z`**: All webhook timestamp fields (`run_started_at`, `run_finished_at`, `timestamp`) now use UTC with `Z` suffix and higher precision. Missing/invalid timestamps emit `1970-01-01T00:00:00Z` instead of empty strings. Update webhook consumers if needed. * **Webhook `run_status` string changed from `Error` to `Errored`**: Update webhook consumers that parse this status value strictly. * **Runs / ingestion** * **Very large exposure sets are now limited during ingestion**: Projects with more than 5,000 exposures will skip exposure ingestion to prevent performance issues. All other artifact ingestion continues normally. Contact support if you need to increase this limit. #### January 14, 2026 ##### New * **dbt platform** * **Fusion migration readiness endpoint**: Added an API endpoint to determine whether a project is eligible for Fusion migration. ##### Enhancements * **Copilot and AI** * **More resilient agent runs**: Agent tool execution errors now return structured responses instead of failing the entire run. * **Better project context retrieval**: Agent toolsets include additional retrieval and search capabilities for more relevant responses. * **Improved Azure OpenAI verification**: Azure OpenAI connection verification now uses GPT-5-compatible parameters for GPT-5 deployments. * **BYOK for Azure OpenAI**: Added support for Azure Foundry URLs with automatic endpoint parsing to reduce setup friction. * **Insights and Catalog** * **Semantic Layer querying now generally available (GA)**: Build SQL queries against the Semantic Layer without writing SQL code. * **Improved search relevance**: Search scoring prioritizes exact and multi-term matches more strongly, with better highlighting and column-description matching. * **Catalog UX improvements**: Search labels are more consistent, and the embedded lineage view loads more responsively. * **Studio IDE** * **Unified Studio IDE**: Studio now loads a single unified IDE package. * **Defer-to-production honors `defer-env-id` override**: Studio now respects `dbt-cloud.defer-env-id` settings when Cloud CLI runtime is supported. * **Improved log exporting**: Download and copy behavior for command logs is more consistent, including debug logs. * **Enhanced multi-edit support**: The IDE now supports multiple explicit edits in one request with safer validation. * **Clearer Cloud CLI session errors**: Session creation returns clearer error messages and guidance for setup issues. * **dbt platform** * **Settings detail pages in resizable drawer**: Settings detail experiences now use an improved drawer-based UI. * **More resilient profile creation**: Profile creation now handles dependencies and failures more gracefully. * **Enhanced logging limits for in-progress runs**: Logs for in-progress runs are also limited by memory usage, in addition to the existing 1,000-line limit. ##### Fixes * **dbt platform** * **Profiles API clearing extended attributes**: The Profiles API now allows unsetting extended attributes by setting `extended_attributes_id` to null. * **Recently viewed more reliable**: Recently viewed entries now update atomically and retain the 5 most recent items. * **Run log tailing improvements**: Debug logs for completed runs now consistently fetch only the tail of the log. * **Studio IDE** * **More reliable `show` and `compile`**: CLI flags to disable caching are now positioned correctly to avoid parsing issues. * **Canvas preview improvements**: Fixed argument ordering so `--no-defer` is interpreted consistently. ##### Behavior changes * **dbt platform** * **dbt v1.7 end-of-life**: dbt v1.7 is now labeled as end-of-life in version lifecycle messaging. #### January 7, 2026 No changes of note this week. #### December 24, 2025 ##### New * **AI Codegen** * **File-aware LangGraph agents**: Analysts can now drop `@path` references in the bundled CLI to stream local files into `/private/v1/agents/run`, which are auto-rendered as text inside the run so copilots have the exact config or SQL snippet you referenced. * **dbt platform** * **Slack Copilot feedback loops**: Copilot replies now carry inline "Did that answer your question?" buttons, so you can rate answers without leaving Slack. * **Codex workflows** * **Databricks cost tracking for Model Cost Over Time**: A Databricks history provider and DBU-based cost query now surface daily model cost alongside Snowflake coverage, so Databricks tenants get unified FinOps reporting. * **Canvas** * **CSV upload GA**: The CSV upload endpoint is now generally available. ##### Enhancements * **Cloud artifacts** * **Better similar-model suggestions**: Attachment workflows now only recommend meaningfully related models. * **dbt platform** * **Unified SSO & SCIM admin**: Settings consolidate SSO + SCIM, add an empty state for auto-generated slugs, and render read-only login URLs so admins can start configuration without touching slug fields. * **SCIM token management polish**: Token tables gain fixed pagination, inline search, consistent iconography, and clearer deletion warnings to avoid accidental cuts to live integrations. * **Twice the per-environment custom variables**: The v3 API/UI now allow up to 20 scoped environment variables before enforcing limits, giving larger projects more room for secrets. * **Canvas** * **Dialect-aware projection SQL**: SELECT \* RENAME/EXCEPT support now respects each warehouse's syntax using schema metadata, so SQL previews and column metadata stay accurate across Snowflake, Databricks, BigQuery, and Redshift. ##### Fixes * **dbt platform** * **Webhook editor keeps job selections**: Default values are cached after the first render and stop resetting once the user edits the form, eliminating accidental job-list clearing while tabbing through fields. * **Codex GraphQL** * **Exposure parents mirror the manifest**: `parentsModels` and `parentsSources` now derive from the manifest's `parents` list, so exposures with mixed upstreams display complete lineage in both the GraphQL API and UI. ##### Behavior changes * **dbt platform** * **Legacy Cost Management UI retired**: All cost management pages and hooks were removed, and platform metadata credentials now only expose catalog ingestion and Cost Insights toggles, eliminating dead-end controls. #### December 17, 2025 ##### New * **dbt platform** * **Feature licensing service**: A new `/accounts/<id>/feature-licenses` endpoint issues short-lived JWTs that encode entitled features, and service/PAT authentication now checks that a caller holds an active license on the target account before any Fusion-enabled workflow runs. * **Databricks platform metadata credentials**: Databricks warehouses can register platform metadata credentials (token plus optional catalog), enabling catalog ingestion, metadata sharing, and Cost Insights pipelines without custom adapters. ##### Enhancements * **dbt platform** * **Large list pagination**: Settings's Projects and Credentials now paginate after 25 rows (with search boxes and skeleton states), keeping navigation responsive for large deployments. * **Metadata Explorer** * **Model context & lineage polish**: Model panels now show materialization type, lineage renders metadata strips only when content exists, and upstream public-model columns load automatically for better cross-project visibility. * **Freshness clarity & Studio navigation**: Source tiles respect the `meta5161ExpiredUnconfiguredSources` flag (showing warn/error thresholds) and "Open in IDE" links now point at `/studio/{accountId}/projects/{projectId}` to drop users directly into dbt Studio. * **Insights UI** * **Copilot guardrails**: The Copilot listener now hydrates builder tabs only when a semantic-layer payload arrives, preventing plain-SQL replies from overwriting editor state. * **dbt CLI** * **Improved monorepo support for file sync and the IDE**: * File sync now anchors itself to the invocation directory, making monorepo structures behave more predictably. * Nested `dependencies.yml` files correctly trigger dependency installs. * The IDE's LSP and file sync now recognize dbt subdirectories properly. * Exclusion lists remain accurate even in multi-project repositories. * **Notifications system** * **Webhook auditability**: Outbound calls now persist the exact JSON body in webhook history, making allowlisting and troubleshooting easier. * **Studio** * **Git sidebar & file refresh parity**: The file tree now mirrors Cloud VCS statuses (including conflicts) and automatically invalidates caches after `dbt deps`/`dbt clean`, so new or removed files appear without a reload. * **Log viewers & Autofix UX**: Command and interactive query logs adopt the new accordion-based viewer, and Autofix sessions in Fusion treat plain `parse` commands as the trigger for deprecation summaries, keeping remediation flows consistent. ##### Fixes * **dbt platform** * **Environment variable editor stability**: Editing one variable no longer backfills blank cells with previously edited values, preventing accidental overrides. * **Cost optimization indicator accuracy**: Job pages once again display "Cost optimization features" whenever Fusion actually runs (and gating conditions are met), so users see the right coverage status regardless of feature-flag permutations. ##### Behavior changes * **dbt platform** * **Stronger tenant identity enforcement**: Service/PAT calls without an active license now fail authentication, Slack Copilot sessions build a scoped identity JWT for the invoking user, and SSO providers enforce auto-generated slugs (draft configs can't be targeted), reducing misconfiguration risk. * **dbt CLI** * **User-isolated invocation history**: Every invocation lookup validates the caller's user ID, preventing admins from accidentally reading another developer's runs when multiple accounts share a CLI server. * **IDE server** * **Enhanced security for support-assisted sessions:** Support impersonation sessions now restrict the execution of `show`, `run`, `build`, and `test` commands. Artifacts generated by `dbt show` are also short-lived and will automatically expire after 15 minutes to limit unintended data retention. * **dbt Orchestration** * **Fusion compare support & new dependency**: Fusion tracks now treat `dbt compare` as a supported command (no more target-path hacks). #### December 10, 2025 ##### Enhancements * **AI codegen API**: Streaming middleware enforces request-scoped instrumentation across every AI endpoint, offload warehouse calls via threads, and expose human-readable tool names while gating keyword search behind feature flag for approved tenants. * **dbt platform** * **Operations clarity**: Environment profile drawers link directly to connection settings and treat Snowflake fields as optional, while Compare Changes and run-step drawers now explain whether steps failed or were skipped so troubleshooting is faster. * **Collaboration & notifications**: Slack Copilot mentions are now more reliable, with hardened workers, support for CSV attachments, and improved logging. Webhook channels now accept longer URLs, handle "warning-only" subscriptions correctly, and automatically clean up corrupted job IDs. * **Profile & credential management**: Environment APIs accept `secondary_profile_ids`, run acquisition favors profile-backed credentials, and whoami/auth metrics are scrubbed so cross-platform profiles stay in sync. * **dbt CLI server**: Improved stability and performance for large projects. * **Studio IDE**: For dbt Fusion logging, node start and end times will now properly be displayed in command output. * **Studio IDE**: Copilot Chat automatically appears anywhere AI entitlements exist, preview runs auto-cancel when nodes change, and keyboard shortcuts respect native keymaps with clear UI labels. * **Studio IDE**: Tab view, console pane, and command drawer have been redesigned to enhance efficiency and multitasking. ##### Fixes * **Studio IDE server**: Branch creation now returns explicit feedback for bad branch names/SHAs and detects unauthorized Git errors earlier, making automation failures actionable. #### December 3, 2025 ##### New * **dbt platform** * **Autofix deprecation warnings**: When deprecations are detected, you now see "Autofix deprecation warnings." * **Autofix Packages detailed results**: After running Autofix, you see a results panel with upgraded packages (with links), packages left unchanged and why, and quick access to `packages.yml` to help assess Fusion readiness and next steps. ##### Enhancements * **dbt platform** * **Code Quality tab improvements** * Clearer lint/format actions (SQLFluff, Prettier), better empty states, visible Config button when applicable, and simplified logs retrieval. * Applies to SQL, JSON, YAML, and Markdown workflows. * **Editor experience** * Upgraded editor for stability. * Improved container sizing/overflow. * "Save" overlay only appears when tabs are open. * Minor action‑bar refinements. ##### Fixes * **dbt platform lineage and command pane stability**: Reliability improved by aligning with updated IDE and VS Code command APIs; eliminates intermittent skips. ##### Behavior changes * **dbt platform:** dbt Core "versionless" renamed to "latest" so it's consistent and clear across tenants. --- ### Write queries with exports dbt platform | Starter, Enterprise, Enterprise+ Exports enhance [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md) by running your saved queries and writing the output to a table or view within your data platform. Saved queries are a way to save and reuse commonly used queries in MetricFlow, exports take this functionality a step further by: * Enabling you to write these queries within your data platform using the dbt job scheduler. * Providing an integration path for tools that don't natively support the Semantic Layer by exposing tables of metrics and dimensions. Essentially, exports are like any other table in your data platform — they enable you to query metric definitions through any SQL interface or connect to downstream tools without a first-class [Semantic Layer integration](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md). Running an export counts towards [queried metrics](https://docs.getdbt.com/docs/platform/billing/how-pricing-works.md#what-counts-as-a-queried-metric) usage. Querying the resulting table or view from the export does not count toward queried metric usage. #### Prerequisites * You have a dbt account on a [Starter or Enterprise-tier](https://www.getdbt.com/pricing/) plan. * You use one of the following data platforms: Snowflake, BigQuery, Databricks, Redshift, or Postgres. * You are on [dbt version](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) 1.7 or newer. * You have the Semantic Layer [configured](https://docs.getdbt.com/docs/use-dbt-semantic-layer/setup-sl.md) in your dbt project. * You have a dbt environment with the [job scheduler](https://docs.getdbt.com/docs/deploy/job-scheduler.md) enabled. * You have a [saved query](https://docs.getdbt.com/docs/build/saved-queries.md) and [export configured](https://docs.getdbt.com/docs/build/saved-queries.md#configure-exports) in your dbt project. In your configuration, leverage [caching](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-cache.md) to cache common queries, speed up performance, and reduce compute costs. #### Benefits of exports The following section explains the main benefits of using exports:  DRY representation Currently, creating tables often involves generating tens, hundreds, or even thousands of tables that denormalize data into summary or metric mart tables. The main benefit of exports is creating a "Don't Repeat Yourself (DRY)" representation of the logic to construct each metric, dimension, join, filter, and so on. This allows you to reuse those components for long-term scalability, even if you're replacing manually written SQL models with references to the metrics or dimensions in saved queries.  Easier changes Exports ensure that changes to metrics and dimensions are made in one place and then cascade to those various destinations seamlessly. This prevents the problem of needing to update a metric across every model that references that same concept.  Caching Use exports to pre-populate the cache, so that you're pre-computing what you need to serve users through the dynamic Semantic Layer APIs. ###### Considerations Exports offer many benefits and it's important to note some use cases that fall outside the advantages: * Business users may still struggle to consume from tens, hundreds, or thousands of tables, and choosing the right one can be a challenge. * Business users may also make mistakes when aggregating and filtering from the pre-built tables. For these use cases, use the dynamic [Semantic Layer APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md) instead of exports. #### Run exports Before you're able to run exports in development or production, you'll need to make sure you've [configured saved queries and exports](https://docs.getdbt.com/docs/build/saved-queries.md) in your dbt project. In your saved query config, you can also leverage [caching](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-cache.md) with the dbt job scheduler to cache common queries, speed up performance, and reduce compute costs. There are two ways to run an export: * [Run exports in development](#exports-in-development) to test the output before production. In Studio IDE or any environment on dbt v2, run `dbt build` instead — enable the [environment variable](#set-environment-variable) first. In the [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) outside dbt v2, run `dbt sl export` or `dbt sl export-all`. * [Run exports in production](#exports-in-production) using the [dbt job scheduler](https://docs.getdbt.com/docs/deploy/job-scheduler.md) to write these queries within your data platform. #### Exports in development You can run an export in your development environment if you want to test its output before production. This section explains the different commands and options available to run exports in development. If your environment runs on dbt v2 (including Studio IDE), use `dbt build` instead of the following commands. Check out [Run exports](#run-exports) for details. * Use the [`dbt sl export` command](#exports-for-single-saved-query) to test and generate exports in your development environment for a singular saved query. You can also use the `--select` flag to specify particular exports from a saved query. * Use the [`dbt sl export-all` command](#exports-for-multiple-saved-queries) to run exports for multiple saved queries at once. This command provides a convenient way to manage and execute exports for several queries simultaneously, saving time and effort. ##### Exports for single saved query These commands apply when you're using the dbt platform CLI outside Studio IDE or dbt v2. If your environment runs on dbt v2, use `dbt build` instead. For more info, check out [Run exports](#run-exports). Use the following command to run exports in the dbt platform CLI: ```bash dbt sl export ``` The following table lists the options for `dbt sl export` command, using the `--` flag prefix to specify the parameters: | Parameters | Type | Required | Description | | ------------- | -------------- | -------- | --------------------------------------------------------------------------------------------------------------------- | | `name` | String | Required | Name of the `export` object. | | `saved-query` | String | Required | Name of a saved query that could be used. | | `select` | List or String | Optional | Specify the names of exports to select from the saved query. | | `exclude` | String | Optional | Specify the names of exports to exclude from the saved query. | | `export_as` | String | Optional | Type of export to create from the `export_as` types available in the config. Options available are `table` or `view`. | | `schema` | String | Optional | Schema to use for creating the table or view. | | `alias` | String | Optional | Table alias to use to write the table or view. | You can also run any export defined for the saved query and write the table or view in your development environment. Refer to the following command example and output: ```bash dbt sl export --saved-query sq_name ``` The output would look something like this: ```bash Polling for export status - query_id: 2c1W6M6qGklo1LR4QqzsH7ASGFs.. Export completed. ``` ##### Use the select flag You can have multiple exports for a saved query and by default, all exports are run for a saved query. You can use the `select` flag in [development](#exports-in-development) to select specific or multiple exports. You can't sub-select metrics or dimensions from the saved query. You can only change the export configuration, such as the table format or schema. For example, the following command runs `export_1` and `export_2` and doesn't work with the `--alias` or `--export_as` flags: ```bash dbt sl export --saved-query sq_name --select export_1,export2 ``` Overriding export configurations The `--select` flag is mainly used to include or exclude specific exports. If you need to change these settings, you can use the following flags to override export configurations: * `--export-as` — Defines the materialization type (table or view) for the export. This creates a new export with its own settings and is useful for testing in development. * `--schema` — Specifies the schema to use for the written table or view. * `--alias` — Assigns a custom alias to the written table or view. This overrides the default export name. Be careful. The `--select` flag *can't* be used with `alias` or `schema`. For example, you can use the following command to create a new export named `new_export` as a table: ```bash dbt sl export --saved-query sq_number1 --export-as table --alias new_export ``` ##### Exports for multiple saved queries Use the command, `dbt sl export-all`, to run exports for multiple saved queries at once. This is different from the `dbt sl export` command, which only runs exports for a singular saved query. For example, to run exports for multiple saved queries, you can use: ```bash dbt sl export-all ``` The output would look something like this: ```bash Exports completed: - Created TABLE at `DBT_SL_TEST.new_customer_orders` - Created VIEW at `DBT_SL_TEST.new_customer_orders_export_alias` - Created TABLE at `DBT_SL_TEST.order_data_key_metrics` - Created TABLE at `DBT_SL_TEST.weekly_revenue` Polling completed ``` The command `dbt sl export-all` provides the flexibility to manage multiple exports in a single command. #### Exports in production Enabling and executing exports in dbt optimizes data workflows and ensures real-time data access. It enhances efficiency and governance for smarter decisions. Exports use the default credentials of the production environment. To enable exports to run saved queries and write them within your data platform, perform the following steps: 1. [Set an environment variable](#set-environment-variable) in dbt. 2. [Create and execute export](#create-and-execute-exports) job run. ##### Set environment variable 1. Click **Deploy** in the top navigation bar and choose **Environments**. 2. Select **Environment variables**. 3. [Set the environment variable](https://docs.getdbt.com/docs/build/environment-variables.md#setting-and-overriding-environment-variables) key to (Applies to dbt v1.11 and later) `DBT_ENGINE_EXPORT_SAVED_QUERIES` and the environment variable's value to `TRUE`. Doing this ensures saved queries and exports are included in your dbt build job. For example, running `dbt build -s sq_name` runs the equivalent of `dbt sl export --saved-query sq_name` in the dbt Job scheduler. If exports aren't needed, you can set the value(s) to `FALSE` ((Applies to dbt v1.11 and later) `DBT_ENGINE_EXPORT_SAVED_QUERIES=FALSE`). [![Add an environment variable to run exports in your production run.](/img/docs/dbt-platform/semantic-layer/env-var-dbt-exports.png?v=2 "Add an environment variable to run exports in your production run.")](#)Add an environment variable to run exports in your production run. When you run a build job, any saved queries downstream of the dbt models in that job will also run. To make sure your export data is up-to-date, run the export as a downstream step (after the model). ##### Create and execute exports 1. Create a [deploy job](https://docs.getdbt.com/docs/deploy/deploy-jobs.md) and ensure the (Applies to dbt v1.11 and later) `DBT_ENGINE_EXPORT_SAVED_QUERIES=TRUE` environment variable is set, as described in [Set environment variable](#set-environment-variable). * This enables you to run any export that needs to be refreshed after a model is built. * Use the [selector syntax](https://docs.getdbt.com/reference/node-selection/syntax.md) `--select` or `-s` option in your build command to specify a particular dbt model or saved query to run. For example, to run all saved queries downstream of the `orders` semantic model, use the following command: ```bash dbt build --select orders+ ``` 2. After dbt finishes building the models, the MetricFlow Server processes the exports, compiles the necessary SQL, and executes this SQL against your data platform. It directly executes a "create table" statement so the data stays within your data platform. 3. Review the exports' execution details in the jobs logs and confirm the export was run successfully. This helps troubleshoot and to ensure accuracy. Since saved queries are integrated into the dbt DAG, all outputs related to exports are available in the job logs. 4. Your data is now available in the data platform for querying! 🎉 #### FAQs  Can I have multiple exports in a single saved query? Yes, this is possible. However, the difference would be the name, schema, and materialization strategy of the export.  How do I run all exports for a saved query? * In production runs, you can build the saved query by calling it directly in the build command, or you build a model and any exports downstream of that model. * In development, run all exports by running `dbt sl export --saved-query sq_name`. If your environment runs on dbt v2 (including Studio IDE), run `dbt build` instead.  Will I run duplicate exports if multiple models are downstream of my saved query? dbt will only run each export once even if it builds multiple models that are downstream of the saved query. For example, you could have a saved query called `order_metrics`, which has metrics from both the `orders` and `order_items` semantic models. You can run a job that includes both models using `dbt build`. This runs both the `orders` and `order_items` models, however, it will only run the `order_metrics` export once.  Can I reference an export as a dbt model using ref() No, you won't be able to reference an export using `ref`. Exports are treated as leaf nodes in your DAG. Modifying an export could lead to inconsistencies with the original metrics from the Semantic Layer.  How can I select saved\_queries by their resource type? To include all saved queries in the dbt build run, use the [`--resource-type` flag](https://docs.getdbt.com/reference/global-configs/resource-type.md) and run the command `dbt build --resource-type saved_query`. #### Related docs * [Validate semantic nodes in a CI job](https://docs.getdbt.com/docs/deploy/ci-jobs.md#semantic-validations-in-ci) * Configure [caching](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-cache.md) * [Semantic Layer FAQs](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-faqs.md) --- ## Faqs ### [Error] Could not find my_project package If a package name is included in the `search_order` of a project-level `dispatch` config, dbt expects that package to contain macros which are viable candidates for dispatching. If an included package does not contain *any* macros, dbt will raise an error like: ```shell Compilation Error In dispatch: Could not find package 'my_project' ``` This does not mean the package or root project is missing—it means that any macros from it are missing, and so it is missing from the search spaces available to `dispatch`. If you've tried the step above and are still experiencing this behavior - reach out to the Support team at <support@getdbt.com> and we'll be happy to help! --- ### Account-specific features The features in dbt are tailored to each organization's unique configuration, including user permissions, project setup, and subscription level, with guidance provided to help teams make the most of their available capabilities. This document provides a comprehensive overview of account-specific features in dbt according to plan type. #### Copilot [dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-platform.md) is an AI-powered assistant designed to accelerate your development workflow and help you focus on delivering high-quality data. dbt Wizard is available to all users in dbt but limits are imposed according to plan type. Have a look at [dbt's pricing](https://www.getdbt.com/pricing) for more information. #### Copilot features ##### Codegen [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") dbt Wizard codegen refers to the code generation capabilities provided by dbt Wizard, an AI-powered assistant integrated into dbt. This feature allows users to generate SQL code, documentation, tests, and semantic models directly from natural language prompts, helping automate and accelerate common analytics engineering workflows.⁠⁠⁠⁠ dbt Wizard codegen uses metadata such as relationships, lineage, and model context from your dbt projects to produce contextually accurate code. This helps avoid mistakes common with generic AI tools by ensuring generated code matches your actual schema and conventions.⁠⁠⁠⁠ The code dbt Wizard generates may include: * Base/staging/semantic models (including SQL for new models) * YAML files for documentation or tests * Inline SQL expressions * Semantic model structures and metrics dbt Wizard codegen is available in the Studio IDE, Canvas, and (soon) Insights, making it possible to generate and edit code directly within these interfaces.⁠ ##### Bring your own key (BYOK) [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") BYOK allows users to provide and manage their own encryption or API keys, rather than relying on keys managed by a vendor or third party. This gives organizations greater control over data security, compliance, and contracts. BYOK means users can bring and configure their own OpenAI, Anthropic, or Azure AI Foundry credentials. With BYOK, users have more control over privacy, observability, and security for their data and metadata. Take note of the following when using BYOK: * When you use your own API key, your contract with the LLM provider (not dbt Labs') applies. You are responsible for managing costs, usage limits, and data handling. This means ownership and liability for API use rests with the user, not dbt Labs. * dbt Labs does not impose usage limits on the user's key, as it does with internally managed keys. * OpenAI projects with [data residency controls](https://platform.openai.com/docs/guides/your-data#data-residency-controls) enabled and configured for the United States (project region set to US) don't currently support BYOK. These projects can only use the API key in the dbt platform configuration. Specifying custom endpoints required for data residency isn’t yet supported, and we’re evaluating a solution for this. To use BYOK, ensure your OpenAI project doesn’t have data residency controls enabled. Projects without project region settings will use the standard OpenAI endpoint (`https://api.openai.com`) and support BYOK. * Currently, BYOK in dbt supports OpenAI and Azure-hosted OpenAI API keys. Users enter their key through the [account settings](https://docs.getdbt.com/docs/platform/account-settings.md), and requests made by dbt Wizard or other AI features are billed directly to the customer by the respective provider.⁠⁠⁠⁠ info The dbt Wizard experience with BYOK and Azure OpenAI will not use metadata information in Insights, Canvas, or the Studio IDE. Without this contextual data, the LLM's responses may be suboptimal compared to those generated by the default dbt AI service. This is a temporary limitation, and we are working on an update that will enable the use of Azure OpenAI APIs. If you choose to BYOK, we don't monitor or collect any data related to your usage. Some of the reasons organizations require BYOK include: * Regulatory and compliance demands (for example, keeping encryption keys or sensitive operations under customer control) * Assurances about how and where data is processed * Ability to negotiate and manage their own vendor contracts * Ability to collect their own observability metrics Note that BYOK is different from bring your own cloud (BYOC). BYOK refers to key or credential management, whereas BYOC refers to running software workloads in your own cloud environment. ##### Natural language in Canvas [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") Natural language in Canvas refers to the ability to build data models visually in Canvas using plain language prompts, powered by GenAI (dbt Wizard). You can describe what you want to build or transform, and the tool generates the underlying SQL and transformation steps for you. No SQL expertise is required. It's aimed at making data modeling more accessible to less-technical users or anyone who prefers a drag-and-drop or conversational interface over hand-coding SQL.⁠⁠ Natural language lets users translate business questions or transformation requests directly into data workflows. This accelerates the process of creating governed, production-ready models while maintaining best practices and version control. You can edit Canvas models collaboratively, and you can see both the graphical workflow and the SQL code it produces.⁠⁠⁠⁠ The natural language capability is fully integrated into the Canvas workspace. You can start with a blank model and generate models or transformation steps by specifying requirements in everyday language. dbt Wizard interprets the request, constructs the model in the Canvas, and presents it visually — making it easy to refine, preview, and publish changes.⁠⁠ This approach is especially valuable for analysts and business users, allowing broader participation in data transformation tasks without losing dbt's governance, reproducibility, and code review processes. #### Canvas [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") Canvas enables efficient data access and transformation through a visual interface, combining the benefits of code-driven development with AI-assisted code generation for a seamless, flexible experience. #### dbt Insights [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") Insights is an interactive feature in dbt designed for writing, running, and analyzing SQL queries within an intuitive interface. It brings together SQL query execution, results visualization, and integration with dbt metadata and documentation — all in one place.⁠⁠ It supports key features such as query history, the ability to export results to CSV, basic charting (for example, line and bar charts), and direct links to Catalog and the Studio IDE for a seamless workflow between exploration and development.⁠⁠⁠⁠ Analysts can quickly analyze metrics across data, while engineers can leverage context, metadata, and dbt lineage details to debug or validate data models.⁠⁠⁠⁠ You can save and share frequently used SQL queries, and explore documentation or data lineage as you work. Each query's results are, for now, limited to 500 rows (with plans to increase this).⁠⁠ The interface supports syntax highlighting, code completion, asset linking (to easily reference dbt models/tables), and connects to the Semantic Layer for querying metrics or columns by name. While Insights supports some light visualizations and query sharing, it is not intended to replace BI tools for reporting or dashboarding. Instead, it's focused on fast ad hoc analysis and insight generation. Integrations allow users to "jump off" into downstream BI tools with their queries if needed.⁠ #### dbt Mesh cross platform [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") dbt Mesh cross-platform (sometimes called "cross-platform Mesh" or "cross-platform dbt Mesh") is a capability in dbt Mesh that allows for referencing models and sharing lineage across multiple dbt projects, even when those projects use different data warehouse platforms. #### SCIM [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") SCIM (System for Cross-Domain Identity Management) automates user identities and groups, enhancing security and simplifying admin tasks. It allows for real-time user provisioning, deprovisioning, and profile updates in dbt, primarily using Okta as the identity provider. #### Hybrid projects [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") Hybrid projects refer to a setup where both dbt v1 and dbt are utilized within the same organization, often working on the same codebase or data platform. This approach enables different teams or contributors to work in the environment that aligns best with their preferences or workflows, while still benefiting from shared assets and centralized metadata. #### Enterprise security [Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") Enterprise security includes robust capabilities for managing network access and user permissions, designed to safeguard sensitive data. Two widely used features that support these efforts are PrivateLink and IP allowlisting. ##### PrivateLink PrivateLink provides a secure and private connection between your organization's environments, such as databases, version control systems, or data warehouses and dbt. This setup ensures that traffic remains within the AWS network, avoiding exposure to the public internet. ##### IP allowlist IP restrictions (IP allowlist/blocklist) let organizations control which IPs can access their dbt account. #### Projects and run slots The number of projects and run slots available to your organization varies based on your selected plan tier. For detailed information, please refer to our [pricing page](https://www.getdbt.com/pricing). #### Upgrade plan dbt offers a range of plans with varying features to suit different organizational needs. For information on the different plan types and upgrading your plan, refer to our document on [How to upgrade a dbt Cloud account](https://docs.getdbt.com/faqs/Accounts/platform-upgrade-instructions.md). #### Related content * [Billing](https://docs.getdbt.com/docs/platform/billing.md) --- ### Add a seed file 1. Add a seed file: seeds/country\_codes.csv ```text country_code,country_name US,United States CA,Canada GB,United Kingdom ... ``` 2. Run `dbt seed` 3. Ref the model in a downstream model models/something.sql ```sql select * from {{ ref('country_codes') }} ``` --- ### Are the results of freshness stored anywhere? Yes! The `dbt source freshness` command will output a pass/warning/error status for each table selected in the freshness snapshot. Additionally, dbt will write the freshness results to a file in the `target/` directory called `sources.json` by default. You can also override this destination, use the `-o` flag to the `dbt source freshness` command. After enabling source freshness within a job, configure [Artifacts](https://docs.getdbt.com/docs/deploy/artifacts.md) in your **Project Details** page, which you can find by selecting your account name on the left side menu in dbt and clicking **Account settings**. You can see the current status for source freshness by clicking **View Sources** in the job page. --- ### Are there any example dbt projects? Yes! * **Quickstart Tutorial:** You can build your own example dbt project in the [quickstart guide](https://docs.getdbt.com/docs/get-started-dbt.md) * **Jaffle Shop:** A demonstration project (closely related to the tutorial) for a fictional e-commerce store. Refer to [Clone the Jaffle Shop sample project](https://docs.getdbt.com/guides/clone-jaffle-shop.md) for clone instructions, or view the [main source code](https://github.com/dbt-labs/jaffle-shop) and [source code using DuckDB](https://github.com/dbt-labs/jaffle_shop_duckdb) on GitHub. * **GitLab:** Gitlab's internal dbt project is open source and is a great example of how to use dbt at scale ([source code](https://gitlab.com/gitlab-com/content-sites/handbook/blob/main/content/handbook/enterprise-data/platform/dbt-guide.md)) * **dummy-dbt:** A containerized dbt project that populates the Sakila database in Postgres and populates dbt seeds, models, snapshots, and tests. The project can be used for testing and experimentation purposes ([source code](https://github.com/gmyrianthous/dbt-dummy)) * **Google Analytics 4:** A demonstration project that transforms the Google Analytics 4 BigQuery exports to various models ([source code](https://github.com/stacktonic-com/stacktonic-dbt-example-project), [docs](https://stacktonic.com/article/google-analytics-big-query-and-dbt-a-dbt-example-project)) * **Make Open Data:** A production-grade ELT with tests, documentation, and CI/CD (GHA) about French open data (housing, demography, geography, etc). It can be used to learn with voluminous and ambiguous data. Contributions are welcome ([source code](https://github.com/make-open-data/make-open-data), [docs](https://make-open-data.fr/)) If you have an example project to add to this list, suggest an edit by clicking **Edit this page** below. --- ### Branches when changing Git providers or repositories This FAQ explains how Git branch names and commit history carry over when you switch Git providers (for example, GitHub or GitLab), start using a new repository, or reconnect your project to a different remote. It is for dbt v1 and dbt platform users: general Git behavior about remotes, branches, and history applies to both; sections on the Studio IDE, managed repositories, and disconnecting a project repository apply only to the dbt platform. Your branch names and history only appear on a new Git provider if you copied the entire repository with its history. If you only copied files or created a fresh repository, old branches do not show up. If you use the dbt platform and the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md), work you saved there but have not pushed is separate and could be lost. The following sections cover common scenarios, what to expect in each case, and what action to take to keep your branches and history intact:  Git provider migration If you fully migrate the same repository to a new Git provider so all commits and branches are preserved, your branches and history typically carry over. If you only copy files, create a new repository with a single initial commit, or import only the default branch, other branches will not appear on the new remote until you push or import them. For steps to disconnect and reconnect your repository in the dbt platform, refer to [How to migrate git providers](https://docs.getdbt.com/faqs/Git/git-migration.md).  Branches missing after creating a new repository Branches may not appear after you create a new repository. When you create a brand-new repository, it starts with only the commits and branches that exist in that new remote. Branches from your previous repository do not appear automatically, even if the files look similar. If you need earlier branches and commit history, migrate the original repository so the full commit history and branches are retained. If you continue with a newly created repository, push any branches you want to keep from your local clone. If you use the dbt platform and need to disconnect or reconnect your project repository, follow [How to migrate git providers](https://docs.getdbt.com/faqs/Git/git-migration.md).  Project repository connection In the dbt platform, the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) shows branches from the Git remote connected to your project. It does not maintain a second, independent copy of your full commit history and branches. Before you disconnect the repository integration in dbt platform or switch to a different remote, save and push work you need to keep. Uncommitted changes are at risk until you **Commit and sync** (push). Until then they will not appear in your Git provider (for example, GitHub). For disconnect steps, refer to [How to migrate git providers](https://docs.getdbt.com/faqs/Git/git-migration.md). Unsaved edits in the browser can also be lost. **Commit and sync** what you need to keep, or copy it outside the Studio IDE. For information on how work is stored in the Studio IDE, refer to [Work retention](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md?version=1.12#about-the-start-up-process-and-work-retention). Pre-cutover verification To confirm branches and connectivity before switching over, create a separate project in the dbt platform pointed at the new repository, verify the branches you expect, then disconnect the old repository and remove the temporary project when finished.  From managed to self-hosted Exporting a project as a ZIP is a snapshot of files only and does not preserve your Git history or branches. To migrate properly and keep your full history, follow [Move from a managed repository to a self-hosted repository](https://docs.getdbt.com/faqs/Git/managed-repo.md). --- ### Can I add tests and descriptions in a SQL config block? dbt has the ability to define node configs in YAML files, in addition to `config()` blocks and `dbt_project.yml`. But the reverse isn't always true: there are some things in `.yml` files that can *only* be defined there. Certain properties are special, because: * They have a unique Jinja rendering context * They create new project resources * They don't make sense as hierarchical configuration * They're older properties that haven't yet been redefined as configs These properties are: * [`description`](https://docs.getdbt.com/reference/resource-properties/description.md) * [`tests`](https://docs.getdbt.com/reference/resource-properties/data-tests.md) * [`docs`](https://docs.getdbt.com/reference/resource-configs/docs.md) * `columns` * [`quote`](https://docs.getdbt.com/reference/resource-properties/columns.md#quote) * [`source` properties](https://docs.getdbt.com/reference/source-properties.md) (e.g. `loaded_at_field`, `freshness`) * [`exposure` properties](https://docs.getdbt.com/reference/exposure-properties.md) (e.g. `type`, `maturity`) * [`macro` properties](https://docs.getdbt.com/reference/resource-properties/arguments.md) (e.g. `arguments`) --- ### Can I build my models in a schema other than my target schema or split my models across multiple schemas? Yes! Use the [schema](https://docs.getdbt.com/reference/resource-configs/schema.md) configuration in your `dbt_project.yml` file, or using a `config` block: dbt\_project.yml ```yml name: jaffle_shop ... models: jaffle_shop: marketing: +schema: marketing # models in the `models/marketing/` subdirectory will use the marketing schema ``` models/customers.sql ```sql {{ config( schema='core' ) }} ``` --- ### Can I build my seeds in a schema other than my target schema or can I split my seeds across multiple schemas? Yes! Use the [schema](https://docs.getdbt.com/reference/resource-configs/schema.md) configuration in your `dbt_project.yml` file. dbt\_project.yml ```yml name: jaffle_shop ... seeds: jaffle_shop: +schema: mappings # all seeds in this project will use the schema "mappings" by default marketing: +schema: marketing # seeds in the "seeds/marketing/" subdirectory will use the schema "marketing" ``` --- ### Can I connect my dbt project to two databases? The meaning of the term 'database' varies with each major warehouse manager. Hence, the answer to "can a dbt project connect to more than one database?" depends on the warehouse used in your tech stack. * dbt projects connecting to warehouses like Snowflake or Bigquery—these empower one set of credentials to draw from all datasets or 'projects' available to an account—are *sometimes* said to connect to more than one database. * dbt projects connecting to warehouses like Redshift and Postgres—these tie one set of credentials to one database—are said to connect to one database only. Sidestep the 'one database problem' by relying on ELT thinking (i.e. extract -> load -> transform). Remember, dbt is not a loader--with few exceptions, it doesn't move data from sources to a warehouse. dbt is a transformer. It enters the picture after extractors and loaders have funneled sources into a warehouse. It moves and combines data inside the warehouse itself. Hence, instead of thinking "how do I connect my dbt project to two databases", ask "what loader services will best prepare our warehouse for dbt transformations." For more on the modern 'ELT-powered' data stack, see the "dbt and the modern BI stack" section of this [dbt blog post](https://blog.getdbt.com/what-exactly-is-dbt). --- ### Can I define private packages in the dependencies.yml file? It depends on how you're accessing your private packages: * If you're using [native private packages](https://docs.getdbt.com/docs/build/packages.md#native-private-packages), you can define them in the `dependencies.yml` file. * If you're using the [git token method](https://docs.getdbt.com/docs/build/packages.md#git-token-method), you must define them in the `packages.yml` file instead of the `dependencies.yml` file. This is because conditional rendering (like Jinja-in-yaml) is not supported in `dependencies.yml`. --- ### Can I document things other than models, like sources, seeds, and snapshots? Yes! You can document almost everything in your project using the `description:` key. Check out the reference docs on [descriptions](https://docs.getdbt.com/reference/resource-properties/description.md) for more info! --- ### Can I pay via invoice? Currently for Starter plans, self-service dbt payments must be made with a credit card and by default, they will be billed monthly based on the number of [active developer seats and usage](https://docs.getdbt.com/docs/platform/billing.md). We don't have any plans to do invoicing for self-service teams in the near future, but we *do* currently support invoices for companies on the **dbt Enterprise or Enterprise+ plans.** Feel free to [contact us](https://www.getdbt.com/contact) to build your Enterprise pricing. --- ### Can I set a different connection at the environment level? dbt supports [Connections](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md#connection-management), available to all dbt users. Connections allows different data platform connections per environment, eliminating the need to duplicate projects. Projects can only use multiple connections of the same warehouse type. Connections are reusable across projects and environments. In dbt v1, you can maintain separate production and development environments through the use of [`targets`](https://docs.getdbt.com/reference/dbt-jinja-functions/target.md) within a [profile](https://docs.getdbt.com/docs/local/profiles.yml.md). dbt v1 users can define different targets in their profiles.yml, which means you can have targets for different data warehouses for the same profile. --- ### Can I set test failure thresholds? You can use the `error_if` and `warn_if` configs to set custom failure thresholds in your tests. For more details, see [reference](https://docs.getdbt.com/reference/resource-configs/severity.md) for more information. You can also try the following solutions: * Setting the [severity](https://docs.getdbt.com/reference/resource-configs/severity.md) to `warn` or `error` * Writing a [custom generic test](https://docs.getdbt.com/best-practices/writing-custom-generic-tests.md) that accepts a threshold argument ([example](https://discourse.getdbt.com/t/creating-an-error-threshold-for-schema-tests/966)) --- ### Can I store my data tests in a directory other than the `tests` directory in my project? By default, dbt expects your singular data test files to be located in the `tests` subdirectory of your project, and generic data test definitions to be located in `tests/generic` or `macros`. To change this, update the [test-paths](https://docs.getdbt.com/reference/project-configs/test-paths.md) configuration in your `dbt_project.yml` file, like so: dbt\_project.yml ```yml test-paths: ["my_cool_tests"] ``` Then, you can define generic data tests in `my_cool_tests/generic/`, and singular data tests everywhere else in `my_cool_tests/`. --- ### Can I store my models in a directory other than the `models` directory in my project? By default, dbt expects the files defining your models to be located in the `models` subdirectory of your project. To change this, update the [model-paths](https://docs.getdbt.com/reference/project-configs/model-paths.md) configuration in your `dbt_project.yml` file, like so: dbt\_project.yml ```yml model-paths: ["transformations"] ``` --- ### Can I store my seeds in a directory other than the `seeds` directory in my project? By default, dbt expects your seed files to be located in the `seeds` subdirectory of your project. To change this, update the [seed-paths](https://docs.getdbt.com/reference/project-configs/seed-paths.md) configuration in your `dbt_project.yml` file, like so: dbt\_project.yml ```yml seed-paths: ["custom_seeds"] ``` --- ### Can I store my snapshots in a directory other than the `snapshot` directory in my project? By default, dbt expects your snapshot files to be located in the `snapshots` subdirectory of your project. To change this, update the [snapshot-paths](https://docs.getdbt.com/reference/project-configs/snapshot-paths.md) configuration in your `dbt_project.yml` file, like so: dbt\_project.yml ```yml snapshot-paths: ["snapshots"] ``` Note that you cannot co-locate snapshots and models in the same directory. --- ### Can I test the uniqueness of two columns? Yes, there's a few different options for testing the uniqueness of two columns. Consider an orders table that contains records from multiple countries, and the combination of ID and country code is unique: | order\_id | country\_code | | --------- | ------------- | | 1 | AU | | 2 | AU | | ... | ... | | 1 | US | | 2 | US | | ... | ... | Here are some approaches: ###### 1. Create a unique key in the model and test that models/orders.sql ```sql select country_code || '-' || order_id as surrogate_key, ... ``` models/orders.yml ```yml models: - name: orders columns: - name: surrogate_key data_tests: - unique ``` ###### 2. Test an expression models/orders.yml ```yml models: - name: orders data_tests: - unique: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. column_name: "(country_code || '-' || order_id)" ``` ###### 3. Use the `dbt_utils.unique_combination_of_columns` test This is especially useful for large datasets since it is more performant. Check out the docs on [packages](https://docs.getdbt.com/docs/build/packages.md) for more information. models/orders.yml ```yml models: - name: orders data_tests: - dbt_utils.unique_combination_of_columns: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. combination_of_columns: - country_code - order_id ``` --- ### Can I use a YAML file extension? No. At present, dbt will only search for files with a `.yml` file extension. In a future release of dbt, dbt will also search for files with a `.yaml` file extension. --- ### Can I use environment variables in my profile? Yes! Check out the docs on [environment variables](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md) for more information. --- ### Can I use seeds to load raw data? Seeds should **not** be used to load raw data (for example, large CSV exports from a production database). Since seeds are version controlled, they are best suited to files that contain business-specific logic, for example a list of country codes or user IDs of employees. Loading CSVs using dbt's seed functionality is not performant for large files. Consider using a different tool to load these CSVs into your data warehouse. --- ### Can self-hosted GitLab instances only be connected via dbt Enterprise plans? Presently yes, this is only available to Enterprise users. This is because of the way you have to set up the GitLab app redirect URL for auth, which can only be customized if you're a user on an Enterprise plan. Check out our [pricing page](https://www.getdbt.com/pricing/) for more information or feel free to [contact us](https://www.getdbt.com/contact) to build your Enterprise pricing. --- ### Debug Snapshot target is not a snapshot table errors If you see the following error when you try executing the snapshot command: > Snapshot target is not a snapshot table (missing `dbt_scd_id`, `dbt_valid_from`, `dbt_valid_to`) Double check that you haven't inadvertently caused your snapshot to behave like table materializations by setting its `materialized` config to be `table`. Prior to dbt version 1.4, it was possible to have a snapshot like this: ```sql {% snapshot snappy %} {{ config(materialized = 'table', ...) }} ... {% endsnapshot %} ``` dbt is treating snapshots like tables (issuing `create or replace table ...` statements) **silently** instead of actually snapshotting data (SCD2 via `insert` / `merge` statements). When upgrading to dbt versions 1.4 and higher, dbt now raises a Parsing Error (instead of silently treating snapshots like tables) that reads: ```text A snapshot must have a materialized value of 'snapshot' ``` This tells you to change your `materialized` config to `snapshot`. But when you make that change, you might encounter an error message saying that certain fields like `dbt_scd_id` are missing. This error happens because, previously, when dbt treated snapshots as tables, it didn't include the necessary [snapshot meta-fields](https://docs.getdbt.com/docs/build/snapshots.md#snapshot-meta-fields) in your target table. Since those meta-fields don't exist, dbt correctly identifies that you're trying to create a snapshot in a table that isn't actually a snapshot. When this happens, you have to start from scratch — re-snapshotting your source data as if it was the first time by dropping your "snapshot" which isn't a real snapshot table. Then dbt snapshot will create a new snapshot and insert the snapshot meta-fields as expected. --- ### Do hooks run with seeds? Yes! The following hooks are available: * [pre-hooks & post-hooks](https://docs.getdbt.com/reference/resource-configs/pre-hook-post-hook.md) * [on-run-start & on-run-end hooks](https://docs.getdbt.com/reference/project-configs/on-run-start-on-run-end.md) Configure these in your `dbt_project.yml` file. --- ### Do hooks run with snapshots? Yes! The following hooks are available for snapshots: * [pre-hooks](https://docs.getdbt.com/reference/resource-configs/pre-hook-post-hook.md) * [post-hooks](https://docs.getdbt.com/reference/resource-configs/pre-hook-post-hook.md) * [on-run-start](https://docs.getdbt.com/reference/project-configs/on-run-start-on-run-end.md) * [on-run-end](https://docs.getdbt.com/reference/project-configs/on-run-start-on-run-end.md) --- ### Do I need to add a YAML entry for column for it to appear in the docs site? Fortunately, no! dbt will introspect your warehouse to generate a list of columns in each relation, and match it with the list of columns in your `.yml` files. As such, any undocumented columns will still appear in your documentation! --- ### Do I need to create my target schema before running dbt? Nope! dbt will check if the schema exists when it runs. If the schema does not exist, dbt will create it for you. --- ### Do ref-able resource names need to be unique? Within one project: yes! To build dependencies between resources (such as models, seeds, and snapshots), you need to use the `ref` function, and pass in the resource name as an argument. dbt uses that resource name to uniquely resolve the `ref` to a specific resource. As a result, these resource names need to be unique, *even if they are in distinct folders*. A resource in one project can have the same name as a resource in another project (installed as a dependency). dbt uses the project name to uniquely identify each resource. We call this "namespacing." If you `ref` a resource with a duplicated name, it will resolve to the resource within the same namespace (package or project), or raise an error because of an ambiguous reference. Use [two-argument `ref`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models) to disambiguate references by specifying the namespace. Those resource will still need to land in distinct locations in the data warehouse. Read the docs on [custom aliases](https://docs.getdbt.com/docs/build/custom-aliases.md) and [custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md) for details on how to achieve this. --- ### Does dbt offer extract and load functionality? dbt is a transformation tool. It is *not* designed for extract or load functionality, and dbt Labs strongly recommends against using dbt in this way. Support is not provided for extract or load functionality. --- ### Does dbt State support incremental models? dbt State works with incremental models. When you make a change to an incremental model and run it in development, dbt State automatically clones the model from production if it exists, then runs the new model logic on top of the clone. If you want to revert to the original dbt behavior and fully refresh the incremental model, pass the [`--full-refresh` flag](https://docs.getdbt.com/reference/commands/run.md#refresh-incremental-models). --- ### Does dbt State support Python models? dbt State builds Python models but does not reuse them. It executes Python models on every run, even when their code and upstream data have not changed. --- ### Does my `.yml` file containing tests and descriptions need to be named `schema.yml`? No! You can name this file whatever you want (including `whatever_you_want.yml`), so long as: * The file is in your `models/` directory¹ * The file has `.yml` extension Check out the [docs](https://docs.getdbt.com/reference/configs-and-properties.md) for more information. ¹If you're declaring properties for seeds, snapshots, or macros, you can also place this file in the related directory — `seeds/`, `snapshots/` and `macros/` respectively. --- ### Does my operating system have prerequisites? Your operating system may require pre-installation setup before installing dbt v1 with pip. After downloading and installing any dependencies specific to your development environment, you can proceed with the [pip installation of dbt v1](https://docs.getdbt.com/docs/local/install-dbt.md). ##### CentOS CentOS requires Python and some other dependencies to successfully install and run dbt v1. To install Python and other dependencies: ```shell sudo yum install redhat-rpm-config gcc libffi-devel \ python-devel openssl-devel ``` ##### MacOS The MacOS requires Python 3.8 or higher to successfully install and run dbt v1. To check the Python version: ```shell python --version ``` If you need a compatible version, you can download and install [Python version 3.9 or higher for MacOS](https://www.python.org/downloads/macos). If your machine runs on an Apple M1 architecture, we recommend that you install dbt via [Rosetta](https://support.apple.com/en-us/HT211861). This is necessary for certain dependencies that are only supported on Intel processors. ##### Ubuntu/Debian Ubuntu requires Python and other dependencies to successfully install and run dbt v1. To install Python and other dependencies: ```shell sudo apt-get install git libpq-dev python-dev python3-pip sudo apt-get remove python-cffi sudo pip install --upgrade cffi pip install cryptography~=3.4 ``` ##### Windows Windows requires Python and git to successfully install and run dbt v1. Install [Git for Windows](https://git-scm.com/downloads) and [Python version 3.9 or higher for Windows](https://www.python.org/downloads/windows/). For further questions, please see the [Python compatibility FAQ](https://docs.getdbt.com/faqs/Core/install-python-compatibility.md) --- ### Does the Cost Insights feature incur warehouse costs? dbt issues lightweight, read-only queries against your warehouse to retrieve metadata and to power features such as Cost Insights. dbt scopes and filters these queries to minimize impact, and most customers see negligible costs (typically on the order of cents). --- ### Errors importing a repository on dbt project set up If you don't see your repository listed, double-check that: * Your repository is in a Gitlab group you have access to. dbt will not read repos associated with a user. If you do see your repository listed, but are unable to import the repository successfully, double-check that: * You are a maintainer of that repository. Only users with maintainer permissions can set up repository connections. If you imported a repository using the dbt native integration with GitLab, you should be able to see if the clone strategy is using a `deploy_token`. If it's relying on an SSH key, this means the repository was not set up using the native GitLab integration, but rather using the generic git clone option. The repository must be reconnected in order to get the benefits described above. --- ### GitLab token refresh message When you connect dbt to a GitLab repository, GitLab automatically creates a [project access token](https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html) in your GitLab repository in the background. This sends the job run status back to Gitlab using the dbt API for CI jobs. By default, the project access token follows a naming pattern: `dbt token for GitLab project: <project_id>`. If you have multiple tokens in your repository, look for one that follows this pattern to identify the correct token used by dbt. If you're receiving a "Refresh token" message, don't worry — dbt automatically refreshes this project access token for you, which means you never have to manually rotate it. If you still experience any token refresh errors, please try disconnecting and reconnecting the repository in your dbt project to refresh the token. For any issues, please reach out to the Support team at <support@getdbt.com> and we'll be happy to help! --- ### How can I connect dbt to a Google Source repository? Although we don't officially support Google Cloud as a git repository, the below workaround using the SSH URL method should help you to connect: * First: "import" your Repository into dbt using the SSH URL provided to you by GCP. That will look something like: `ssh://drew@fishtownanalytics.com@source.developers.google.com:2022/p/dbt-integration-tests/r/drew-debug` * After importing the repo, you should see a public key generated by dbt for the repository. You'll want to copy that public key into a new SSH Key for your user here: <https://source.cloud.google.com/user/ssh_keys> * After saving this SSH key, dbt should be able to read and write to this repo. If you've tried the workaround above and are still having issues connecting - reach out to the Support team at <support@getdbt.com> and we'll be happy to help! --- ### How can I consolidate projects in dbt? Consolidating your dbt projects can be an enormous task, and there is no universal solution. But, there are some common approaches to project consolidation in dbt that you can follow, depending on the scope of the work that needs to be done. If you have multiple projects that contain production-worthy code, there are rarely straightforward solutions to merging them. Let's suppose you have `Main Project` and `Smaller Subset Project`. #### Files and Folders ##### Git and the local directory Reference the [merge git commands](https://gist.github.com/msrose/2feacb303035d11d2d05) to help complete the migration plan. Using the commands will help retain git commit history, but you might result in duplicate folders called `models`, `tests`, etc. You will most likely still have to move files around manually. Another option would be to use an external code editor (for example, VS Code) to move files from the `Smaller Subset Project` to the `Main Project`. This is what internal dbt Labs experts recommend to stay informed about what comes over to the main project and also allows you to be more aware of the incoming files, with the ability to make any minor tweaks to folder hierarchy that you might want to do at the same time. ##### Manual migration with multiple browser tabs If you only have a couple of models or macros that you want to consolidate, copy the raw file contents from your git provider in `Smaller Subset Project`. Then, in the Studio IDE, paste the contents into a new file in your `Main Project`. Alternatively, you can download those files from your git provider (`Smaller Subset Project` repo) and upload them back to your other repository (`Main Project` repo). This doesn’t scale well and could bypass change controls, so it might only be a viable solution for organizations with only a few files. #### Production jobs If you have multiple projects with deployment environments deploying jobs, this poses another challenge. Assuming all the models from `Smaller Subset Project` can be consolidated into `Main Project`, your commands within your jobs will take on a new meaning. In lieu of refactoring your global job strategy at the same time, you can add tags to the incoming project models and utilize that in your job command syntax, with the help of node selection syntax. Main Project job command example: `dbt build --exclude tag:smaller_subset_project` Smaller Subset Project commands: `dbt build --select tag:smaller_subset_project` --- ### How can I fix my .gitignore file? A `.gitignore` file specifies which files git should intentionally ignore or 'untrack'. dbt indicates untracked files in the project file explorer pane by putting the file or folder name in *italics*. If you encounter issues like problems reverting changes, checking out or creating a new branch, or not being prompted to open a pull request after a commit in the Studio IDE — this usually indicates a problem with the [.gitignore](https://github.com/dbt-labs/dbt-starter-project/blob/main/.gitignore) file. The file may be missing or lacks the required entries for dbt to work correctly. The following sections describe how to fix the `.gitignore` file in:  Fix in the Studio IDE To resolve issues with your `gitignore` file, adding the correct entries won't automatically remove (or 'untrack') files or folders that have already been tracked by git. The updated `gitignore` will only prevent new files or folders from being tracked. So you'll need to first fix the `gitignore` file, then perform some additional git operations to untrack any incorrect files or folders. 1. Launch the Studio IDE into the project that is being fixed, by selecting **Develop** on the menu bar. 2. In your **File Explorer**, check to see if a `.gitignore` file exists at the root of your dbt project folder. If it doesn't exist, create a new file. 3. Open the new or existing `gitignore` file, and add the following: ```bash # ✅ Correct target/ dbt_packages/ logs/ # legacy -- renamed to dbt_packages in dbt v1 dbt_modules/ ``` * **Note** — You can place these lines anywhere in the file, as long as they're on separate lines. The lines shown are wildcards that will include all nested files and folders. Avoid adding a trailing `'*'` to the lines, such as `target/*`. For more info on `gitignore` syntax, refer to the [Git docs](https://git-scm.com/docs/gitignore). 4. Save the changes but *don't commit*. 5. Restart the IDE by clicking on the three dots next to the **IDE Status button** on the lower right corner of the IDE screen and select **Restart IDE**. [![Restart the IDE by clicking the three dots on the lower right or click on the Status bar](/img/docs/dbt-platform/platform-ide/restart-ide.png?v=2 "Restart the IDE by clicking the three dots on the lower right or click on the Status bar")](#)Restart the IDE by clicking the three dots on the lower right or click on the Status bar 6. Once the Studio IDE restarts, go to the **File Catalog** to delete the following files or folders (if they exist). No data will be lost: * `target`, `dbt_modules`, `dbt_packages`, `logs` 7. **Save** and then **Commit and sync** the changes. 8. Restart the Studio IDE again using the same procedure as step 5. 9. Once the Studio IDE restarts, use the **Create a pull request** (PR) button under the **Version Control** menu to start the process of integrating the changes. 10. When the git provider's website opens to a page with the new PR, follow the necessary steps to complete and merge the PR into the main branch of that repository. * **Note** — The 'main' branch might also be called 'master', 'dev', 'qa', 'prod', or something else depending on the organizational naming conventions. The goal is to merge these changes into the root branch that all other development branches are created from. 11. Return to the Studio IDE and use the **Change Branch** button, to switch to the main branch of the project. 12. Once the branch has changed, click the **Pull from remote** button to pull in all the changes. 13. Verify the changes by making sure the files/folders in the `.gitignore` file are in italics. [![A dbt project on the main branch that has properly configured gitignore folders (highlighted in italics).](/img/docs/dbt-platform/platform-ide/gitignore-italics.png?v=2 "A dbt project on the main branch that has properly configured gitignore folders (highlighted in italics).")](#)A dbt project on the main branch that has properly configured gitignore folders (highlighted in italics).  Fix in the Git provider Sometimes it's necessary to use the git providers web interface to fix a broken `.gitignore` file. Although the specific steps may vary across providers, the general process remains the same. There are two options for this approach: editing the main branch directly if allowed, or creating a pull request to implement the changes if required: ##### Edit in main branch When permissions allow it, it's possible to edit the `.gitignore` directly on the main branch of your repo. Here are the following steps: 1. Go to your repository's web interface. 2. Switch to the main branch and the root directory of your dbt project. 3. Find the `.gitignore` file. Create a blank one if it doesn't exist. 4. Edit the file in the web interface, adding the following entries: ```bash target/ dbt_packages/ logs/ # legacy -- renamed to dbt_packages in dbt v1 dbt_modules/ ``` 5. Commit (save) the file. 6. Delete the following folders from the dbt project root, if they exist. No data or code will be lost: * `target`, `dbt_modules`, `dbt_packages`, `logs` 7. Commit (save) the deletions to the main branch. 8. Switch to the Studio IDE , and open the project that you're fixing. 9. [Rollback your repo to remote](https://docs.getdbt.com/docs/platform/git/version-control-basics.md#the-git-button-in-the-cloud-ide) in the IDE by clicking on the three dots next to the **IDE Status** button on the lower right corner of the IDE screen, then select **Rollback to remote**. * **Note** — Rollback to remote resets your repo back to an earlier clone from your remote. Any saved but uncommitted changes will be lost, so make sure you copy any modified code that you want to keep in a temporary location outside of dbt. 10. Once you rollback to remote, open the `.gitignore` file in the branch you're working in. If the new changes aren't included, you'll need to merge the latest commits from the main branch into your working branch. 11. Go to the **File Explorer** to verify the `.gitignore` file contains the correct entries and make sure the untracked files/folders in the .gitignore file are in *italics*. 12. Great job 🎉! You've configured the `.gitignore` correctly and can continue with your development! ##### Unable to edit main branch If you can't edit the `.gitignore` directly on the main branch of your repo, follow these steps: 1. Go to your repository's web interface. 2. Switch to an existing development branch, or create a new branch just for these changes (This is often faster and cleaner). 3. Find the `.gitignore` file. Create a blank one if it doesn't exist. 4. Edit the file in the web interface, adding the following entries: ```bash target/ dbt_packages/ logs/ # legacy -- renamed to dbt_packages in dbt v1 dbt_modules/ ``` 5. Commit (save) the file. 6. Delete the following folders from the dbt project root, if they exist. No data or code will be lost: * `target`, `dbt_modules`, `dbt_packages`, `logs` 7. Commit (save) the deleted folders. 8. Open a merge request using the git provider web interface. The merge request should attempt to merge the changes into the 'main' branch that all development branches are created from. 9. Follow the necessary procedures to get the branch approved and merged into the 'main' branch. You can delete the branch after the merge is complete. 10. Once the merge is complete, go back to the Studio IDE, and open the project that you're fixing. 11. [Rollback your repo to remote](https://docs.getdbt.com/docs/platform/git/version-control-basics.md#the-git-button-in-the-cloud-ide) in the Studio IDE by clicking on the three dots next to the **Studio IDE Status** button on the lower right corner of the Studio IDE screen, then select **Rollback to remote**. * **Note** — Rollback to remote resets your repo back to an earlier clone from your remote. Any saved but uncommitted changes will be lost, so make sure you copy any modified code that you want to keep in a temporary location outside of dbt. 12. Once you rollback to remote, open the `.gitignore` file in the branch you're working in. If the new changes aren't included, you'll need to merge the latest commits from the main branch into your working branch. 13. Go to the **File Explorer** to verify the `.gitignore` file contains the correct entries and make sure the untracked files/folders in the .gitignore file are in *italics*. 14. Great job 🎉! You've configured the `.gitignore` correctly and can continue with your development! For more info, refer to this [detailed video](https://www.loom.com/share/9b3b8e2b617f41a8bad76ec7e42dd014) for additional guidance. --- ### How can I fix my .gitignore file? A gitignore file specifies which files Git should intentionally ignore. You can identify these files in your project by their italics formatting. If you can't revert changes, check out a branch, or click commit — this is usually do to your project missing a [.gitignore](https://github.com/dbt-labs/dbt-starter-project/blob/main/.gitignore) file OR your gitignore file doesn't contain the necessary content inside the folder. To fix this, complete the following steps: 1. In the Studio IDE, add the following [.gitignore contents](https://github.com/dbt-labs/dbt-starter-project/blob/main/.gitignore) in your dbt project `.gitignore` file: ```bash target/ dbt_packages/ logs/ # legacy -- renamed to dbt_packages in dbt v1 dbt_modules/ ``` 2. Save your changes but *don't commit* 3. Restart the Studio IDE by clicking on the three dots next to the **Studio IDE Status button** on the lower right of the Studio IDE. [![Restart the IDE by clicking the three dots on the lower right or click on the Status bar](/img/docs/dbt-platform/platform-ide/restart-ide.png?v=2 "Restart the IDE by clicking the three dots on the lower right or click on the Status bar")](#)Restart the IDE by clicking the three dots on the lower right or click on the Status bar 4. Select **Restart Studio IDE**. 5. Go back to the **File explorer** in the IDE and delete the following files or folders if you have them: * `target`, `dbt_modules`, `dbt_packages`, `logs` 6. **Save** and then **Commit and sync** your changes. 7. Restart the Studio IDE again. 8. Create a pull request (PR) under the **Version Control** menu to integrate your new changes. 9. Merge the PR on your git provider page. 10. Switch to your main branch and click on **Pull from remote** to pull in all the changes you made to your main branch. You can verify the changes by making sure the files/folders in the .gitignore file are in italics. [![A dbt project on the main branch that has properly configured gitignore folders (highlighted in italics).](/img/docs/dbt-platform/platform-ide/gitignore-italics.png?v=2 "A dbt project on the main branch that has properly configured gitignore folders (highlighted in italics).")](#)A dbt project on the main branch that has properly configured gitignore folders (highlighted in italics). For more info, refer to this [detailed video](https://www.loom.com/share/9b3b8e2b617f41a8bad76ec7e42dd014) for additional guidance. --- ### How can I see the SQL that dbt is running? To check out the SQL that dbt is running, you can look in: * dbt: * Within the run output, click on a model name, and then select "Details" * dbt v1: * The `target/compiled/` directory for compiled `select` statements * The `target/run/` directory for compiled `create` statements * The `logs/dbt.log` file for verbose logging. --- ### How can I set up the right permissions in BigQuery? To use this functionality, first create the service account you want to impersonate. Then grant users that you want to be able to impersonate this service account the `roles/iam.serviceAccountTokenCreator` role on the service account resource. Then, you also need to grant the service account the same role on itself. This allows it to create short-lived tokens identifying itself, and allows your human users (or other service accounts) to do the same. More information on this scenario is available [here](https://cloud.google.com/iam/docs/understanding-service-accounts#directly_impersonating_a_service_account). Once you've granted the appropriate permissions, you'll need to enable the [IAM Service Account Credentials API](https://console.cloud.google.com/apis/library/iamcredentials.googleapis.com). Enabling the API and granting the role are eventually consistent operations, taking up to 7 minutes to fully complete, but usually fully propagating within 60 seconds. Give it a few minutes, then add the `impersonate_service_account` option to your BigQuery profile configuration. --- ### How can I update my billing information? If you want to change your account's credit card details, go to the left side panel, click **Account settings** → **Billing** → scroll to **Payment information**. Enter the new credit card details on the respective fields then click on **Update payment information**. Only the *account owner* can make this change. To change your billing name or location address, send our Support team a message at <support@getdbt.com> with the newly updated information, and we can make that change for you! --- ### How can we move our project from a managed repository, to a self-hosted repository? dbt Labs can send your managed repository through a ZIP file in its current state for you to push up to a git provider. After that, you'd just need to switch over to the [repo in your project](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md) to point to the new repository. When you're ready to do this, [contact the dbt Labs Support team](mailto:support@getdbt.com) with your request and your managed repo URL, which you can find by navigating to your project setting. To find project settings: 1. From dbt, click on your account name in the left side menu and select **Account settings**. 2. Click **Projects**, and then select your project. 3. Under **Repository** in the project details page, you can find your managed repo URL. --- ### How did dbt choose which schema to build my models in? By default, dbt builds models in your target schema. To change your target schema: * If you're developing in **dbt**, these are set for each user when you first use a development environment. * If you're developing with **dbt v1**, this is the `schema:` parameter in your `profiles.yml` file. If you wish to split your models across multiple schemas, check out the docs on [using custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md). Note: on BigQuery, `dataset` is used interchangeably with `schema`. --- ### How do I access documentation in dbt Catalog? If you're using dbt to deploy your project and have a [Starter, Enterprise, or Enterprise+ plan](https://www.getdbt.com/pricing/), you can use Catalog to view your project's [resources](https://docs.getdbt.com/docs/build/projects.md) (such as models, tests, and metrics) and their lineage to gain a better understanding of its latest production state. Access Catalog in dbt by clicking the **Catalog** link in the navigation. Read-only users can access the documentation for your project — for license availability by plan, refer to [dbt pricing](https://www.getdbt.com/pricing). dbt developer plan and dbt v1 users can use [dbt Docs](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md#dbt-docs), which generates basic documentation but it doesn't offer the same speed, metadata, or visibility as Catalog. --- ### How do I build one seed at a time? You can use a `--select` option with the `dbt seed` command, like so: ```shell $ dbt seed --select country_codes ``` There is also an `--exclude` option. Check out more in the [model selection syntax](https://docs.getdbt.com/reference/node-selection/syntax.md) documentation. --- ### How do I change a user license type to read-only in dbt? To change the license type for a user from `developer` to `read-only` or `IT` in dbt, you must be an account owner or have admin privileges. You might make this change to free up a billable seat but retain the user’s access to view the information in the dbt account. 1. From dbt, click on your account name in the left side menu and, select **Account settings**. [![Navigate to account settings](/img/docs/dbt-platform/Navigate-to-account-settings.png?v=2 "Navigate to account settings")](#)Navigate to account settings 2. In **Account Settings**, select **Users** under **Teams**. 3. Select the user you want to remove and click **Edit** in the bottom of their profile. 4. For the **License** option, choose **Read-only** or **IT** (from **Developer**), and click **Save**. [![Change user's license type](/img/docs/dbt-platform/change_user_to_read_only_20221023.gif?v=2 "Change user's license type")](#)Change user's license type License types override group permissions **User license types always override their assigned group permission sets.** For example, a user with a Read-Only license cannot perform administrative actions, even if they belong to an Account Admin group. This ensures that license restrictions are always enforced, regardless of group membership. --- ### How do I create dependencies between models? When you use the `ref` [function](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md), dbt automatically infers the dependencies between models. For example, consider a model, `customer_orders`, like so: models/customer\_orders.sql ```sql select customer_id, min(order_date) as first_order_date, max(order_date) as most_recent_order_date, count(order_id) as number_of_orders from {{ ref('stg_orders') }} group by 1 ``` **There's no need to explicitly define these dependencies.** dbt will understand that the `stg_orders` model needs to be built before the above model (`customer_orders`). When you execute `dbt run`, you will see these being built in order: ```txt $ dbt run Running with dbt=1.9.0 Found 2 models, 28 data tests, 0 snapshots, 0 analyses, 130 macros, 0 operations, 0 seed files, 3 sources 11:42:52 | Concurrency: 8 threads (target='dev_snowflake') 11:42:52 | 11:42:52 | 1 of 2 START sql view model dbt_claire.stg_jaffle_shop__orders ....... [RUN] 11:42:55 | 1 of 2 OK created sql view model dbt_claire.stg_jaffle_shop__orders .. [CREATE VIEW in 2.50s] 11:42:55 | 2 of 2 START sql view model dbt_claire.customer_orders .............. [RUN] 11:42:56 | 2 of 2 OK created sql view model dbt_claire.customer_orders ......... [CREATE VIEW in 0.60s] 11:42:56 | Finished running 2 view models in 15.13s. Done. PASS=2 WARN=0 ERROR=0 SKIP=0 TOTAL=2 ``` To learn more about building a dbt project, we recommend you complete the [quickstart guide](https://docs.getdbt.com/guides.md). --- ### How do I debug my Jinja? You should get familiar with checking the compiled SQL in `target/compiled/<your_project>/` and the logs in `logs/dbt.log` to see what dbt is running behind the scenes. You can also use the [log](https://docs.getdbt.com/reference/dbt-jinja-functions/log.md) function to debug Jinja by printing objects to the command line. --- ### How do I define a column type? Your warehouse's SQL engine automatically assigns a [datatype](https://www.w3schools.com/sql/sql_datatypes.asp) to every column, whether it's found in a source or model. To force SQL to treat a columns a certain datatype, use `cast` functions: models/order\_prices.sql ```sql select cast(order_id as integer), cast(order_price as double(6,2)) -- a more generic way of doing type conversion from {{ ref('stg_orders') }} ``` Many modern data warehouses now support `::` syntax as a shorthand for `cast( as )`. models/orders\_prices\_colon\_syntax.sql ```sql select order_id::integer, order_price::numeric(6,2) -- you might find this in Redshift, Snowflake, and Postgres from {{ ref('stg_orders') }} ``` Be warned, reading in data and casting that data may not always yield expected results, and every warehouse has its own subtleties. Certain casts may not be allowed (e.g. on Bigquery, you can't cast a `boolean`-type value to a `float64`). Casts that involve a loss in precision loss (e.g. `float` to `integer`) rely on your SQL engine to make a best guess or follow a specific schema not used by competing services. When performing casts, it's imperative that you are familiar with your warehouse's casting rules to best label fields in your sources and models. Thankfully, popular database services tend to have type docs--[Redshift](https://docs.amazonaws.cn/en_us/redshift/latest/dg/r_CAST_function.html) and [Bigquery](https://cloud.google.com/bigquery/docs/reference/standard-sql/conversion_rules). --- ### How do I delete a project in dbt? To delete a project in dbt, you must be the account owner or have admin privileges. 1. From dbt, click on your account name in the left side menu and select **Account settings**. [![Navigate to account settings](/img/docs/dbt-platform/Navigate-to-account-settings.png?v=2 "Navigate to account settings")](#)Navigate to account settings 2. In **Account Settings**, select **Projects**. Click the project you want to delete from the **Projects** page. 3. Click the edit icon in the lower right-hand corner of the **Project Details**. A **Delete** option appears on the left side of the same details view. 4. Click **Delete**. Confirm the action to immediately delete the user without additional password prompts. The project is deleted immediately after confirmation. Once a project is deleted, this action cannot be undone. [![Delete projects](/img/docs/dbt-platform/delete_projects_from_dbt_cloud.png?v=2 "Delete projects")](#)Delete projects --- ### How do I delete a user in dbt? To delete a user in dbt, you must be an account owner or have admin privileges. If the user has a `developer` license type, this will open up their seat for another user or allow the admins to lower the total number of seats. 1. From dbt, click on your account name in the left side menu and, select **Account settings**. [![Navigate to account settings](/img/docs/dbt-platform/Navigate-to-account-settings.png?v=2 "Navigate to account settings")](#)Navigate to account settings 2. In **Account settings**, select **Users** under **Teams**. 3. Select the user you want to delete, then click **Edit**. 4. Click **Delete** in the bottom left. Click **Confirm Delete** to immediately delete the user without additional password prompts. This action cannot be undone. However, you can re-invite the user with the same information if the deletion was made in error. [![Deleting a user](/img/docs/dbt-platform/delete_user.png?v=2 "Deleting a user")](#)Deleting a user If you are on a **Starter** plan and you're deleting users to reduce the number of billable seats, follow these steps to lower the license count to avoid being overcharged: 1. In **Account Settings**, select **Billing**. 2. Under **Billing details**, enter the number of developer seats you want and make sure you fill in all the payment details, including the **Billing address** section. If you leave any field blank, you won't be able to save your changes. 3. Click **Update Payment Information** to save your changes. [![Navigate to Account settings -> Users to modify dbt users](/img/docs/dbt-platform/faq-account-settings-billing.png?v=2 "Navigate to Account settings -> Users to modify dbt users")](#)Navigate to Account settings -> Users to modify dbt users #### Related docs * [dbt licenses](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md#licenses) --- ### How do I document macros? To document macros, use a [properties file](https://docs.getdbt.com/reference/macro-properties.md) and nest the configurations under a `macros:` key #### Example macros/properties.yml ```yml macros: - name: cents_to_dollars description: A macro to convert cents to dollars arguments: - name: column_name type: column description: The name of the column you want to convert - name: precision type: integer description: Number of decimal places. Defaults to 2. ``` tip From dbt v1.10, you can opt into validating the arguments you define in macro documentation using the `validate_macro_args` behavior change flag. When enabled, dbt will: * Infer arguments from the macro and includes them in the [manifest.json](https://docs.getdbt.com/reference/artifacts/manifest-json.md) file if no arguments are documented. * Raise a warning if documented argument names don't match the macro definition. * Raise a warning if `type` fields don't follow [supported formats](https://docs.getdbt.com/reference/resource-properties/arguments.md#supported-types). Learn more about [macro argument validation](https://docs.getdbt.com/reference/global-configs/behavior-flags/validate_macro_args.md). #### Document a custom materialization When you create a [custom materialization](https://docs.getdbt.com/guides/create-new-materializations.md), dbt creates an associated macro with the following format: ```text materialization_{materialization_name}_{adapter} ``` To document a custom materialization, use the previously mentioned format to determine the associated macro name(s) to document. macros/properties.yml ```yaml macros: - name: materialization_my_materialization_name_default description: A custom materialization to insert records into an append-only table and track when they were added. - name: materialization_my_materialization_name_xyz description: A custom materialization to insert records into an append-only table and track when they were added. ``` --- ### How do I exclude a table from a freshness snapshot? Some tables in a data source may be updated infrequently. If you've set a `freshness` property at the source level, this table is likely to fail checks. To work around this, you can set the table's freshness to null (`freshness: null`) to "unset" the freshness for a particular table: models/\<filename>.yml ```yaml sources: - name: jaffle_shop database: raw schema: jaffle_shop config: freshness: warn_after: {count: 12, period: hour} error_after: {count: 24, period: hour} loaded_at_field: _etl_loaded_at tables: - name: orders - name: product_skus config: freshness: null # do not check freshness for this table ``` --- ### How do I load data into my warehouse? dbt assumes that you already have a copy of your data, in your data warehouse. We recommend you use an off-the-shelf tool like [Stitch](https://www.stitchdata.com/) or [Fivetran](https://fivetran.com/) to get data into your warehouse. **Can dbt be used to load data?** No, dbt does not extract or load data. It focuses on the transformation step only. --- ### How do I populate the owner column in the generated docs? You cannot change the `owner` column in your generated documentation. dbt pulls the `owner` field in `dbt-docs` from database metadata ([catalog.json](https://docs.getdbt.com/reference/artifacts/catalog-json.md)), meaning the `owner` of that table in the database. With the exception of [exposures](https://docs.getdbt.com/docs/build/exposures.md), dbt does not pull this value from an `owner` field set within dbt. Generally, dbt's database user owns the tables created in the database. The service responsible for ingesting or loading the data usually owns the source tables. If you set `meta.owner`, that field appears under **meta** (pulled from dbt), but still not under the top-level `owner` field. #### Example The following example shows a model with `meta.owner` so it appears under **meta** in the docs. Replace `DATA_TEAM_EMAIL` with your own values. models/stg\_orders.yml ```yaml models: - name: stg_orders description: "Staging table for order events." config: meta: owner: "DATA_TEAM_EMAIL" columns: - name: order_id description: "Primary key for orders." - name: order_date description: "Date when order was placed." ``` --- ### How do I preserve leading zeros in a seed? If you need to preserve leading zeros (for example in a zipcode or mobile number), include leading zeros in your seed file, and use the `column_types` [configuration](https://docs.getdbt.com/reference/resource-configs/column_types.md) with a varchar datatype of the correct length. --- ### How do I remove deleted models from my data warehouse? If you delete a model from your dbt project, dbt does not automatically drop the relation from your schema. This means that you can end up with extra objects in schemas that dbt creates, which can be confusing to other users. (This can also happen when you switch a model from being a view or table, to ephemeral) When you remove models from your dbt project, you should manually drop the related relations from your schema. --- ### How do I run data tests on just my sources? To run data tests on all sources, use the following command: ```shell dbt test --select "source:*" ``` (You can also use the `-s` shorthand here instead of `--select`) To run data tests on one source (and all of its tables): ```shell $ dbt test --select source:jaffle_shop ``` And, to run data tests on one source table only: ```shell $ dbt test --select source:jaffle_shop.orders ``` --- ### How do I run models downstream of a seed? You can run models downstream of a seed using the [model selection syntax](https://docs.getdbt.com/reference/node-selection/syntax.md), and treating the seed like a model. For example, the following would run all models downstream of a seed named `country_codes`: ```shell $ dbt run --select country_codes+ ``` --- ### How do I run models downstream of one source? To run models downstream of a source, use the `source:` selector: ```shell $ dbt run --select source:jaffle_shop+ ``` (You can also use the `-s` shorthand here instead of `--select`) To run models downstream of one source table: ```shell $ dbt run --select source:jaffle_shop.orders+ ``` Check out the [model selection syntax](https://docs.getdbt.com/reference/node-selection/syntax.md) for more examples! --- ### How do I run one model at a time? To run one model, use the `--select` flag (or `-s` flag), followed by the name of the model: ```shell $ dbt run --select customers ``` Check out the [model selection syntax documentation](https://docs.getdbt.com/reference/node-selection/syntax.md) for more operators and examples. --- ### How do I run one snapshot at a time? To run one snapshot, use the `--select` flag, followed by the name of the snapshot: ```shell $ dbt snapshot --select order_snapshot ``` Check out the [model selection syntax documentation](https://docs.getdbt.com/reference/node-selection/syntax.md) for more operators and examples. --- ### How do I set a datatype for a column in my seed? dbt will infer the datatype for each column based on the data in your CSV. You can also explicitly set a datatype using the `column_types` [configuration](https://docs.getdbt.com/reference/resource-configs/column_types.md) like so: dbt\_project.yml ```yml seeds: jaffle_shop: # you must include the project name warehouse_locations: +column_types: zipcode: varchar(5) ``` --- ### How do I snapshot freshness for one source only? Use the `--select` flag to snapshot freshness for specific sources. Eg: ```shell # Snapshot freshness for all Jaffle Shop tables: $ dbt source freshness --select source:jaffle_shop # Snapshot freshness for a particular source <Term id="table" />: $ dbt source freshness --select source:jaffle_shop.orders # Snapshot freshness for multiple particular source tables: $ dbt source freshness --select source:jaffle_shop.orders source:jaffle_shop.customers ``` See the [`source freshness` command reference](https://docs.getdbt.com/reference/commands/source.md) for more information. --- ### How do I specify column types? Simply cast the column to the correct type in your model: ```sql select id, created::timestamp as created from some_other_table ``` You might have this question if you're used to running statements like this: ```sql create table dbt_alice.my_table id integer, created timestamp; insert into dbt_alice.my_table ( select id, created from some_other_table ) ``` In comparison, dbt would build this table using a `create table as` statement: ```sql create table dbt_alice.my_table as ( select id, created from some_other_table ) ``` So long as your model queries return the correct column type, the table you create will also have the correct column type. To define additional column options: * Rather than enforcing uniqueness and not-null constraints on your column, use dbt's [data testing](https://docs.getdbt.com/docs/build/data-tests.md) functionality to check that your assertions about your model hold true. * Rather than creating default values for a column, use SQL to express defaults (e.g. `coalesce(updated_at, current_timestamp()) as updated_at`) * In edge-cases where you *do* need to alter a column (e.g. column-level encoding on Redshift), consider implementing this via a [post-hook](https://docs.getdbt.com/reference/resource-configs/pre-hook-post-hook.md). --- ### How do I test and document seeds? To test and document seeds, use a [properties file](https://docs.getdbt.com/reference/configs-and-properties.md) and nest the configurations under a `seeds:` key #### Example seeds/properties.yml ```yml seeds: - name: country_codes description: A mapping of two letter country codes to country names columns: - name: country_code data_tests: - unique - not_null - name: country_name data_tests: - unique - not_null ``` --- ### How do I test one model at a time? Running tests on one model looks very similar to running a model: use the `--select` flag (or `-s` flag), followed by the name of the model: ```shell dbt test --select customers ``` Check out the [model selection syntax documentation](https://docs.getdbt.com/reference/node-selection/syntax.md) for full syntax, and [test selection examples](https://docs.getdbt.com/reference/node-selection/test-selection-examples.md) in particular. --- ### How do I transfer account ownership to another user? You can transfer your dbt [access control](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md) to another user by following the steps below, depending on your dbt account plan: | Account plan | Steps | | ------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Developer** | You can transfer ownership by changing the email directly on your dbt profile page, which you can access using this URL when you replace `YOUR_ACCESS_URL` with the [appropriate Access URL](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for your region and plan: `https://YOUR_ACCESS_URL/settings/profile`. Before doing this, please ensure that you unlink your GitHub profile. The email address of the new account owner cannot be associated with another dbt account. | | **Starter** | Existing account admins with account access can add users to, or remove users from the owner group. | | **Enterprise or Enterprise+** | Account admins can add users to, or remove users from a group with Account Admin permissions. | | **If all account owners left the company** | If the account owner has left your organization, you will need to work with *your* IT department to have incoming emails forwarded to the new account owner. Once your IT department has redirected the emails, you can request to reset the user password. Once you log in, you can change the email on the Profile page when you replace `YOUR_ACCESS_URL` with the [appropriate Access URL](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) for your region and plan: `https://YOUR_ACCESS_URL/settings/profile`. | When you make any account owner and email changes: * The new email address *must* be verified through our email verification process. * You can update any billing email address or [Notifications Settings](https://docs.getdbt.com/docs/deploy/job-notifications.md) to reflect the new account owner changes, if applicable. * When transferring account ownership, please ensure you [unlink](https://docs.getdbt.com/faqs/Accounts/git-account-in-use.md) your GitHub account in dbt. This is because you can only have your Git account linked to one dbt user account. --- ### How do I troubleshoot if cost data isn't appearing? If cost data isn't appearing in Cost Insights, check the following: * Verify that platform metadata credentials are configured in your account settings and that the credential test is passing. For more information, see [Set up Cost Insights](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md#configure-platform-metadata-credentials). * Ensure you have one of the required permissions to view cost data. For more information, see [Assign required permissions](https://docs.getdbt.com/docs/explore/set-up-cost-insights.md#assign-required-permissions). * Confirm that at least one job is running in a production environment. Cost data only appears after jobs have executed. * Cost data refreshes daily and reflects the previous day's usage, which means there is a lag of up to one day between when a job runs and when its cost data appears. If you just ran a job, wait until the next day to see the data. * After enabling Cost Insights, dbt looks back 10 days to build baselines for cost reduction calculations. If you don't see cost reduction data, ensure you have sufficient job history within the last 10 days. --- ### How do I use the 'Custom Branch' settings in a dbt Environment? In dbt environments, you can change your git settings to use a different branch in your dbt project repositories besides the default branch. When you make this change, you run dbt on a custom branch. When specified, dbt executes models using the custom branch setting for that environment. Development and deployment environments have slightly different effects. To specify a custom branch: 1. Edit an existing environment or create a new one 2. Select **Only run on a custom branch** under General Settings 3. Specify the **branch name or tag** #### Development In a development environment, the primary branch (usually named `main`) is protected in your connected repositories. You can directly edit, format, or lint files and execute dbt commands in your protected default git branch. Since the Studio IDE prevents commits to the protected branch, you can commit those changes to a new branch when you're ready. Specifying a **Custom branch** overrides the default behavior. It makes the custom branch protected and enables you to create new development branches from it. You can directly edit, format, or lint files and execute dbt commands in your custom branch, but you cannot make commits to it. dbt prompts you to commit those changes to a new branch. Only one branch can be protected. If you specify a custom branch, the primary branch is no longer protected. If you want to protect the primary branch and prevent any commits on it, you need to set up branch protection rules in your git provider settings. This ensures your primary branch remains secure and no new commits can be made to it. For example, if you want to use the `develop` branch of a connected repository: 1. Go to an environment and click **Settings** > **Edit** to edit the environment. 2. Select **Only run on a custom branch** in **General settings**. 3. Enter **develop** as the name of your custom branch. 4. Click **Save**. [![Configuring a custom base repository branch](/img/docs/dbt-platform/platform-configuring-dbt-platform/dev-environment-custom-branch.png?v=2 "Configuring a custom base repository branch")](#)Configuring a custom base repository branch #### Deployment When running jobs in a deployment environment, dbt will clone your project from your connected repository before executing your models. By default, dbt uses the default branch of your repository (commonly the `main` branch). To specify a different version of your project for dbt to execute during job runs in a particular environment, you can edit the Custom Branch setting as shown in the previous steps. --- ### How do I write long-form explanations in my descriptions? If you need more than a sentence to explain a model, you can: 1. Split your description over multiple lines using `>`. Interior line breaks are removed and Markdown can be used. This method is recommended for simple, single-paragraph descriptions: ```yml models: - name: customers description: > Lorem ipsum **dolor** sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. ``` 2. Split your description over multiple lines using `|`. Interior line breaks are maintained and Markdown can be used. This method is recommended for more complex descriptions: ```yml models: - name: customers description: | ### Lorem ipsum * dolor sit amet, consectetur adipisicing elit, sed do eiusmod * tempor incididunt ut labore et dolore magna aliqua. ``` 3. Use a [docs block](https://docs.getdbt.com/docs/build/documentation.md#using-docs-blocks) to write the description in a separate Markdown file. --- ### How does dbt State calculate that a model has changed? dbt State only considers substantial changes to a model. Because dbt State understands the entire lineage of your models, it can see through things like whitespace and aliases to determine whether a model is the same or different across environments. By default, dbt State compares rendered SQL to detect code changes. Any change to the rendered SQL — including from non-deterministic macros or environment variables — triggers a rebuild. You can enable [`compare_unrendered_code`](https://docs.getdbt.com/reference/resource-configs/compare-unrendered-code.md) to also check the Jinja template (unrendered code). When enabled, a rebuild only occurs when both the template *and* the rendered SQL have changed. Non-deterministic values (for example, `{{ env_var('AIRFLOW_RUN_ID') }}` or a macro that calls `uuid()`) don't trigger rebuilds as long as the template itself is unchanged, which helps avoid unnecessary warehouse compute costs. --- ### How does increasing job frequency affect cost reduction estimates? Cost reduction metrics reflect how dbt optimizes compute costs by reusing existing results instead of running the same model again. When you increase your job run frequency (for example, because performance improvements make it easier to schedule jobs more often), dbt has more opportunities to reuse models. As reuse increases, dbt optimizes more compute, which means your reported cost reductions may also increase. This metric shows the efficiency impact of reuse within your current workload. It reflects the compute costs that dbt reduces by reusing models instead of rebuilding them, rather than showing your total warehouse spend reduction. --- ### How is data stored in dbt State? dbt State sends the following metadata to dbt Labs servers: * **Last-modified timestamps**: Used to determine whether upstream data has changed since the last run * **SQL statement hashes**: SQL statements are processed to detect and classify changes, then hashed. Only the hash is persisted for future comparisons. No actual data from your warehouse is transmitted. The dbt State service runs in a single US multi-tenant (MT) instance. The service never connects to your data warehouse. No actual data from your warehouse is transmitted. The only connection is to your running dbt process (CLI or platform) in order to exchange the metadata described above. For data retention details, refer to the [dbt Labs privacy policy](https://www.getdbt.com/cloud/privacy-policy). --- ### How is dbt State different from using state:modified? `state:modified` in dbt v1 requires manual management of `manifest.json`, which is cumbersome and error-prone. dbt State is completely managed with almost zero setup and no workflow changes. `state:modified` only checks if a file has changed. dbt State has semantic understanding of SQL, so meaningless changes like whitespace or table aliases are not counted as a change — making dbt State smarter about what actually needs to rebuild. `state:modified` does not consider upstream data changes. dbt State checks all sources to see if there is any new data or if the schema has been modified. This enables dbt State to skip running models if the result of the run would be the same as before. For example, if you run `dbt run` with `state:modified` twice, it runs all modified models both times. dbt State only reruns models the second time if upstream sources have changed. dbt State also has the ability to auto-defer refs and automatically clone tables when the result of the clone would have been the same as a full model run. `state:modified` has a limitation on seed files over 1MB, while dbt State does not. --- ### How often is cost data refreshed? Cost data refreshes daily and reflects the previous day's usage. This means there is a lag of up to one day between when a job runs and when its cost data appears in Cost Insights. --- ### How often should I run the snapshot command? Snapshots are a batch-based approach to [change data capture](https://en.wikipedia.org/wiki/Change_data_capture). The `dbt snapshot` command must be run on a schedule to ensure that changes to tables are actually recorded! While individual use-cases may vary, snapshots are intended to be run between hourly and daily. If you find yourself snapshotting more frequently than that, consider if there isn't a more appropriate way to capture changes in your source data tables. --- ### How should I structure my project? There's no one best way to structure a project! Every organization is unique. If you're just getting started, check out how we (dbt Labs) [structure our dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md). --- ### How to delete a job or environment in dbt? To delete an environment or job in dbt, you must have a `developer` [license](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md) and have the necessary [access permissions](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md). ##### Delete a job To delete a job or multiple jobs in dbt: 1. Click **Deploy** on the navigation header. 2. Click **Jobs** and select the job you want to delete. 3. Click **Settings** on the top right of the page and then click **Edit**. 4. Scroll to the bottom of the page and click **Delete job** to delete the job. <br /> [![Delete a job](/img/docs/dbt-platform/platform-configuring-dbt-platform/delete-job.png?v=2 "Delete a job")](#)Delete a job 5. Confirm your action in the pop-up by clicking **Confirm delete** in the bottom right to delete the job immediately. This action cannot be undone. However, you can create a new job with the same information if the deletion was made in error. 6. Refresh the page, and the deleted job should now be gone. If you want to delete multiple jobs, you'll need to perform these steps for each job. If you're having any issues, feel free to [contact us](mailto:support@getdbt.com) for additional help. ##### Delete an environment Deleting an environment automatically deletes its associated job(s). If you want to keep those jobs, move them to a different environment first. Follow these steps to delete an environment in dbt: 1. Navigate to **Orchestration** > **Environments**. 2. Select the environment you want to delete. 3. Click **Settings** on the top right of the page and then click **Edit**. 4. Scroll to the bottom of the page and click **Delete** to delete the environment. [![Delete an environment](/img/docs/dbt-platform/platform-configuring-dbt-platform/delete-environment.png?v=2 "Delete an environment")](#)Delete an environment 5. Confirm your action in the pop-up by clicking **Confirm delete** in the bottom right to delete the environment immediately. This action cannot be undone. However, you can create a new environment with the same information if the deletion was made in error. 6. Refresh your page and the deleted environment should now be gone. To delete multiple environments, you'll need to perform these steps to delete each one. If you're having any issues, feel free to [contact us](mailto:support@getdbt.com) for additional help. --- ### How to generate HAR files HTTP Archive (HAR) files are used to gather data from users’ browser, which dbt Support uses to troubleshoot network or resource issues. This information includes detailed timing information about the requests made between the browser and the server. The following sections describe how to generate HAR files using common browsers such as [Google Chrome](#google-chrome), [Mozilla Firefox](#mozilla-firefox), [Apple Safari](#apple-safari), and [Microsoft Edge](#microsoft-edge). info Remove or hide any confidential or personally identifying information before you send the HAR file to dbt Labs. You can edit the file using a text editor. ##### Google Chrome 1. Open Google Chrome. 2. Click on **View** --> **Developer Tools**. 3. Select the **Network** tab. 4. Ensure that Google Chrome is recording. A red button (🔴) indicates that a recording is already in progress. Otherwise, click **Record network log**. 5. Select **Preserve Log**. 6. Clear any existing logs by clicking **Clear network log** (🚫). 7. Go to the page where the issue occurred and reproduce the issue. 8. Click **Export HAR** (the down arrow icon) to export the file as HAR. The icon is located on the same row as the **Clear network log** button. 9. Save the HAR file. 10. Upload the HAR file to the dbt Support ticket thread. ##### Mozilla Firefox 1. Open Firefox. 2. Click the application menu and then **More tools** --> **Web Developer Tools**. 3. In the developer tools docked tab, select **Network**. 4. Go to the page where the issue occurred and reproduce the issue. The page automatically starts recording as you navigate. 5. When you're finished, click **Pause/Resume recording network log**. 6. Right-click anywhere in the **File** column and select **Save All as HAR**. 7. Save the HAR file. 8. Upload the HAR file to the dbt Support ticket thread. ##### Apple Safari 1. Open Safari. 2. In case the **Develop** menu doesn't appear in the menu bar, go to **Safari** and then **Settings**. 3. Click **Advanced**. 4. Select the **Show features for web developers** checkbox. 5. From the **Develop** menu, select **Show Web Inspector**. 6. Click the **Network tab**. 7. Go to the page where the issue occurred and reproduce the issue. 8. When you're finished, click **Export**. 9. Save the file. 10. Upload the HAR file to the dbt Support ticket thread. ##### Microsoft Edge 1. Open Microsoft Edge. 2. Click the **Settings and more** menu (...) to the right of the toolbar and then select **More tools** --> **Developer tools**. 3. Click **Network**. 4. Ensure that Microsoft Edge is recording. A red button (🔴) indicates that a recording is already in progress. Otherwise, click **Record network log**. 5. Go to the page where the issue occurred and reproduce the issue. 6. When you're finished, click **Stop recording network log**. 7. Click **Export HAR** (the down arrow icon) or press **Ctrl + S** to export the file as HAR. 8. Save the HAR file. 9. Upload the HAR file to the dbt Support ticket thread. ##### Additional resources Check out the [How to generate a HAR file in Chrome](https://www.loom.com/share/cabdb7be338243f188eb619b4d1d79ca) video for a visual guide on how to generate HAR files in Chrome. --- ### How to migrate git providers To migrate from one git provider to another, refer to the following steps to avoid minimal disruption: 1. Outside of dbt, you'll need to import your existing repository into your new provider. By default, connecting your repository in one account won't automatically disconnected it from another account. As an example, if you're migrating from GitHub to Azure DevOps, you'll need to import your existing repository (GitHub) into your new Git provider (Azure DevOps). For detailed steps on how to do this, refer to your Git provider's documentation (Such as [GitHub](https://docs.github.com/en/migrations/importing-source-code/using-github-importer/importing-a-repository-with-github-importer), [GitLab](https://docs.gitlab.com/ee/user/project/import/repo_by_url.html), [Azure DevOps](https://learn.microsoft.com/en-us/azure/devops/repos/git/import-git-repository?view=azure-devops)) 2. Go back to dbt and set up your [integration for the new Git provider](https://docs.getdbt.com/docs/platform/git/configure-git.md), if needed. 3. Disconnect the old repository in dbt by going to **Account Settings** and then **Projects**. 4. Click on the **Repository** link, then click **Edit** and **Disconnect**. [![Disconnect and reconnect your Git repository in your dbt Account settings page.](/img/docs/dbt-platform/disconnect-repo.png?v=2 "Disconnect and reconnect your Git repository in your dbt Account settings page.")](#)Disconnect and reconnect your Git repository in your dbt Account settings page. 5. Click **Confirm Disconnect**. 6. On the same page, connect to the new Git provider repository by clicking **Configure Repository** * If you're using the native integration, you may need to OAuth to it. 7. That's it, you should now be connected to the new Git provider! 🎉 Note — As a tip, we recommend you refresh your page and Studio IDE before performing any actions. --- ### How to upgrade a dbt account dbt offers [several plans](https://www.getdbt.com/pricing/) with different features that meet your needs. This document is for dbt admins and explains how to select a plan in order to continue using dbt. #### Prerequisites Before you begin: * You *must* be part of the [Owner](https://docs.getdbt.com/docs/platform/manage-access/self-service-permissions.md) user group to make billing changes. Users not included in this group will not see these options. * All amounts shown in dbt are in U.S. Dollars (USD) * When your trial expires, your account's default plan enrollment will be a Starter plan. #### Select a plan When your [14 day trial](https://www.getdbt.com/signup/) ends or if your subscription payment is past due , you'll need to select a plan in order to continue using your account: * Upon logging in, you should see an "Account locked" pop up message with instructions to unlock your account and update your payment details * Click **Go to Billing** to go to the billing page * Under **Billing**, you can review the available dbt [plans](https://www.getdbt.com/pricing/) and their features To unlock your account and select a plan, review the following guidance per plan type: ##### Developer plan 1. To select a Developer plan, click Select plan on the right. 2. Confirm your plan selection on the pop up message. 3. This automatically unlocks your dbt account, and you can now enjoy the benefits of the Developer plan. 🎉 [![](/img/docs/dbt-platform/downgrade-dev-flow.gif?v=2)](#) ###### Plan allocation If you select a plan but have too many seats or projects for that plan (for example, if you select the Developer plan but have more than one developer seat), you'll be directed to the users & projects pages to make edits. ##### Starter plan 1. When your trial expires, your account's default plan enrollment will be a Starter plan. 2. To unlock your account and continue using the Starter plan, click on **Select plan** under the Starter column. 3. Enter your payment information and seat purchases. Then click **Save**. 4. This automatically unlocks your dbt account, and you can now enjoy the benefits of the Starter plan. 🎉 [![](/img/docs/dbt-platform/trial-team-flow.png?v=2)](#) [![](/img/docs/dbt-platform/trial-team-payments-flow.png?v=2)](#) ##### Enterprise plan 1. If you're interested in one of our Enterprise-tier plans, select the Enterprise tab under **Billing**. 2. Click **Contact Sales** on the right. This opens a chat window for you to contact the dbt Support team, who will connect you to our Sales team. 3. Once you submit your request, our Sales team will contact you with more information. [![](/img/docs/dbt-platform/enterprise-upgrade.gif?v=2)](#) 4. Alternatively, you can [contact](https://www.getdbt.com/contact/) our Sales team directly to chat about how dbt can help you and your team. #### Related questions For commonly asked billings questions, refer to the dbt [pricing page](https://www.getdbt.com/pricing/). How does billing work? Starter plans are billed monthly on the credit card used to sign up, based on [developer seat count and usage](https://docs.getdbt.com/docs/platform/billing.md). You’ll also be sent a monthly receipt to the billing email of your choice. You can change any billing information in your **Account Settings** > **Billing page**. Enterprise-tier plan customers are billed annually based on the number of developer seats, as well as any additional services + features in your chosen plan. Can I upgrade or downgrade my plan? Yes, you can upgrade or downgrade at any time. Account Owners can access their dedicated billing section via the account settings page. If you’re not sure which plan is right for you, get in touch and we’ll be happy to help you find one that fits your needs. What happens when I upgrade from Developer to Starter? The free two-week trial is only available to new accounts at signup — but if you're already on the Developer plan, you can upgrade to Starter and immediately access all Starter features without needing to go through the trial. When you upgrade from Developer to Starter, you move directly to paid billing — you'll enter your payment information and select your developer license seats to complete the upgrade. Your existing projects and settings carry over. Can I pay by invoice? Currently, dbt Starter plan payments must be made with a credit card, and by default they will be billed monthly based on the number of [developer seats and usage](https://docs.getdbt.com/docs/platform/billing.md). We don’t have any plans to do invoicing for Starter plan accounts in the near future, but we do currently support invoices for companies on the dbt Enterprise-tier plan. Feel free to [contact](https://www.getdbt.com/contact/) us to build your Enterprise pricing plan. Why did I receive a **Failed payment** error email? This means we were unable to charge the credit card you have on file, or you have not provided an updated card for payment. If you're a current account owner with a card on file, contact your credit card issuer to inquire as to why your card was declined or update the credit card on your account. Your Account Owner can update payment details in the **Account Settings** -> **Billing** page. Click **Edit** next to your card details, double check your information is up-to-date, and we'll give it another go at the next billing run. --- ### I got an "unused model configurations" error message, what does this mean? You might have forgotten to nest your configurations under your project name, or you might be trying to apply configurations to a directory that doesn't exist. Check out this [article](https://discourse.getdbt.com/t/faq-i-got-an-unused-model-configurations-error-message-what-does-this-mean/112) to understand more. --- ### I need to use quotes to select from my source, what should I do? This is reasonably common on Snowflake in particular. By default, dbt will not quote the database, schema, or identifier for the source tables that you've specified. To force dbt to quote one of these values, use the [`quoting` property](https://docs.getdbt.com/reference/resource-properties/quoting.md): models/\<filename>.yml ```yaml sources: - name: jaffle_shop database: raw schema: jaffle_shop quoting: database: true schema: true identifier: true tables: - name: order_items - name: orders # This overrides the `jaffle_shop` quoting config quoting: identifier: false ``` --- ### I'm getting a "Partial parsing enabled: 1 files deleted, 0 files added, 2 files changed" compilation error in dbt? If you're receiving this error, try deleting the `target/partial_parse.msgpack` file from your project and refresh your IDE. If you've tried the workaround above and are still experiencing this behavior - reach out to the Support team at <support@getdbt.com> and we'll be happy to help! --- ### I'm getting a "Session occupied" error in dbt platform CLI? If you're receiving a `Session occupied` error in the dbt platform CLI or if you're experiencing a long-running session, you can use the `dbt invocation list` command in a separate terminal window to view the status of your active session. This helps debug the issue and identify the arguments that are causing the long-running session. To cancel an active session, use the `Ctrl + Z` shortcut. To learn more about the `dbt invocation` command, see the [dbt invocation command reference](https://docs.getdbt.com/reference/commands/invocation.md?version=2.0). Alternatively, you can reattach to your existing session with `dbt reattach` and then press `Control-C` and choose to cancel the invocation. --- ### I'm receiving a 'This run exceeded your account's run memory limits' error in my failed job If you're receiving a `This run exceeded your account's run memory limits` error in your failed job, it means that the job exceeded the [memory limits](https://docs.getdbt.com/docs/deploy/job-scheduler.md#job-memory) set for your account. All dbt accounts have a pod memory of 600Mib and memory limits are on a per run basis. They're typically influenced by the amount of result data that dbt has to ingest and process, which is small but can become bloated unexpectedly by project design choices. ##### Common reasons Some common reasons for higher memory usage are: * dbt run/build: Macros that capture large result sets from run query may not all be necessary and may be memory inefficient. * dbt docs generate: Source or model schemas with large numbers of tables (even if those tables aren't all used by dbt) cause the ingest of very large results for catalog queries. ##### Resolution There are various reasons why you could be experiencing this error but they are mostly the outcome of retrieving too much data back into dbt. For example, using the `run_query()` operations or similar macros, or even using database/schemas that have a lot of other non-dbt related tables/views. Try to reduce the amount of data / number of rows retrieved back into dbt by refactoring the SQL in your `run_query()` operation using `group`, `where`, or `limit` clauses. Additionally, you can also use a database/schema with fewer non-dbt related tables/views. Video example As an additional resource, check out [this example video](https://www.youtube.com/watch?v=sTqzNaFXiZ8), which demonstrates how to refactor the sample code by reducing the number of rows returned. If you've tried the earlier suggestions and are still experiencing failed job runs with this error about hitting the memory limits of your account, please [reach out to support](mailto:support@getdbt.com). We're happy to help! ##### Additional resources * [Blog post on how we shaved 90 mins off](https://docs.getdbt.com/blog/how-we-shaved-90-minutes-off-model) --- ### I'm receiving a "Permission denied while getting Drive credential" error when trying to query from Google Drive? If you're seeing the below error when you try to query a dataset from a Google Drive document in the Studio IDE, the Studio IDE due to the below error message, we'll do our best to get you unstuck with the below steps! ```text Access denied: BigQuery BigQuery: Permission denied while getting Drive credentials ``` Usually, this error indicates that you haven't granted the BigQuery service account access to the specific Google Drive document. If you're seeing this error, try giving the service account (Client email field seen [here](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-bigquery.md)) you are using for your BigQuery connection in dbt, permission to your Google Drive or Google Sheet. You'll want to do this directly in your Google Document, click the **Share** button, and enter the client email. If you are experiencing this error when using OAuth, and you have verified your access to the Google Sheet, you may need to grant permissions for gcloud to access Google Drive: ```text gcloud auth application-default login --disable-quota-project ``` For more info see the [gcloud auth application-default documentation](https://cloud.google.com/sdk/gcloud/reference/auth/application-default/login) If you've tried the earlier steps and are still experiencing this behavior, try using the following command to log into Google Cloud and enable access to Google Drive. It also updates the Application Default Credentials (ADC) file, which many Google Cloud libraries use to authenticate API calls. ```text gcloud auth login --enable-gdrive-access --update-adc ``` For more info, refer to [gcloud auth login documentation](https://cloud.google.com/sdk/gcloud/reference/auth/login#--enable-gdrive-access). If you've tried the steps above and are still experiencing this behavior - reach out to the Support team at <support@getdbt.com> and we'll be happy to help! --- ### I'm receiving a 403 error 'Forbidden: Access denied' when using service tokens All [service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) traffic is subject to IP restrictions. When using a service token, the following 403 response error indicates the IP is not on the allowlist. To resolve this, you should add your third-party integration CIDRs (network addresses) to your allowlist. The following is an example of the 403 response error: ```json { "status": { "code": 403, "is_success": False, "user_message": ("Forbidden: Access denied"), "developer_message": None, }, "data": { "account_id": <account_id>, "user_id": <user_id>, "is_service_token": <boolean describing if it's a service token request>, "account_access_denied": True, }, } ``` --- ### I'm receiving a git rev-list master error in the IDE? If you're unable to access the Studio IDE due to the below error message, we'll do our best to get you unstuck with the below steps! ```shell git rev-list master..origin/main --count fatal: ambiguous argument 'master..origin/main': unknown revision or path not in the working tree. Use '--' to separate paths from revisions, like this: 'git <command> [<revision>...] -- [<file>...]' ``` Usually this error indicates that the "main" branch name has changed or it is possible that dbt was unable to determine what your primary branch was. No worries, we have a few workarounds for you to try: **Workaround 1** Take a look at your Environment Settings - If you **do not** have a custom branch filled in your Environment Settings: 1. Disconnect and reconnect your repository [connection](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md) on your Project Settings page. This should then allow dbt to pick up that the "main" branch is now called `main`. 2. In the Environment Settings, set the custom branch to 'master' and refresh the Studio IDE. **Workaround 2** Take a look at your Environment Settings - If you **do** have a custom branch filled in your Environment Settings: 1. Disconnecting and reconnecting your repository [connection](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md) on your Project Settings page. This should then allow dbt to pick up that the "main" branch is now called `main`. 2. In the Environment Settings, remove the custom branch and refresh the Studio IDE. If you've tried the workarounds above and are still experiencing this behavior - reach out to the Support team at <support@getdbt.com> and we'll be happy to help! --- ### I'm receiving a NoneType object has no attribute error in the IDE? If you're unable to access the Studio IDE due to the below error message, we'll do our best to get you unstuck with the below steps! ```shell NoneType object has no attribute enumerate_fields' ``` Usually this error indicates that you tried connecting your database via [SSH tunnel](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-redshift.md#connecting-using-an-ssh-tunnel). If you're seeing this error, double-check you have supplied the following items: * the hostname * username * port of bastion server If you've tried the step above and are still experiencing this behavior - reach out to the Support team at <support@getdbt.com> and we'll be happy to help! --- ### I'm receiving a Runtime Error Could not find profile named 'user' error? If you're unable to access the Studio IDE due to the below error message, we'll do our best to get you unstuck with the below steps! ```shell Running with dbt=1.9.0 Encountered an error while reading the project: ERROR: Runtime Error Could not find profile named 'user' Runtime Error Could not run dbt' ``` Usually this errors indicates that there is an issue with missing/stale credentials/authentication. No worries, we have a few workarounds for you to try: **In the Studio IDE:** Click your account name in the bottom left, select **Your profile**, then go to **Credentials**. Re-enter or re-authorize your credentials to resolve the error. **In a job:** This can happen if you changed the deployment environment and didn't re-enter your deployment credentials when saving. Go back to the deployment environment settings, re-enter your credentials (private key/passphrase or username and password), and kick off a new job run. If you've tried the step above and are still experiencing this behavior - reach out to the Support team at <support@getdbt.com> and we'll be happy to help! --- ### I'm receiving an 'Your IDE session experienced an unknown error and was terminated. Please contact support'. If you're seeing the following error when you launch the Studio IDE, it could be due to a few scenarios but, commonly, it indicates a missing repository: ```shell Your Studio IDE session experienced an unknown error and was terminated. Please contact support. ``` You can try to resolve this by adding a repository like a [managed repository](https://docs.getdbt.com/docs/platform/git/managed-repository.md) or your preferred Git account. To add your Git account, navigate to **Project** > **Repository** and select your repository. If you're still running into this error, please contact the Support team at <support@getdbt.com> for help. --- ### I'm receiving an `Failed ALPN` error when trying to connect to the dbt Semantic Layer. If you're receiving a `Failed ALPN` error when trying to connect the dbt Semantic Layer with the various [data integration tools](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) (such as Tableau, DBeaver, Datagrip, ADBC, or JDBC), it typically happens when connecting from a computer behind a corporate VPN or Proxy (like Zscaler or Check Point). The root cause is typically the proxy interfering with the TLS handshake as the Semantic Layer uses gRPC/HTTP2 for connectivity. To resolve this: * If your proxy supports gRPC/HTTP2 but isn't configured to allow ALPN, adjust its settings accordingly to allow ALPN. Or create an exception for the dbt domain. * If your proxy does not support gRPC/HTTP2, add an SSL interception exception for the dbt domain in your proxy settings This should help in successfully establishing the connection without the Failed ALPN error. --- ### I'm seeing a 'GitHub and dbt latest permissions' error If you see the error `This account needs to accept the latest permissions for the dbt GitHub App` in dbt — this usually occurs when the permissions for the dbt GitHub App are out-of-date. To solve this issue, you'll need to update the permissions for the dbt GitHub App in your GitHub account. This FAQ shares a couple of ways you can do it. #### Update permissions A GitHub organization admin will need to update the permissions in GitHub for the dbt GitHub App. If you're not the admin, reach out to your organization admin to request this. 1. Navigate to your GitHub account. Click on the top right profile icon and then **Settings** (or personal if using a non-organization account). 2. Then go to **Integrations** and then select **Applications** to identify any necessary permission changes. Note that a GitHub repository admin may not see the same permission request. [![Navigate to Application settings to identify permission changes.](/img/docs/dbt-platform/platform-configuring-dbt-platform/github-applications.png?v=2 "Navigate to Application settings to identify permission changes.")](#)Navigate to Application settings to identify permission changes. 3. Click on **Review request** and then click on the **Accept new permissions** button on the next page. [![Grant access to the dbt app by accepting the new permissions.](/img/docs/dbt-platform/platform-configuring-dbt-platform/github-review-request.png?v=2 "Grant access to the dbt app by accepting the new permissions.")](#)Grant access to the dbt app by accepting the new permissions. For more info on GitHub permissions, refer to [access permissions](https://docs.github.com/en/get-started/learning-about-github/access-permissions-on-github). Alternatively, try [disconnecting your GitHub account](#disconnect-github) in dbt, detailed in the following section. #### Disconnect GitHub Disconnect the GitHub and dbt integration in dbt. 1. In dbt, go to **Account Settings**. 2. In **Projects**, select the project experiencing the issue. 3. Click the repository link under **Repository**. 4. In the **Repository details** page, click **Edit**. 5. Click **Disconnect** to remove the GitHub integration. [![Disconnect and reconnect your git repository in your dbt Account settings pages.](/img/docs/dbt-platform/disconnect-repo.png?v=2 "Disconnect and reconnect your git repository in your dbt Account settings pages.")](#)Disconnect and reconnect your git repository in your dbt Account settings pages. 6. Click **Confirm Disconnect**. 7. Return to your **Project details** page and reconnect your repository by clicking the **Configure Repository** link. 8. Click **GitHub** and select your repository. #### Support If you've tried these workarounds and are still experiencing this behavior — reach out to the [dbt Support](mailto:support@getdbt.com) team and we'll be happy to help! --- ### I'm seeing a Gitlab authentication out of date error loop If you're seeing a 'GitLab Authentication is out of date' 500 server error page - this usually occurs when the deploy key in the repository settings in both dbt and GitLab do not match. No worries - this is a current issue the dbt Labs team is working on and we have a few workarounds for you to try: ###### First workaround 1. Disconnect repo from project in dbt. 2. Go to Gitlab and click on Settings > Repository. 3. Under Repository Settings, remove/revoke active dbt deploy tokens and deploy keys. 4. Attempt to reconnect your repository via dbt. 5. You would then need to check Gitlab to make sure that the new deploy key is added. 6. Once confirmed that it's added, refresh dbt and try developing once again. ###### Second workaround 1. Keep repo in project as is -- don't disconnect. 2. Copy the deploy key generated in dbt. 3. Go to Gitlab and click on Settings > Repository. 4. Under Repository Settings, manually add to your Gitlab project deploy key repo (with `Grant write permissions` box checked). 5. Go back to dbt, refresh your page and try developing again. If you've tried the workarounds above and are still experiencing this behavior - reach out to the Support team at <support@getdbt.com> and we'll be happy to help! --- ### I'm seeing a Gitlab authentication out of date error loop If you're seeing a 'GitLab Authentication is out of date' 500 server error page - this usually occurs when the deploy key in the repository settings in both dbt and GitLab do not match. No worries - this is a current issue the dbt Labs team is working on and we have a few workarounds for you to try: ##### 1st Workaround 1. Disconnect repo from project in dbt. 2. Go to Gitlab and click on Settings > Repository. 3. Under Repository Settings, remove/revoke active dbt deploy tokens and deploy keys. 4. Attempt to reconnect your repository via dbt. 5. You would then need to check Gitlab to make sure that the new deploy key is added. 6. Once confirmed that it's added, refresh dbt and try developing once again. ##### 2nd Workaround 1. Keep repo in project as is -- don't disconnect. 2. Copy the deploy key generated in dbt. 3. Go to Gitlab and click on Settings > Repository. 4. Under Repository Settings, manually add to your Gitlab project deploy key repo (with `Grant write permissions` box checked). 5. Go back to dbt, refresh your page and try developing again. If you've tried the workarounds above and are still experiencing this behavior - reach out to the Support team at <support@getdbt.com> and we'll be happy to help! --- ### If I can name these files whatever I'd like, what should I name them? It's up to you! Here's a few options: * Default to the existing terminology: `schema.yml` (though this does make it hard to find the right file over time) * Use the same name as your directory (assuming you're using sensible names for your directories) * If you test and document one model (or seed, snapshot, macro etc.) per file, you can give it the same name as the model (or seed, snapshot, macro etc.) Choose what works for your team. We have more recommendations in our guide on [structuring dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md). --- ### If I rerun dbt, will there be any downtime as models are rebuilt? Nope! The SQL that dbt generates behind the scenes ensures that any relations are replaced atomically (i.e. your business users won't experience any downtime). The implementation of this varies on each warehouse, check out the [logs](https://docs.getdbt.com/faqs/Runs/checking-logs.md) to see the SQL dbt is executing. --- ### If models can only be `select` statements, how do I insert records? For those coming from an ETL (Extract Transform Load) paradigm, there's often a desire to write transformations as `insert` and `update` statements. In comparison, dbt will wrap your `select` query in a `create table as` statement, which can feel counter-productive. * If you wish to use `insert` statements for performance reasons (i.e. to reduce data that is processed), consider [incremental models](https://docs.getdbt.com/docs/build/incremental-models.md) * If you wish to use `insert` statements since your source data is constantly changing (e.g. to create "Type 2 Slowly Changing Dimensions"), consider [snapshotting your source data](https://docs.getdbt.com/docs/build/sources.md#source-data-freshness), and building models on top of your snaphots. --- ### My compiled SQL has a lot of spaces and new lines, how can I get rid of it? This is known as "whitespace control". Use a minus sign (`-`, e.g. `{{- ... -}}`, `{%- ... %}`, `{#- ... -#}`) at the start or end of a block to strip whitespace before or after the block (more docs [here](https://jinja.palletsprojects.com/page/templates/#whitespace-control)). Check out the [tutorial on using Jinja](https://docs.getdbt.com/guides/using-jinja.md#use-whitespace-control-to-tidy-up-compiled-code) for an example. Take caution: it's easy to fall down a rabbit hole when it comes to whitespace control! --- ### One of my tests failed, how can I debug it? To debug a failing test, find the SQL that dbt ran by: * dbt: * Within the test output, click on the failed test, and then select "Details". * dbt v1: * Open the file path returned as part of the error message. * Navigate to the `target/compiled/schema_tests` directory for all compiled test queries. Copy the SQL into a query editor (in dbt, you can paste it into a new `Statement`), and run the query to find the records that failed. --- ### Receiving a 'Could not parse dbt_project.yml' error in dbt job The error message `Could not parse dbt_project.yml: while scanning for...` in your dbt job run or development usually occurs for several reasons: * There's a parsing failure in a YAML file (such as a tab indentation or Unicode characters). * Your `dbt_project.yml` file has missing fields or incorrect formatting. * Your `dbt_project.yml` file doesn't exist in your dbt project repository. To resolve this issue, consider the following: * Use an online YAML parser or validator to check for any parsing errors in your YAML file. Some known parsing errors include missing fields, incorrect formatting, or tab indentation. * Or ensure your `dbt_project.yml` file exists. Once you've identified the issue, you can fix the error and rerun your dbt job. --- ### Receiving a `Failed to connect to DB` error when connecting to Snowflake 1. If you see the following error: ```text Failed to connect to DB: xxxxxxx.snowflakecomputing.com:443. The role requested in the connection, or the default role if none was requested in the connection ('xxxxx'), is not listed in the Access Token or was filtered. Please specify another role, or contact your OAuth Authorization server administrator. ``` 2. Edit your OAuth Security integration and explicitly specify this scope mapping attribute: ```sql ALTER INTEGRATION <my_int_name> SET EXTERNAL_OAUTH_SCOPE_MAPPING_ATTRIBUTE = 'scp'; ``` You can read more about this error in [Snowflake's documentation](https://community.snowflake.com/s/article/external-custom-oauth-error-the-role-requested-in-the-connection-is-not-listed-in-the-access-token). *** 1. If you see the following error: ```text Failed to connect to DB: xxxxxxx.snowflakecomputing.com:443. Incorrect username or password was specified. ``` * **Unique email addresses** — Each user in Snowflake must have a unique email address. You can't have multiple users (for example, a human user and a service account) using the same email, such as `alice@acme.com`, to authenticate to Snowflake. * **Match email addresses with identity provider** — The email address of your Snowflake user must exactly match the email address you use to authenticate with your Identity Provider (IdP). For example, if your Snowflake user's email is `alice@acme.com` but you log in to Entra or Okta with `alice_adm@acme.com`, this mismatch can cause an error. --- ### Reconnecting to Snowflake OAuth after authentication expires When you connect Snowflake to dbt platform using [OAuth](https://docs.getdbt.com/docs/platform/manage-access/set-up-snowflake-oauth.md), dbt stores a refresh token. This allows your user credentials to be reused in tools like the Studio IDE and the dbt Semantic Layer without needing to re-authenticate each time. If you see an `authentication has expired` error when you try to run queries, you must renew your connection between Snowflake and the dbt platform. To resolve the issue, complete the following steps: 1. Go to **Your profile**, accessible from the navigation menu. 2. Navigate to **Credentials** and then choose the project where you're experiencing the issue. 3. Under **User credentials**, click the **Reconnect Snowflake Account** button. This will guide you through re-authenticating using your SSO workflow. Your Snowflake administrator can [configure the refresh token validity period](https://docs.getdbt.com/docs/platform/manage-access/set-up-snowflake-oauth.md#create-a-security-integration), up to the maximum 90 days. If you've tried these step and are still getting this error, please contact the Support team at <support@getdbt.com> for further assistance. --- ### Should I use separate files to declare resource properties, or one large file? It's up to you: * Some folks find it useful to have one file per model (or source / snapshot / seed etc) * Some find it useful to have one per directory, documenting and testing multiple models in one file Choose what works for your team. We have more recommendations in our guide on [structuring dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md). --- ### Studio IDE freezes when opening the Lineage tab If the Studio IDE freezes with two or more models open and the **Lineage** tab active, use the following workarounds. ##### Why this happens The Studio IDE can freeze when the **Lineage** tab is open or loads automatically, and when two or more models are open, especially if a model has a large or slow-loading DAG. #### What you might observe * The editor becomes unresponsive. * After a few minutes, your browser may display a **Wait** or **Exit** dialog. #### Workaround 1: Avoid the Lineage tab on startup 1. Refresh the page. 2. Keep the console focused on another tab (for example, **Results** or **Compiled code**) instead of **Lineage**. 3. Close extra model tabs before opening **Lineage**. 4. Open **Lineage** only when needed. #### Workaround 2: Clear Local Storage for your account URL This workaround is browser-dependent. In DevTools, select the Local Storage origin that matches the URL in your browser address bar. For example, your account access URL might look like `https://ab123.us1.dbt.com`. ##### Google Chrome and Microsoft Edge 1. Open the dbt platform page that’s freezing. 2. Open DevTools (right-click the page and select **Inspect**). 3. Click the **Application** tab. 4. In the left sidebar, expand **Storage** → **Local Storage**. 5. Right-click the origin that matches your current URL (for example, `https://ab123.us1.dbt.com`) and select **Clear**. 6. Refresh the page and sign in again if prompted. ##### Mozilla Firefox 1. Open the dbt platform page that’s freezing. 2. Open DevTools (right-click the page and select **Inspect**). 3. Open the **Storage** tab. * If you don’t see it, open DevTools **Settings** and enable **Storage**. 4. In the left sidebar, expand **Local Storage**. 5. Right-click the origin that matches your current URL (for example, `https://ab123.us1.dbt.com`) and select **Delete All**. 6. Refresh the page and sign in again if prompted. ##### Safari 1. Open Safari and go to the dbt platform page that’s freezing. 2. If needed, enable developer features: * Go to **Safari** > **Settings**. * Click **Advanced**. * Enable developer features. 3. Open Web Inspector from **Develop** > **Show Web Inspector**. 4. Open the **Storage** tab. 5. In the left sidebar, expand **Local Storage** and select the origin that matches your current URL (for example, `https://ab123.us1.dbt.com`). 6. Delete the Local Storage entries for that origin. 7. Refresh the page and sign in again if prompted. #### Need more help? If freezing continues after trying these workarounds, [contact dbt Support](mailto:support@getdbt.com) and include: * A HAR file (refer to [How to generate HAR files](https://docs.getdbt.com/faqs/Troubleshooting/generate-har-file.md)) * Your account URL * Browser and version * Approximate number of open model tabs * Whether the issue happens only when the **Lineage** tab is active --- ### The columns of my seed changed, and now I get an error when running the `seed` command, what should I do? If you changed the columns of your seed, you may get a `Database Error`: ##### Snowflake ```shell $ dbt seed Running with dbt=1.6.0-rc2 Found 0 models, 0 tests, 0 snapshots, 0 analyses, 130 macros, 0 operations, 1 seed file, 0 sources 12:12:27 | Concurrency: 8 threads (target='dev_snowflake') 12:12:27 | 12:12:27 | 1 of 1 START seed file dbt_claire.country_codes...................... [RUN] 12:12:30 | 1 of 1 ERROR loading seed file dbt_claire.country_codes.............. [ERROR in 2.78s] 12:12:31 | 12:12:31 | Finished running 1 seed in 10.05s. Completed with 1 error and 0 warnings: Database Error in seed country_codes (seeds/country_codes.csv) 000904 (42000): SQL compilation error: error line 1 at position 62 invalid identifier 'COUNTRY_NAME' Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1 ``` ##### Redshift ```shell $ dbt seed Running with dbt=1.6.0-rc2 Found 0 models, 0 tests, 0 snapshots, 0 analyses, 149 macros, 0 operations, 1 seed file, 0 sources 12:14:46 | Concurrency: 1 threads (target='dev_redshift') 12:14:46 | 12:14:46 | 1 of 1 START seed file dbt_claire.country_codes...................... [RUN] 12:14:46 | 1 of 1 ERROR loading seed file dbt_claire.country_codes.............. [ERROR in 0.23s] 12:14:46 | 12:14:46 | Finished running 1 seed in 1.75s. Completed with 1 error and 0 warnings: Database Error in seed country_codes (seeds/country_codes.csv) column "country_name" of relation "country_codes" does not exist Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1 ``` In this case, you should rerun the command with a `--full-refresh` flag, like so: ```text dbt seed --full-refresh ``` **Why is this the case?** When you typically run dbt seed, dbt truncates the existing table and reinserts the data. This pattern avoids a `drop cascade` command, which may cause downstream objects (that your BI users might be querying!) to get dropped. However, when column names are changed, or new columns are added, these statements will fail as the table structure has changed. The `--full-refresh` flag will force dbt to `drop cascade` the existing table before rebuilding it. --- ### Unable to trigger a CI job with GitLab When you connect dbt to a GitLab repository, GitLab automatically registers a webhook in the background, viewable under the repository settings. This webhook is also used to trigger [CI jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md) when you push to the repository. If you're unable to trigger a CI job, this usually indicates that the webhook registration is missing or incorrect. To resolve this issue, navigate to the repository settings in GitLab and view the webhook registrations by navigating to GitLab --> **Settings** --> **Webhooks**. Some things to check: * The webhook registration is enabled in GitLab. * The webhook registration is configured with the correct URL and secret. If you're still experiencing this issue, reach out to the Support team at <support@getdbt.com> and we'll be happy to help! --- ### What are the best practices for installing dbt v1 with pip? info dbt v2 is a next-generation, Rust-based engine that powers dbt development across the platform and local tooling. See [dbt v2](https://docs.getdbt.com/docs/introduction.md) for more information. #### Best practices Managing Python local environments can be challenging! You can use these best practices to improve the dbt v1 installation with `pip`. | Best practice | Recommendation | Why it matters | | ---------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | [Install dbt v1 with an adapter](https://docs.getdbt.com/docs/local/install-dbt.md?version=1#installing-the-adapter) and keep versions in sync | Install with: `python -m pip install dbt-ADAPTER_NAME`<br /><br />(For example, `python -m pip install dbt-snowflake`)<br /><br />Match adapter versions to your dbt v1 version<br /><br /> | Provides a complete, compatible, and ready-to-run dbt setup<br /><br /><br /><br />Prevents runtime errors and adapter incompatibilities | | For tooling without a warehouse connection, install v1 without an adapter | `python -m pip install` | Keeps your setup lean, predictable, and easier to maintain | | Use [virtual environments](https://docs.getdbt.com/faqs/Core/install-pip-best-practices.md#using-virtual-environments) | Install dbt in an isolated environment (for example, `venv`, `pipenv`, `poetry`) | Avoids dependency conflicts | | Reactivate your virtual environment for each session | Reactivate your virtual environment at the start of each new session before installing dependencies or running dbt commands | Keeps your dbt setup predictable, isolated, and reproducible | | [Create a project](https://docs.getdbt.com/reference/commands/init.md) | Use the `dbt init` command to create and initialize your first project | Creates a standard dbt project and verifies your installation | | Ensure you have the latest versions of `pip`, `wheel`, and `setuptools` | Before installing dbt, upgrade your Python packaging tools:<br /><br />`python -m pip install --upgrade pip wheel setuptools` | Helps ensure a smoother, more predictable dbt installation | <br /> Note, dbt adapters and dbt v1 are versioned independently to make it easier for us to maintain and independently evolve adapters going forward. ##### Using virtual environments We recommend using [virtual environments](https://docs.python-guide.org/dev/virtualenvs/) to namespace `pip` modules. Here's an example setup: ```shell python3 -m venv dbt-env # create the environment source dbt-env/bin/activate # activate the environment for Mac and Linux dbt-env\Scripts\activate # activate the environment for Windows ``` If you install `dbt` in a virtual environment, you need to reactivate that same virtual environment each time you create a shell window or session. *Tip:* You can create an alias for the `source` command in your `$HOME/.bashrc`, `$HOME/.zshrc`, or whichever rc file your shell draws from. For example, you can add a command like `alias env_dbt='source <PATH_TO_VIRTUAL_ENV_CONFIG>/bin/activate'`, replacing `<PATH_TO_VIRTUAL_ENV_CONFIG>` with the path to your virtual environment configuration. ##### Using the latest versions dbt installations are tested using the latest versions of `pip` and `setuptools`. Newer versions have improved behavior around dependency resolution, as well as much faster install times by using precompiled "wheels" when available for your operating system. Before installing dbt, make sure you have the latest versions: ```shell python -m pip install --upgrade pip wheel setuptools ``` --- ### What data tests are available for me to use in dbt? Out of the box, dbt ships with the following data tests: * `unique` * `not_null` * `accepted_values` * `relationships` (for example, referential integrity) You can also write your own [custom generic tests](https://docs.getdbt.com/docs/build/data-tests.md#generic-data-tests). Some additional generic tests have been open-sourced in the [dbt-utils package](https://github.com/dbt-labs/dbt-utils#generic-tests). Check out the docs on [packages](https://docs.getdbt.com/docs/build/packages.md) to learn how to make these tests available in your project. --- ### What data tests should I add to my project? We recommend that every model has a data test on a primary key, that is, a column that is `unique` and `not_null`. We also recommend that you test any assumptions on your source data. For example, if you believe that your payments can only be one of three payment methods, you should test that assumption regularly — a new payment method may introduce logic errors in your SQL. In advanced dbt projects, we recommend using [sources](https://docs.getdbt.com/docs/build/sources.md) and running these source data-integrity tests against the sources rather than models. --- ### What happened to state-aware orchestration? On June 1, 2026, dbt Labs and Fivetran announced **[dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md)**[Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") as a new and improved version of state-aware orchestration. A key feature is [`lag_tolerance`](https://docs.getdbt.com/reference/resource-configs/lag-tolerance.md), which controls how much time must pass since the last upstream data change before a node is eligible for a rebuild. dbt State improves upon state-aware orchestration in a few key ways: * **Works everywhere** — dbt State works with dbt v1, dbt v2, and dbt platform, as well as external orchestrators, across both development and deployment environments. * **Smarter data freshness tracking** — dbt State tracks data freshness across the DAG and automatically propagates it through models materialized as views. Unlike state-aware orchestration's `build_after` config which compares against the model's last successful execution, dbt State's `lag_tolerance` compares against the freshness of the underlying data. * **Advanced change detection** — dbt State can detect and ignore file modifications that don't change actual transformation logic, such as adding a comment or cleaning up whitespace. If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. For details on billing after the trial ends, refer to [dbt State usage and pricing](https://docs.getdbt.com/docs/platform/billing.md#dbt-state-usage). While dbt State is in preview, there is no required migration timeline — dbt Labs will communicate a timeline when dbt State reaches general availability. To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). --- ### What happens if dbt State servers fail? If dbt State servers are unavailable, dbt gracefully falls back to normal dbt behavior. --- ### What happens if I add new columns to my snapshot query? When the columns of your source query changes, dbt will attempt to reconcile this change in the destination snapshot table. dbt does this by: 1. Creating new columns from the source query in the destination table 2. Expanding the size of string types where necessary (eg. `varchar`s on Redshift) dbt *will not* delete columns in the destination snapshot table if they are removed from the source query. It will also not change the type of a column beyond expanding the size of varchar columns. That is, if a `string` column is changed to a `date` column in the snapshot source query, dbt will not attempt to change the type of the column in the destination table. --- ### What happens if one of my runs fails? If you're using dbt, we recommend setting up email and Slack notifications (`Account Settings > Notifications`) for any failed runs. Then, debug these runs the same way you would debug any runs in development. --- ### What happens if the SQL in my query is bad or I get a database error? If there's a mistake in your SQL, dbt will return the error that your database returns. ```shell $ dbt run --select customers Running with dbt=1.9.0 Found 3 models, 9 tests, 0 snapshots, 0 analyses, 133 macros, 0 operations, 0 seed files, 0 sources 14:04:12 | Concurrency: 1 threads (target='dev') 14:04:12 | 14:04:12 | 1 of 1 START view model dbt_alice.customers.......................... [RUN] 14:04:13 | 1 of 1 ERROR creating view model dbt_alice.customers................. [ERROR in 0.81s] 14:04:13 | 14:04:13 | Finished running 1 view model in 1.68s. Completed with 1 error and 0 warnings: Database Error in model customers (models/customers.sql) Syntax error: Expected ")" but got identifier `your-info-12345` at [13:15] compiled SQL at target/run/jaffle_shop/customers.sql Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1 ``` Any models downstream of this model will also be skipped. Use the error message and the [compiled SQL](https://docs.getdbt.com/faqs/Runs/checking-logs.md) to debug any errors. --- ### What if I work on multiple projects that each use their own dbt State? You can specify your org ID in `dbt_project.yml`: ```yaml dbt-cloud: state-org-id: <your-org-id> ``` --- ### What if my prod environment isn't named prod? You can specify the defer-to environment using the [`defer_to_target`](https://docs.getdbt.com/reference/resource-configs/defer-to-target.md) config in `profiles.yml`: ```yaml my_project: outputs: prod: type: snowflake defer_to_target: production ``` `defer_to_target` only applies to self-managed deployments. If you're using the dbt platform, deferral is configured through your environment settings in the UI. For more details, refer to [Configuring deferral](https://docs.getdbt.com/docs/deploy/dbt-state-deferral.md). --- ### What if my source is in a different database to my target database? Use the [`database` property](https://docs.getdbt.com/reference/resource-properties/database.md) to define the database that the source is in. models/\<filename>.yml ```yml sources: - name: jaffle_shop database: raw schema: jaffle_shop tables: - name: orders - name: customers ``` --- ### What if my source is in a poorly named schema or table? By default, dbt will use the `name:` parameters to construct the source reference. If these names are a little less-than-perfect, use the [schema](https://docs.getdbt.com/reference/resource-properties/schema.md) and [identifier](https://docs.getdbt.com/reference/resource-properties/identifier.md) properties to define the names as per the database, and use your `name:` property for the name that makes sense! models/\<filename>.yml ```yml sources: - name: jaffle_shop database: raw schema: postgres_backend_public_schema tables: - name: orders identifier: api_orders ``` In a downstream model: ```sql select * from {{ source('jaffle_shop', 'orders') }} ``` Will get compiled to: ```sql select * from raw.postgres_backend_public_schema.api_orders ``` --- ### What materializations are available in dbt? dbt ships with five built-in materializations: `view`, `table`, `incremental`, `ephemeral`, and `materialized_view`. Check out the documentation on [materializations](https://docs.getdbt.com/docs/build/materializations.md) for more information on each of these options. You can also create your own [custom materializations](https://docs.getdbt.com/guides/create-new-materializations.md). This is an advanced feature of dbt. --- ### What model configurations exist? You can also configure: * [tags](https://docs.getdbt.com/reference/resource-configs/tags.md) to support easy categorization and graph selection * [custom schemas](https://docs.getdbt.com/reference/resource-properties/schema.md) to split your models across multiple schemas * [aliases](https://docs.getdbt.com/reference/resource-configs/alias.md) if your view/table name should differ from the filename * Snippets of SQL to run at the start or end of a model, known as [hooks](https://docs.getdbt.com/docs/build/hooks-operations.md) * Warehouse-specific configurations for performance (e.g. `sort` and `dist` keys on Redshift, `partitions` on BigQuery) Check out the docs on [model configurations](https://docs.getdbt.com/reference/model-configs.md) to learn more. --- ### What parts of Jinja are dbt-specific? There are certain expressions that are specific to dbt — these are documented in the [Jinja function reference](https://docs.getdbt.com/reference/dbt-jinja-functions-context-variables.md) section of these docs. Further, docs blocks, snapshots, and materializations are custom Jinja *blocks* that exist only in dbt. --- ### What privileges does my database user need to use dbt? Your user will need to be able to: * `select` from raw data in your warehouse (i.e. data to be transformed) * `create` schemas, and therefore create tables/views within that schema¹ * read system views to generate documentation (i.e. views in `information_schema`) On Postgres, Redshift, Databricks, and Snowflake, use a series of `grants` to ensure that your user has the correct privileges. Check out [example permissions](https://docs.getdbt.com/reference/database-permissions/about-database-permissions.md) for these warehouses. On BigQuery, use the "BigQuery User" role to assign these privileges. *** ¹Alternatively, a separate user can create a schema for the dbt user, and then grant the user privileges to create within this schema. We generally recommend granting your dbt user the ability to create schemas, as it is less complicated to implement. --- ### What should I name my profile? We typically use a company name for a profile name, and then use targets to differentiate between `dev` and `prod`. Check out the docs on [environments in dbt v1](https://docs.getdbt.com/docs/local/dbt-environments.md) for more information. --- ### What should I name my target? We typically use targets to differentiate between development and production runs of dbt, naming the targets `dev` and `prod`, respectively. Check out the docs on [managing environments in dbt v1](https://docs.getdbt.com/docs/local/dbt-environments.md) for more information. --- ### What should my profiles.yml file look like for my warehouse? The structure of a profile looks different on each warehouse. Check out the [Supported Data Platforms](https://docs.getdbt.com/docs/supported-data-platforms.md) page, and navigate to the `Profile Setup` section for your warehouse. --- ### What version of Python can I use? Use this table to match dbt v1 versions with their compatible Python versions. New [dbt minor versions](https://docs.getdbt.com/docs/dbt-versions.md#minor-versions) will add support for new Python3 minor versions when all dependencies can support it. In addition, dbt minor versions will withdraw support for old Python3 minor versions before their [end of life](https://endoflife.date/python). #### Python compatibility matrix | dbt-core version | v1.12 | v1.11 | v1.10 | v1.9 | v1.8 | v1.7 | v1.6 | v1.5 | v1.4 | v1.3 | v1.2 | v1.1 | v1.0 | | ---------------- | ----- | ----- | ----- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | | Python 3.14 | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | Python 3.13 | ✅ | ✅ | ⚠️ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | Python 3.12 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | Python 3.11 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | | Python 3.10 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ⚠️ Python 3.13 is supported in dbt v1.10 for the Postgres adapter. Adapter plugins and their dependencies are not always compatible with the latest version of Python. Note that this shouldn't be confused with [dbt Python models](https://docs.getdbt.com/docs/build/python-models.md#specific-data-platforms). If you're using a data platform that supports Snowpark, use the `python_version` config to run a Snowpark model with [Python versions](https://docs.snowflake.com/en/developer-guide/snowpark/python/setup) 3.9, 3.10, or 3.11. --- ### When should I run my data tests? You should run your data tests whenever you are writing new code (to ensure you haven't broken any existing models by changing SQL), and whenever you run your transformations in production (to ensure that your assumptions about your source data are still valid). --- ### When should I use a UDF instead of a macro? Both user-defined functions (UDFs) and macros let you reuse logic across your dbt project, but they work in fundamentally different ways. Here's when to use each: ###### Use UDFs when:  You need logic accessible outside dbt UDFs are created in your warehouse and can be used by BI tools, data science notebooks, SQL clients, or any other tool that connects to your warehouse. Macros only work within dbt.  You want to standardize warehouse-native functions UDFs let you create reusable warehouse functions for data validation, custom formatting, or business-specific calculations that need to be consistent across all your data tools. Once created, they become part of your warehouse's function catalog.  You want dbt to manage the function lifecycle dbt manages UDFs as part of your DAG execution, ensuring they're created before models that reference them. You can version control UDF definitions alongside your models, test changes in development environments, and deploy them together through CI/CD pipelines.  Jinja compiles at creation time, not on each function call You can use Jinja (loops, conditionals, macros, `ref`, `source`, `var`) inside a UDF configuration. dbt resolves that Jinja **when the UDF is created**, and the resulting SQL body is what gets stored in your warehouse. Jinja influences the function when it’s created, whereas arguments influence it when it runs in the warehouse: * ✅ **Allowed:** Jinja that depends on project or build-time state — for example, `var(“can_do_things”)`, static `ref(‘orders’)`, or environment-specific logic. These are all evaluated once at creation time. * ❌ **Not allowed:** Jinja that depends on **function arguments** passed at runtime. The compiler can’t see those, so dynamic `ref(ref_name)` or conditional Jinja based on argument values won’t work.  You need Python logic that runs in your warehouse A Python UDF creates a Python function directly within your data warehouse, which you can invoke using SQL.<br />This makes it easier to apply complex transformations, calculations, or logic that would be difficult or verbose to express in SQL. Python UDFs support conditionals and looping within the function logic itself (using Python syntax), and execute at runtime, not at compile time like macros. Python UDFs are currently supported in Snowflake and BigQuery. ###### Use macros when:  You need to generate SQL at compile time Macros generate SQL dynamically **before** it's sent to the warehouse (at compile time). This is essential for: * Building different SQL for different warehouses * Generating repetitive SQL patterns (like creating dozens of similar columns) * Creating entire model definitions or DDL statements * Dynamically referencing models based on project structure UDFs execute **at query runtime** in the warehouse. While they can use Jinja templating in their definitions, they don't generate new SQL queries—they're pre-defined functions that get called by your SQL. Expanding UDFs Currently, SQL and Python UDFs are supported. Java and Scala UDFs are planned for future releases.  You want to generate DDL or DML statements Currently, SQL and Python UDFs are supported. Java and Scala UDFs are planned for future releases.  You need to adapt SQL across different warehouses Macros can use Jinja conditional logic to generate warehouse-specific SQL (see [cross-database macros](https://docs.getdbt.com/reference/dbt-jinja-functions/cross-database-macros.md)), making your dbt project portable across platforms. UDFs are warehouse-specific objects. Even though UDFs can include Jinja templating in their definitions, each warehouse has different syntax for creating functions, different supported data types, and different SQL dialects. You would need to define separate UDF files for each warehouse you support.  Your logic needs access to dbt context Both macros and UDFs can use Jinja, which means they can access dbt context variables like `{{ ref() }},` `{{ source() }}`, environment variables, and project configurations. You can even call a macro from within a UDF (and vice versa) to combine dynamic SQL generation with runtime execution. However, the difference between the two is *when* the logic runs: * Macros run at compile time, generating SQL before it’s sent to the warehouse. * UDFs run inside the warehouse at query time.  You want to avoid creating warehouse objects Macros don't create anything in your warehouse; they just generate SQL at compile time. UDFs create actual function objects in your warehouse that need to be managed. ###### Can I use both together? Yes! You can use a macro to call a UDF or call a macro from within a UDF, combining the benefits of both. So the following example shows how to use a macro to define default values for arguments alongside your logic, for your UDF ```sql {% macro cents_to_dollars(column_name, scale=2) %} {{ function('cents_to_dollars') }}({{ column_name }}, {{scale}}) {% endmacro %} ``` ###### Related documentation * [User-defined functions](https://docs.getdbt.com/docs/build/udfs.md) * [Jinja macros](https://docs.getdbt.com/docs/build/jinja-macros.md) --- ### Where can I find my user ID? Knowing your dbt user ID can help with interacting with support. To find your user ID in the dbt platform, read the following steps: 1. Click your account name at the bottom left-side menu and go to **Account settings** > **Users**. 2. Select your user. <br /> 3. Go to the address bar. The number after `/users` is your user ID. <br /> For example, if the URL is `https://YOUR_ACCESS_URL/settings/accounts/12345/users/67891` — the user ID is `67891`. <br /> 4. Copy that number and save it somewhere safe. <br /> --- ### Where can I find SCIM FAQs and troubleshooting? System for Cross-domain Identity Management (SCIM) lets you automatically provision, update, and deprovision dbt platform users and groups directly from your identity provider (IdP). Instead of managing users manually in dbt platform, your IdP becomes the source of truth. SCIM is available on [Enterprise plans](https://www.getdbt.com/pricing). For common questions and troubleshooting guidance, refer to the [SCIM FAQ and troubleshooting](https://docs.getdbt.com/docs/platform/manage-access/scim-faq.md) page. --- ### Where can I find SSO FAQs and troubleshooting? Single sign-on (SSO) lets your team log into dbt platform using your existing identity provider (IdP) — such as Okta, Microsoft Entra ID, or Google Workspace \&mdsah; without needing a separate dbt platform password. SSO is available on [Enterprise tiered plans](https://www.getdbt.com/pricing). For common questions and troubleshooting guidance, refer to the [SSO FAQs and troubleshooting](https://docs.getdbt.com/docs/platform/manage-access/sso-faq.md) page. ddd --- ### Where does the metadata about last updated timestamp come from? Last updated timestamps come directly from the data warehouse, for example from `INFORMATION_SCHEMA` tables. --- ### Which docs should I use when writing Jinja or creating a macro? If you are stuck with a Jinja issue, it can get confusing where to check for more information. We recommend you check (in order): 1. [Jinja's Template Designer Docs](https://jinja.palletsprojects.com/page/templates/): This is the best reference for most of the Jinja you'll use 2. [Our Jinja function reference](https://docs.getdbt.com/reference/dbt-jinja-functions-context-variables.md): This documents any additional functionality we've added to Jinja in dbt. 3. [Agate's table docs](https://agate.readthedocs.io/page/api/table.html): If you're operating on the result of a query, dbt will pass it back to you as an agate table. This means that the methods you call on the table belong to the Agate library rather than Jinja or dbt. --- ### Which materialization should I use for my model? Start out with views, and then change models to tables when required for performance reasons (i.e. downstream queries have slowed). Check out the [docs on materializations](https://docs.getdbt.com/docs/build/materializations.md) for advice on when to use each materialization. --- ### Which SQL dialect should I write my models in? Or which SQL dialect does dbt use? dbt can feel like magic, but it isn't actually magic. Under the hood, it's running SQL in your own warehouse — your data is not processed outside of your warehouse. As such, your models should just use the **SQL dialect of your own database**. Then, when dbt wraps your `select` statements in the appropriate DDL or DML, it will use the correct DML for your warehouse — all of this logic is written in to dbt. You can find more information about the databases, platforms, and query engines that dbt supports in the [Supported Data Platforms](https://docs.getdbt.com/docs/supported-data-platforms.md) docs. Want to go a little deeper on how this works? Consider a snippet of SQL that works on each warehouse: models/test\_model.sql ```sql select 1 as my_column ``` To replace an existing table, here's an *illustrative* example of the SQL dbt will run on different warehouses (the actual SQL can get much more complicated than this!) ##### Redshift ```sql -- you can't create or replace on redshift, so use a transaction to do this in an atomic way begin; create table "dbt_alice"."test_model__dbt_tmp" as ( select 1 as my_column ); alter table "dbt_alice"."test_model" rename to "test_model__dbt_backup"; alter table "dbt_alice"."test_model__dbt_tmp" rename to "test_model" commit; begin; drop table if exists "dbt_alice"."test_model__dbt_backup" cascade; commit; ``` ##### BigQuery ```sql -- Make an API call to create a dataset (no DDL interface for this)!!; create or replace table `dbt-dev-87681`.`dbt_alice`.`test_model` as ( select 1 as my_column ); ``` ##### Snowflake ```sql create schema if not exists analytics.dbt_alice; create or replace table analytics.dbt_alice.test_model as ( select 1 as my_column ); ``` --- ### Why am I getting an "account in use" error? If you're receiving an 'Account in use' error when trying to integrate GitHub in your Profile page, this is because the Git integration is a 1-to-1 integration, so you can only have your Git account linked to one dbt user account. Here are some steps to take to get you unstuck: * Log in to the dbt account integrated with your Git account. Go to your user profile and click on Integrations to remove the link. If you don't remember which dbt account is integrated, please email dbt Support at <support@getdbt.com> and we'll do our best to disassociate the integration for you. --- ### Why am I receiving a Runtime Error in my packages? If you're receiving the runtime error below in your packages.yml folder, it may be due to an old version of your dbt\_utils package that isn't compatible with your current dbt version. ```shell Running with dbt=xxx Runtime Error Failed to read package: Runtime Error Invalid config version: 1, expected 2 Error encountered in dbt_utils/dbt_project.yml ``` Try updating the old version of the dbt\_utils package in your packages.yml to the latest version found in the [dbt hub](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/): ```shell packages: - package: dbt-labs/dbt_utils version: xxx ``` If you've tried the workaround above and are still experiencing this behavior - reach out to the Support team at <support@getdbt.com> and we'll be happy to help! --- ### Why are profiles stored outside of my project? Profiles are stored separately to dbt projects to avoid checking credentials into version control. Database credentials are extremely sensitive information and should **never be checked into version control**. --- ### Why can't I just write DML in my transformations? ###### `select` statements make transformations accessible More people know how to write `select` statements, than DML, making the transformation layer accessible to more people! ###### Writing good DML is hard If you write the DDL / DML yourself you can end up getting yourself tangled in problems like: * What happens if the table already exists? Or this table already exists as a view, but now I want it to be a table? * What if the schema already exists? Or, should I check if the schema already exists? * How do I replace a model atomically (such that there's no down-time for someone querying the table) * What if I want to parameterize my schema so I can run these transformations in a development environment? * What order do I need to run these statements in? If I run a `cascade` does it break other things? Each of these problems *can* be solved, but they are unlikely to be the best use of your time. ###### dbt does more than generate SQL You can test your models, generate documentation, create snapshots, and more! ###### You reduce your vendor lock in SQL dialects tend to diverge the most in DML and DDL (rather than in `select` statements) — check out the example [here](https://docs.getdbt.com/faqs/Models/sql-dialect.md). By writing less SQL, it can make a migration to a new database technology easier. If you do need to write custom DML, there are ways to do this in dbt using [custom materializations](https://docs.getdbt.com/guides/create-new-materializations.md). --- ### Why dbt compile needs a data platform connection `dbt compile` needs a data platform connection in order to gather the info it needs (including from introspective queries) to prepare the SQL for every model in your project. ##### dbt compile The [`dbt compile` command](https://docs.getdbt.com/reference/commands/compile.md) generates executable SQL from `source`, `model`, `test`, and `analysis` files. `dbt compile` is similar to `dbt run` except that it doesn't materialize the model's compiled SQL into an existing table. So, up until the point of materialization, `dbt compile` and `dbt run` are similar because they both require a data platform connection, run queries, and have an [`execute` variable](https://docs.getdbt.com/reference/dbt-jinja-functions/execute.md) set to `True`. However, here are some things to consider: * You don't need to execute `dbt compile` before `dbt run` * In dbt, `compile` doesn't mean `parse`. This is because `parse` validates your written `YAML`, configured tags, and so on. ##### Introspective queries To generate the compiled SQL for many models, dbt needs to run introspective queries, (which is when dbt needs to run SQL in order to pull data back and do something with it) against the data platform. These introspective queries include: * Populating the relation cache. For more information, refer to the [Create new materializations](https://docs.getdbt.com/guides/create-new-materializations.md) guide. Caching speeds up the metadata checks, including whether an [incremental model](https://docs.getdbt.com/docs/build/incremental-models.md) already exists in the data platform. * Resolving [macros](https://docs.getdbt.com/docs/build/jinja-macros.md#macros), such as `run_query` or `dbt_utils.get_column_values` that you're using to template out your SQL. This is because dbt needs to run those queries during model SQL compilation. * [`dbt docs generate`](https://docs.getdbt.com/reference/commands/cmd-docs.md) compiles your project by default (unless you pass [`--no-compile`](https://docs.getdbt.com/reference/commands/cmd-docs.md)), so introspective macros such as [`run_query`](https://docs.getdbt.com/reference/dbt-jinja-functions/run_query.md) run against the warehouse during documentation builds the same way they do during other compile workflows. Refer to [`run_query`](https://docs.getdbt.com/reference/dbt-jinja-functions/run_query.md) for how that works and for using `flags.WHICH` when you want to limitDML or other side-effecting SQL to specific dbt commands. Without a data platform connection, dbt can't perform these introspective queries and won't be able to generate the compiled SQL needed for the next steps in the dbt workflow. You can [`parse`](https://docs.getdbt.com/reference/commands/parse.md) a project and use the [`list`](https://docs.getdbt.com/reference/commands/list.md) resources in the project, without an internet or data platform connection. Parsing a project is enough to produce a [manifest](https://docs.getdbt.com/reference/artifacts/manifest-json.md), however, keep in mind that the written-out manifest won't include compiled SQL. To configure a project, you do need a [connection profile](https://docs.getdbt.com/docs/local/profiles.yml.md) (`profiles.yml` if using the CLI). You need this file because the project's configuration depends on its contents. For example, you may need to use [`{{target}}`](https://docs.getdbt.com/reference/dbt-jinja-functions/target.md) for conditional configs or know what platform you're running against so that you can choose the right flavor of SQL. --- ### Why do I need to quote column names in Jinja? In the [macro example](https://docs.getdbt.com/docs/build/jinja-macros.md#macros) we passed the column name `amount` quotes: ```sql {{ cents_to_dollars('amount') }} as amount_usd ``` We have to use quotes to pass the *string* `'amount'` to the macro. Without the quotes, the Jinja parser will look for a variable named `amount`. Since this doesn't exist, it will compile to nothing. Quoting in Jinja can take a while to get used to! The rule is that you're within a Jinja expression or statement (i.e. within `{% ... %}` or `{{ ... }}`), you'll need to use quotes for any arguments that are strings. Single and double quotes are equivalent in Jinja – just make sure you match them appropriately. And if you do need to pass a variable as an argument, make sure you [don't nest your curlies](https://docs.getdbt.com/best-practices/dont-nest-your-curlies.md). --- ### Why do model and source YAML files always start with `version: 2`? Once upon a time, the structure of these `.yml` files was very different (s/o to anyone who was using dbt back then!). Adding `version: 2` allowed us to make this structure more extensible. From [dbt v1.5](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.5.md#quick-hits>), the top-level `version:` key is optional in all resource YAML files. If present, only `version: 2` is supported. Also starting in v1.5, both the [`config-version: 2`](https://docs.getdbt.com/reference/project-configs/config-version.md) and the top-level `version:` key in the `dbt_project.yml` are optional. Resource YAML files do not currently require this config. We only support `version: 2` if it's specified. Although we do not expect to update YAML files to `version: 3` soon, having this config will make it easier for us to introduce new structures in the future --- ### Why does my dbt output have so many macros in it? The output of a dbt run counts over 100 macros in your project! ```shell $ dbt run Running with dbt=1.7.0 Found 1 model, 0 tests, 0 snapshots, 0 analyses, 138 macros, 0 operations, 0 seed files, 0 sources ``` This is because dbt ships with its own project, which also includes macros! You can learn more about this [here](https://discourse.getdbt.com/t/did-you-know-dbt-ships-with-its-own-project/764). --- ### Why does the BigQuery OAuth application require scopes to Google Drive? BigQuery supports external tables over both personal Google Drive files and shared files. For more information, refer to [Create Google Drive external tables](https://cloud.google.com/bigquery/docs/external-data-drive). --- ### Why doesn’t an indirectly referenced upstream public model appear in Explorer? For [project dependencies](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md) in Mesh, [Catalog](https://docs.getdbt.com/docs/explore/explore-multiple-projects.md) only displays directly referenced [public models](https://docs.getdbt.com/docs/mesh/govern/model-access.md) from upstream projects, even if an upstream model indirectly depends on another public model. So for example, if: * `project_b` adds `project_a` as a dependency * `project_b`'s model `downstream_c` references `project_a.upstream_b` * `project_a.upstream_b` references another public model, `project_a.upstream_a` Then: * In Explorer, only directly referenced public models (`upstream_b` in this case) appear. * In the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) lineage view, however, `upstream_a` (the indirect dependency) *will* appear because dbt dynamically resolves the full dependency graph. This behavior makes sure that Catalog only shows the immediate dependencies available to that specific project. --- ### Why is my model being rebuilt instead of reused? dbt State decides whether to reuse a model by parsing the rendered SQL into a syntax tree and comparing the hash. If the hash has changed (implying the model's logic has changed), dbt State rebuilds the model. dbt State prioritizes safety and precision; if it can't guarantee skipping a node is safe, then it rebuilds the node to be sure. A few patterns that commonly cause overeager rebuilds are listed on this page, along with recommendations to increase reuse rate. The following patterns commonly cause unexpected rebuilds: * [Views with `select *`](#views-with-select) * [Non-deterministic Jinja templating](#non-deterministic-jinja-templating) * [Models with external sources in BigQuery](#models-with-external-sources-in-bigquery) * [Models with custom materializations](#models-with-custom-materializations) #### Views with `select *` dbt State reuses a model when its compiled SQL matches the stored hash. When a view uses `select *` directly on a `ref()` or `source()`, dbt can't determine the column list at parse time — the upstream model or source table might have gained or lost columns since the last run. To be safe, dbt State forces a rebuild. For example, this view will be rebuilt even if `stg_orders` hasn't changed because dbt can't know at parse time whether `stg_orders` has the same columns as before: ```sql -- stg_orders_view.sql (materialized: view) select * from {{ ref('stg_orders') }} ``` However, if you use `select *` on a CTE, dbt can resolve the columns from the CTE definition and safely reuse the view: ```sql with renamed as ( select order_id, customer_id, order_total from {{ ref('stg_orders') }} ) select * from renamed ``` If a CTE explicitly names its columns, a `select *` that reads from that CTE won't force a rebuild even if an earlier CTE used `select *` on a `ref()` or `source()`. The typical staging pattern is reused: ```sql with source as ( select * from {{ source('jaffle_shop', 'orders') }} ), renamed as ( select id as order_id, user_id as customer_id, amount as order_total from source ) select * from renamed ``` tip To avoid forced rebuilds, use explicit column names when selecting directly from a `ref()` or `source()`. You can also exclude views from execution using `--exclude config.materialized:view`. #### Non-deterministic Jinja templating Some macros and environment variables can cause unexpected rebuilds. For example, `dbt_utils.get_relations_by_pattern` (an introspective macro) combined with `dbt_utils.union_relations` can return relations in a different order on each run, producing different rendered SQL even when your project logic hasn't changed. Similarly, environment variables that change between runs produce different rendered SQL on every run: ```sql select '{{ env_var("AIRFLOW_RUN_ID") }}' as airflow_run_id, ... ``` Because the query result order or the environment variable's value changes, the rendered SQL differs from the stored hash on every run. dbt State treats this as a code change and rebuilds the model, even though the underlying project logic hasn't changed. This pattern can affect any model type, not just views; if a base or staging model rebuilds on every run, all of its downstream models rebuild, too. To avoid these unnecessary rebuilds, enable [`compare_unrendered_code`](https://docs.getdbt.com/reference/resource-configs/compare-unrendered-code.md). When enabled, dbt State checks both the Jinja template and rendered SQL; non-deterministic values that don't change the template don't trigger a rebuild. For example: ```sql {{ config(state={"compare_unrendered_code": true}) }} select '{{ env_var("AIRFLOW_RUN_ID") }}' as airflow_run_id, ... ``` #### Models with external sources on BigQuery On BigQuery, models that use external sources (such as Google Sheets) always rebuild because BigQuery doesn't expose modification timestamps for external sources, so dbt State can't determine freshness. tip To prevent external sources from always being considered stale, configure [`loaded_at_field`](https://docs.getdbt.com/reference/resource-properties/freshness.md#loaded_at_field) or [`loaded_at_query`](https://docs.getdbt.com/reference/resource-properties/freshness.md#loaded_at_query) in your source definition to point to a timestamp field. This lets dbt State query a timestamp field directly to determine freshness, rather than relying on warehouse metadata. #### Models with custom materializations Models using custom materializations are always built and are never reused. Custom materializations may have side effects (for example, modifying table properties or writing to other schemas), and dbt State cannot safely determine whether skipping the run would produce the same result. #### How to diagnose After a run, use (Applies to dbt v2.0 and later) [`dbt state explain`](https://docs.getdbt.com/reference/commands/state-explain.md) to see why dbt State rebuilt, reused, or cloned a specific model. For a detailed breakdown, use the `--verbose` flag with `-s` to select your model: note The command name differs by version: dbt v2 uses `dbt state explain` (with a space), while dbt v1 uses `dbt-state explain` (with a hyphen). (Applies to dbt v2.0 and later) ```bash dbt state explain --verbose -s my_model_name ``` If you use the dbt platform, the same information is available without running a command — go to the [**Explain** tab](https://docs.getdbt.com/docs/deploy/dbt-state-interface.md#explain-tab) on the job run details page to see the full decision breakdown for each node. --- ### Why is Run on Pull request grayed out? If you're unable to enable Run on Pull requests, you'll want to make sure your existing repo was not added via the Deploy Key auth method. If it was added via a deploy key method, you'll want to use the [GitHub auth method](https://docs.getdbt.com/docs/platform/git/connect-github.md) to enable CI in dbt. To go ahead and enable 'Run on Pull requests', you'll want to remove dbt from the Apps & Integration on GitHub and re-integrate it again via the GitHub app method. If you've tried the workaround above and are still experiencing this behavior - reach out to the Support team at <support@getdbt.com> and we'll be happy to help! --- ### Why might my actual warehouse costs differ from displayed costs? Cost Insights shows estimates based on warehouse-reported usage and your configured pricing variables. These estimates are based on a retroactive analysis of historical runs and reflect actual usage, *not* forecasts of future costs. Adjustments and differences may occur if: * Your warehouse has custom pricing that differs from the default compute credit unit. * There are discounts or credits applied at the billing level that aren't reflected in usage tables. * Costs include other charges beyond compute. Costs Insights in the dbt platform is designed to be directionally accurate, showing you dbt-specific components rather than matching your billing exactly. --- ### Why would I want to impersonate a service account? You may want your models to be built using a dedicated service account that has elevated access to read or write data to the specified project or dataset. Typically, this requires you to create a service account key for running under development or on your CI server. By specifying the email address of the service account you want to build models as, you can use [Application Default Credentials](https://cloud.google.com/sdk/gcloud/reference/auth/application-default) or the service's configured service account (when running in GCP) to assume the identity of the service account with elevated permissions. This allows you to reap the advantages of using federated identity for developers (via ADC) without needing to grant individual access to read and write data directly, and without needing to create separate service account and keys for each user. It also allows you to completely eliminate the need for service account keys in CI as long as your CI is running on GCP (Cloud Build, Jenkins, GitLab/Github Runners, etc). --- ## Fusion ### Arrow ADBC and dbt v2 This document provides technical guidance for dbt partners and vendors on how to design, build, and maintain ADBC (Apache Arrow Database Connectivity) drivers for dbt v2, the new dbt engine. dbt v2 leverages ADBC as a unified driver layer for seamless, high-performance integration with data platforms. Building an ADBC driver is the first step to connecting dbt v2 with a new platform. #### Why dbt v2 uses ADBC dbt v2 represents a major evolution in the dbt engine with minimal changes to the authoring layer. Built in Rust, dbt v2 delivers speed, language understanding, and seamless integration with numerous data warehouses. A key aspect of the new engine is its adoption of ADBC — a modern, open standard from the Apache Arrow project that simplifies columnar data interchange across platforms. Historically, dbt v1 adapters required bespoke connection logic for each data platform. dbt v2 improves on this model with a unified ADBC driver layer that offers several key advantages: * **Standardization**: ADBC standardizes common platform features across a single interface. * **Performance**: Drivers leverage Arrow's columnar memory format for efficient query execution with minimal transformations. * **Maintainability**: ADBC drivers follow a shared specification, reducing the complexity of implementing new adapters. #### Technical overview This technical specification covers the ADBC specification. The specification maintains backwards compatibility, so guidance here remains valid as the spec evolves. For the latest information and detailed documentation, refer to the [ADBC documentation](https://arrow.apache.org/adbc/current/), which is the source of truth. The ADBC API provides a powerful array of features, but you don't need to implement all of them. This section covers the API surface required for dbt v2 compatibility. ##### Programming language **tl;dr: Use Go.** One distinct advantage of Arrow ADBC is portability. You can write drivers in various languages and load them via driver managers. This portability allows dbt v2 (written in Rust) to leverage drivers written in other languages. For dbt v2 compatibility, drivers must: * Compile into shared libraries that can be loaded from any program * Produce a platform-specific, standalone binary We recommend **Go** as the language of choice, though Rust or C++ also work. Go has a runtime and garbage collector, but it's engineered to compile into well-behaved shared libraries—unlike languages like C# or Java. A standalone binary allows users to download and run the driver out of the box without setting up an interpreter. Compiled languages like Go also enable dbt v2 and its drivers to share memory directly over FFI without external dependencies. ##### ADBC specifications This section covers the minimum requirements for a dbt v2-compatible ADBC driver. For complete details on the ADBC specification and driver development, refer to the [ADBC driver authoring guide](https://arrow.apache.org/adbc/current/driver/authoring.html). Drivers consist of several key abstractions: 1. **Driver**: Load a driver to create databases. 2. **Database**: Create databases and set configuration options (including authentication). Use databases to open connections. 3. **Connection**: Establish connections to the warehouse. Connections create statements. 4. **Statement**: Set options and SQL queries on statements, then execute them against the warehouse. dbt v2 achieves high performance through aggressive parallelism, so expect many simultaneous connections during project execution. ##### Authentication Drivers handle authentication through key-value options set on the database. dbt v2 translates options from user-authored `profiles.yml` files before passing them to the driver. For example, what dbt calls `client_secret` in a Snowflake profile gets set on the driver as `adbc.snowflake.sql.client_option.client_secret`. For a complete example of how dbt v2 translates profile options, see the [Snowflake authentication source code](https://github.com/dbt-labs/dbt-fusion/blob/main/crates/dbt-auth/src/snowflake/mod.rs). For more information on profile configuration, refer to [dbt profiles](https://docs.getdbt.com/docs/local/profiles.yml.md). ###### Credential caching Simple authentication methods (like username/password stored in `profiles.yml`) support fully parallel connection creation with no special handling required. For authentication methods that require browser interaction (user-to-machine OAuth, SSO, or MFA), implement credential caching. Due to dbt v2's highly parallel execution, without caching, every new connection prompts the user for authentication repeatedly. Your credential cache for browser-based authentication must: * Block new connections until an initial connection establishes and stores a token in memory (avoiding the thundering herd problem). * Handle token refresh using the same blocking principle when invalidation occurs. * Use interprocess, file-system-based storage to support the LSP, which runs in a separate process. This caching is critical for any browser-based or MFA authentication option, but is not needed for simple credential-based authentication. #### Required APIs This section covers the minimum API set for dbt v2 compatibility. The requirements are: * Authentication via options. * SQL query execution. * Metadata queries to understand remote warehouse state (certain connection-level metadata functions can use cheaper or more performant APIs to pull table schemas). These requirements are not exhaustive. dbt Labs encourages implementing the full ADBC specification to benefit both dbt v2 and the broader ADBC community. ###### Driver | Method | Description | | ------------- | ------------------------------- | | `NewDatabase` | Create a new database instance. | ###### Database | Method | Description | | ------------ | ----------------------------------------------------- | | `SetOptions` | Set configuration options (including authentication). | | `Open` | Open a connection. | | `Close` | Close the database. | ###### Connection | Method | Description | | ---------------- | --------------------------------------- | | `GetObjects` | Pull metadata from the warehouse. | | `GetTableSchema` | Pull schema metadata for tables. | | `NewStatement` | Create a statement for query execution. | | `Close` | Close the connection. | ###### Statement | Method | Description | | --------------- | ----------------------------------- | | `SetOption` | Set options on queries. | | `SetSqlQuery` | Set the SQL query text. | | `ExecuteQuery` | Execute a query and return results. | | `ExecuteUpdate` | Execute DML queries. | | `Close` | Close the statement. | --- ### Compare changes during development Beta ### Compare changes during development [Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") Available in v2 | Local development | Login required info This feature is in beta. All dbt VS Code extension users can use compare changes during the 14-day trial. After the trial, [sign in or register](https://docs.getdbt.com/docs/sign-in-dbt-extension.md) for a dbt platform account to keep using it. The dbt VS Code extension previews and compares how your local edits affect your data in your dbt platform account — including added/removed rows and join verification — without waiting on CI. Use compare changes to check impact early and validate changes before you open a PR or run a [CI job](https://docs.getdbt.com/docs/deploy/ci-jobs.md): * Validate outputs are correct when refactoring logic, adding or removing columns, or implementing join modifications. * It compares your current working copy against your `manifest.json` (for example, your last production state) and shows changes to primary keys, rows, and columns in the **Compare** tab. The dbt VS Code extension's compare changes feature is different from the [Advanced CI compare changes feature](#how-this-differs-from-advanced-ci), which runs at the PR stage in deployment rather than locally during development. info Compare changes in development is available for models only. Support for seeds, snapshots, ephemeral models, and Python models is coming soon. [![Make changes to a model and see the changes in the Compare tab](/img/docs/extension/vs-compare-changes.png?v=2 "Make changes to a model and see the changes in the Compare tab")](#)Make changes to a model and see the changes in the Compare tab #### Prerequisites To use the dbt VS Code extension compare changes feature, you need: * A dbt v2 [supported data platform](https://docs.getdbt.com/docs/dbt/supported-features.md?version=2.0#requirements) (BigQuery, Databricks, Redshift, or Snowflake) * The [dbt VS Code extension](https://docs.getdbt.com/docs/install-dbt-extension.md) installed with a self-hosted installation of the [dbt v2](https://docs.getdbt.com/docs/dbt/get-started-dbt.md) * A baseline state to compare your changes against — refer to [How it works](#how-it-works) to choose between [automatic deferral](https://docs.getdbt.com/docs/platform/about-defer.md) or [`manifest.json`](https://docs.getdbt.com/reference/artifacts/manifest-json.md?version=2.0) manual setup. **Sign-in and authentication** Compare changes is available to all dbt VS Code extension users during the 14-day trial. After the trial: * **If you use dbt platform deferral (recommended):** [Sign in or register](https://docs.getdbt.com/docs/sign-in-dbt-extension.md) for a free dbt platform account. You'll also need a [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md) file in your local `.dbt` directory (`~/.dbt/dbt_cloud.yml` on macOS/Linux) so the extension can fetch the deferred manifest from your environment. * **If you set the baseline manually with a `manifest.json`:** No dbt platform account is required. Point the extension at a local `manifest.json` and compare changes runs against it. #### How it works Compare changes in development works by comparing two materialized models in your warehouse. Specifically, it compares the model built in your dev schema (determined by your active profile) against the model referenced in your `manifest.json` (for example, your last production state). Both sides of the comparison are always warehouse tables; it does not compare SQL file contents. The path you choose determines whether you need a dbt platform account: * **dbt platform deferral (recommended):** Requires a dbt platform account and at least one successful job run in the environment you're deferring to (usually staging or production). dbt v2 auto-downloads the deferred manifest and uses it as your baseline state. * **Manual `state` directory:** Point the extension at a local `manifest.json` (for example, copied from another environment). No dbt platform account is required; no job run is required. <br /> #### Use compare changes To use compare changes in development, follow these steps: 1. Open a SQL model file in your editor. 2. Make some changes to the model, like adding a new column or modifying an existing one (for example, `left_join_customers` to `right_join_customers`). 3. Run compare changes in the [VS Code interface](https://code.visualstudio.com/docs/getstarted/userinterface#_command-palette): * Command palette: Open the VS Code command palette (Shift + Command + P (Mac) / Ctrl + Shift + P (Windows/Linux)) and search for the [**dbt: Compare changes**](vscode://dbtLabsInc/dbt.compareModel) * Bottom panel: Click the **Compare** tab and then click the **Compare** button. [![Compare changes in development](/img/docs/extension/vs-compare-changes-options.png?v=2 "Compare changes in development")](#)Compare changes in development 4. Once you click the **Compare** button, the extension will execute a `dbt build` command to build the model you're working on and then runs the comparison. The [**Compare** tab](#compare-tab) displays the changes to the data's primary keys, rows, and columns. Clicking the tabs will display more details about the changes, like specific columns that were added or modified. 5. Once you've compared changes and see the changes in the **Compare** tab, you can then decide to commit your changes or continue editing. #### Compare tab results The **Compare** tab displays the changes to the data's primary keys, rows, and columns. Clicking the tabs will display more details about the changes, like specific columns that were added or modified. [![Example of the Compare tab](/img/docs/extension/vs-compare-changes-tab.png?v=2 "Example of the Compare tab")](#)Example of the Compare tab * **Overview tab**: High-level summary about the changes to the models, such as the number of primary keys that were added or removed, rows modified, and so on. It will also include the relation between models that were added or modified. * **Primary keys tab**: Details about the changes to the records. * **Modified rows tab**: Details about the modified rows. * **Columns tab**: Details about the changes to the columns. #### FAQs  Are queries run on behalf of the developer? Yes. All comparison queries in development run using your development credentials or platform-based user credentials, directly from the dbt VS Code extension. If you authenticated with a [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md) file, these are the credentials configured in your **Account settings**; otherwise it uses the credentials from your active dbt profile. dbt v2 uses your credentials to execute comparison queries in your warehouse. The results are stored in memory, so that we can keep them populated into the Compare tab for that file until you re-run.  Is this using my warehouse credits? Yes. Because the comparison runs in your development environment using your dev credentials, it will use your warehouse’s compute.  Do I need to run dbt build every time I make a change? No. When you click **Compare**, the extension builds the model into your development schema automatically, so you don’t need to run `dbt build` yourself. It then compares that development build against the version in your deferred environment (usually staging or production). If the model hasn’t been built yet in your deferred environment, the comparison can’t run. For more details, see [Defer to production](https://docs.getdbt.com/docs/platform/about-defer.md).  How is this different from Advanced CI compare changes? The dbt VS Code extension's compare changes feature applies only to your self-hosted dbt development environment. If you're looking to compare changes between your production environment and the pull request's latest commit, check out [Advanced CI compare changes](https://docs.getdbt.com/docs/deploy/advanced-ci.md#compare-changes). | Aspect | In development (compare changes) | In deployment (Advanced CI) | | ------------------- | ----------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | | **Affects** | Development for one modified model at a time | Deployment for all modified models in a project | | **Trigger** | On-demand in editor | PR open/update and CI job | | **Scope** | Your working copy and local target | Branch head versus prod state in CI | | **Output** location | Compare panel in VS Code/Cursor. Does not create a PR comment in Git provider | Deployment job compare tab and PR summary comment in Git provider | | **Data caching** | Editor-side | dbt platform [caches](https://docs.getdbt.com/docs/deploy/advanced-ci.md#about-the-cached-data) limited samples | | **Governance** | Local development credentials | Production credentials | #### Related docs * [Advanced CI compare changes](https://docs.getdbt.com/docs/deploy/advanced-ci.md#compare-changes) --- ### dbt v2 readiness checklist dbt v2 is here and is now generally available for dbt platform projects on Snowflake! We currently offer it as a [preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles.md) for all other supported adapters. Even if we haven't enabled it for your account, you can still start preparing your projects for upgrade. Use this checklist to ensure a smooth upgrade once dbt v2 becomes available. If this is all new to you, first [learn about dbt v2](https://docs.getdbt.com/docs/introduction.md), its current state, and the features available. Enable the dbt v2 readiness panel The dbt v2 readiness panel shows each project's eligibility status and blockers in the dbt platform. It's rolling out in phases — if it's not enabled for your account yet, an [account admin](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md#account-admin) can turn it on in **Account settings** → **Account**. Refer to [Enable dbt v2 readiness features](https://docs.getdbt.com/guides/prepare-v2-upgrade.md?step=2) for setup steps. If you have access to dbt Wizard, use the [dbt Wizard's dbt v2 migration workflow](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md#fusion-migration-workflow) to help you fix compatibility errors directly from the Studio IDE using dbt Wizard — no manual log investigation needed! #### Preparing for dbt v2 Use the following checklist to prepare your projects for dbt v2 For walkthroughs of both the preparation and upgrade processes, check out our detailed guides: * \[ ] [Upgrade to dbt v2 Pt. 1: Preparing to upgrade](https://docs.getdbt.com/guides/prepare-v2-upgrade.md?step=1) * \[ ] [Upgrade to dbt v2 Pt. 2: Making the move](https://docs.getdbt.com/guides/upgrade-to-v2.md?step=1) ##### Upgrade to the latest dbt version (recommended) The **v1 Latest** [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) has all of the most recent features to help you prepare for dbt v2. * \[ ] Make sure all your projects are on the **v1 Latest** release track across all deployment environments and jobs. This is not a strict requirement for upgrading, but it will ensure the simplest, most predictable experience by allowing you to pre-validate that your project doesn't rely on deprecated behaviors. ##### Resolve all deprecation warnings You must resolve deprecations while your projects are on a dbt v1 release track, as they result in warnings that will become errors once you upgrade to dbt v2. The autofix tool can automatically resolve many deprecations (such as moving arbitrary configs into the meta dictionary). For a full list of deprecations and how to resolve them, refer to [Deprecations](https://docs.getdbt.com/reference/deprecations.md). Start a new branch to begin resolving deprecation warnings using one of the following methods: * \[ ] **Run autofix in the dbt platform:** You can address deprecation warnings using the [autofix tool in the Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/autofix-deprecations.md). You can run the autofix tool on the **v1 Compatible** or **v1 Latest** release track. * \[ ] **Run autofix locally:** Use the [VS Code extension](https://docs.getdbt.com/docs/about-dbt-extension.md). The extension has a built-in ["Getting Started" workflow](https://docs.getdbt.com/docs/install-dbt-extension.md#getting-started) that will debug your dbt project in the VS Code or Cursor IDE and execute the autofix tool. This has the added benefit of installing dbt v2 to your computer so you can begin testing locally before implementing in your dbt platform account. * \[ ] **Run autofix locally (without the extension):** Visit the autofix [GitHub repo](https://github.com/dbt-labs/dbt-autofix) to run the tool locally if you're not using VS Code or Cursor. This will only run the tool, it will not install dbt v2. * \[ ] **Remove behavior change flag overrides:** dbt v2 forcibly enables all behavior change flags. Remove any `flags:` overrides in your `dbt_project.yml` that opt out of these behaviors and validate that your project works correctly with them enabled. ##### Upgrade YAML spec * \[ ] **Migrate Semantic Layer configs:** If your project uses the Semantic Layer, make sure your metric configurations use the [latest YAML spec](https://docs.getdbt.com/docs/build/latest-metrics-spec.md). ##### Validate and upgrade your dbt packages The most commonly used dbt Labs managed packages (such as `dbt_utils` and `dbt_project_evaluator`) are already compatible with dbt v2, as are a large number of external and community packages. Review [the dbt package hub](https://hub.getdbt.com) to see verified dbt v2-compatible packages by checking that the `require-dbt-version` configuration includes `2.0.0` or higher. Refer to [package support](https://docs.getdbt.com/docs/dbt/supported-features.md#package-support) for more information. * \[ ] Make sure that all of your packages are upgraded to the most recent version, many of which contain enhancements to support dbt v2. * \[ ] Check package repositories to make sure they're compatible with dbt v2. If a package you use is not yet compatible, we recommend opening an issue with the maintainer, making the contribution yourself, or removing the package temporarily before you upgrade. ##### Validate user-defined functions Check that dbt v2 supports all user-defined functions (UDFs) in your project. dbt v2 supports nearly all built-in data platform functions out of the box. However, data platforms continuously add new functions that dbt v2 may not yet support. If you see the error `dbt0209: No function <function name>`, you can resolve it depending on whether the function is a UDF or a built-in function: * \[ ] **For custom UDFs:** Recreate it as a [native dbt UDF](https://docs.getdbt.com/docs/build/udfs.md#defining-udfs-in-dbt) to get the full dbt v2 experience. With `static_analysis: baseline` (the default), most UDFs will work out of the box. * \[ ] **For Warehouse-native functions:** Submit a [GitHub issue](https://github.com/dbt-labs/dbt-fusion). dbt v2's `baseline` mode handles most cases, but will throw warnings and not errors. You can set `static_analysis: off` for specific models if needed. For more information about using `strict` in development and `baseline` in deployment, refer to [Optimize static analysis for development and deployment](https://docs.getdbt.com/best-practices/optimize-static-analysis-for-development-and-deployment.md). ##### Check for known dbt v2 limitations Your project may implement features that dbt v2 currently [limits](https://docs.getdbt.com/docs/dbt/supported-features.md#limitations) or doesn't support. * \[ ] Remove unnecessary features from your project to make it dbt v2 compatible. * \[ ] Monitor progress for critical features, knowing we are working to bring them to dbt v2. You can monitor their progress using the issues linked in the [limitations table](https://docs.getdbt.com/docs/dbt/supported-features.md#limitations). ##### Review jobs configured in the dbt platform We determine dbt v2 eligibility using data from your job runs. * \[ ] Ensure you have at least one job running in each of your projects in the dbt platform. * \[ ] Ensure each job has run within the last 7 days. Jobs that haven't run recently are considered inactive and are ineligible for dbt v2. If you see a "no active jobs" ineligibility reason in the v2 readiness UI, run the job manually or adjust its schedule. * \[ ] Ensure all jobs are running on the [**v1 Latest** release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md#which-release-tracks-are-available) for the smoothest upgrade experience. * \[ ] Resolve any job failures — all jobs must run successfully for eligibility checks to work. * \[ ] Delete any jobs that are no longer in use to ensure accurate eligibility reporting. * \[ ] Make sure you've promoted the changes for deprecation resolution and package upgrades to your git branches that map to your deployment environments. * \[ ] For eligible jobs, use **Debug on dbt v2** to debug in Studio IDE or run once on dbt v2. Refer to [Update your jobs](https://docs.getdbt.com/guides/prepare-v2-upgrade.md?step=7). ##### Stay informed about dbt v2 progress dbt v2 is generally available for dbt platform projects on Snowflake and in preview for all other eligible projects! Keep up-to-date with these resources: * \[ ] Check out the [v2 homepage](https://www.getdbt.com/product/fusion) for available resources, including supported adapters, prerequisites, installation instructions, limitations, and deprecations. * \[ ] Read the [Upgrade guide](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md) to learn about the new features and functionality that impact your dbt projects. * \[ ] Learn how [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) can reduce warehouse costs by 30%+ by rebuilding models only when data or code changes. --- ### Feature availability You can get started with many dbt features right away, and even more are available when you sign in with any dbt platform account, even the free developer tier! #### What you get with v2 Feature availability Feature availability may change as dbt v2 moves toward general availability. You can get started right away with many dbt features, free forever! You can also try advanced features by running [`dbt login`](https://docs.getdbt.com/reference/commands/login.md?version=2.0) to create a free dbt platform account for the best experience: ```shell dbt login ``` Creating an account also unlocks additional free-tier access to dbt services. | Feature | Free forever (for real!) | Requires login<br />to any dbt platform account, free or paid | | ----------------------------------------------------- | ------------------------ | ------------------------------------------------------------- | | dbt v1 workflows, except dbt docs v1 | ✅ | ✅ | | Syntax error detection (Jinja, YAML, SQL) | ✅ | ✅ | | dbt lint | ✅ | ✅ | | dbt docs v2 (lite) | ✅ | ✅ | | LSP (lite): go-to ref, source, and macro | ✅ | ✅ | | Full LSP: CTE, hover to see schema, and more | - | ✅ | | SQL comprehension, type checking, and impact analysis | - | ✅ | | Precise column-level lineage | - | ✅ | | dbt docs v2 (full), including column-level lineage | - | ✅ | For the best dbt experience, use the dbt VS Code extension. You can get started for free, and when you create a free dbt platform account, you’ll unlock additional access to advanced dbt features in your editor and beyond, including those shown in the table above. To learn more about VS Code-specific capabilities, refer to [dbt VS Code extension features](https://docs.getdbt.com/docs/dbt-extension-features.md). #### Related docs * [Install dbt](https://docs.getdbt.com/docs/local/install-dbt.md) locally * Install the [dbt VS Code extension](https://docs.getdbt.com/docs/about-dbt-extension.md) * Upgrade environments in the [dbt platform](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md#dbt-v2) --- ### Get started with dbt ##### New to dbt? Start here Once you've caught up on everything [dbt has to offer](https://docs.getdbt.com/docs/introduction.md), start with the path that matches how you want to use dbt. [](https://docs.getdbt.com/docs/local/install-dbt.md) [](https://docs.getdbt.com/docs/local/install-dbt.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/local/install-dbt.md)[Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") ###### Install dbt Install dbt from your command line. [](https://docs.getdbt.com/docs/about-dbt-extension.md) [](https://docs.getdbt.com/docs/about-dbt-extension.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/docs/about-dbt-extension.md)[Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") ###### Install the dbt VS Code extension Get dbt's performance plus autocomplete, inline errors, and lineage in your editor. [![](/img/icons/dbt-bit.svg)](https://www.getdbt.com/signup) ###### [Sign up for the dbt platform](https://www.getdbt.com/signup) [Create a dbt platform account to unlock the full dbt feature set.](https://www.getdbt.com/signup) ##### Already using dbt? Start here Upgrade your existing projects to v2 and learn about the tools available to you once you're there. [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/guides/prepare-v2-upgrade.md) ###### [Preparing to upgrade from v1 to v2](https://docs.getdbt.com/guides/prepare-v2-upgrade.md) [Step-by-step guide to prepare your dbt platform projects for upgrading to v2.](https://docs.getdbt.com/guides/prepare-v2-upgrade.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/guides/upgrade-to-v2.md) ###### [Upgrade to v2](https://docs.getdbt.com/guides/upgrade-to-v2.md) [Learn how to upgrade your eligible projects on the dbt platform to v2.](https://docs.getdbt.com/guides/upgrade-to-v2.md) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/guides/dbt-platform-local-workflow.md) ###### [Hybrid dbt platform and local workflows](https://docs.getdbt.com/guides/dbt-platform-local-workflow.md) [Learn how to keep credentials, environment variables, and dbt versions in sync between dbt platform and local development.](https://docs.getdbt.com/guides/dbt-platform-local-workflow.md) --- ### Self-hosted dbt releases Preview ### Self-hosted dbt releases [Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") This page shows release information for local builds of dbt v2 only. v2 releases on the dbt platform adhere to the [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) categories, giving you control over release cadence and stability. Track current versions and full release history for v2. This data updates live from dbt release channels. Each of the versions on this page links to the matching section in the [v2 changelog](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md) on GitHub. #### Release channels dbt v2 is distributed through three release channels: | Channel | Description | Stability | | -------- | -------------------------------------------- | ------------------------------------------------------------------- | | `latest` | The known `good` stable version | ✅ Recommended for production | | `canary` | The latest version to be officially released | ⚠️ Most recent stable version but still undergoing thorough testing | | `dev` | The latest development build | ❌ May be unstable; may not have passed all internal tests | #### Known-bad releases If a shipped v2 release is later found to contain a regression, dbt Labs flags it as a known-bad release. If you have a flagged version installed, the dbt VS Code extension shows a warning notification telling you which version to update to. To move off a flagged version, update it using your installation method (for example, pip or Homebrew). For details, including how air-gapped users receive these notifications, refer to [Known-bad releases](https://docs.getdbt.com/docs/dbt-versions/dbt-version-compatibility.md#known-bad-releases). #### dbt platform release tracks On dbt platform, each [environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md) uses the account default or your chosen **release track**. Release tracks control how often that environment receives new v2 builds. They're separate from the local CLI release channels in the previous section. For cadence, plan availability, and API values (`nightly`, `stable`, and more), refer to [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md#dbt-v2-release-tracks). To change the release track for an environment, follow [Upgrade dbt in dbt platform](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md). Live data below is for local CLI channels The **Current versions** cards and full release list below pull the public v2 manifest used for *local* installs (`dev`, `canary`, `latest`). You should use [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) for dbt platform planning. Updating your self-hosted installation The following commands apply only to *local* installations of dbt. They don't affect which v2 build your dbt platform environments use. Instead, you can set a [release track](https://github.com/docs/dbt-versions/dbt-release-tracks#dbt-v2-release-tracks) per environment in dbt platform. Running the system update command without a version flag installs the `latest` stable release: ```shell dbt system update ``` To install a specific channel or version, pass the `--version` flag: ```shell dbt system update --version canary # Install the canary release dbt system update --version dev # Install the dev release dbt system update --version 2.0.0-preview.126 # Install a specific version ``` ##### Current versions ###### Dev [`v2.0.0`](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200 "View this version in the dbt v2 changelog") 2026-09-10 ###### Canary [`v2.0.0`](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200 "View this version in the dbt v2 changelog") 2026-09-10 ###### Latest [`v2.0.0-preview.218`](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview218 "View this version in the dbt v2 changelog") 2026-08-27 ##### All releases Search versions... Status:All (all) Channel:All (all) Showing 180 of 180 releases [v2.0.0-preview.221](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview221 "View this release in the dbt v2 changelog")GoodDevnightly Released by: **jasonlin45**Sep 9, 2026, 03:12 AM Automated promotion [v2.0.0-preview.220](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview220 "View this release in the dbt v2 changelog")GoodDevnightly Released by: **jasonlin45**Sep 3, 2026, 11:23 PM Automated promotion [v2.0.0-preview.219](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview219 "View this release in the dbt v2 changelog")GoodDevnightlyCanary Released by: **aiguofer**Sep 3, 2026, 07:56 PM planned promotion [v2.0.0-preview.218](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview218 "View this release in the dbt v2 changelog")GoodDevnightlyCanaryLatestST MondayST WednesdaystableST Thursdayst-monday-stablest-wednesday-stablest-thursday-stable Released by: **johnchappelledbt**Sep 9, 2026, 10:19 PM Automated ST snapshot [v2.0.0-preview.217](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview217 "View this release in the dbt v2 changelog")GoodDevnightly Released by: **TIHan**Aug 27, 2026, 08:53 AM Automated promotion [v2.0.0-preview.216](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview216 "View this release in the dbt v2 changelog")Known BadDevnightly Released by: **github-merge-queue**Aug 26, 2026, 08:59 AM Automated promotion [v2.0.0-preview.215](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview215 "View this release in the dbt v2 changelog")Known BadDevnightlyCanary Released by: **mishamsk**Aug 25, 2026, 11:15 PM Planned canary promotion [v2.0.0-preview.214](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview214 "View this release in the dbt v2 changelog")Known BadDevnightly Released by: **mach-kernel**Aug 25, 2026, 03:50 PM Automated promotion [v2.0.0-preview.213](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview213 "View this release in the dbt v2 changelog")GoodDevnightlyCanaryLatest Released by: **mishamsk**Aug 26, 2026, 10:25 PM planned [v2.0.0-preview.212](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview212 "View this release in the dbt v2 changelog")GoodDevnightlyCanarystablest-monday-stableLatestst-wednesday-stableextendedst-thursday-stablest-monday-extendedst-wednesday-extendedst-thursday-extended Released by: **johnchappelledbt**Sep 9, 2026, 10:19 PM Automated ST snapshot [v2.0.0-preview.210](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview210 "View this release in the dbt v2 changelog")GoodDevnightlyCanaryLateststableST MondayST Wednesday Released by: **jeremyhutt11**Aug 26, 2026, 11:46 PM Automated ST promotion [v2.0.0-preview.209](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview209 "View this release in the dbt v2 changelog")GoodDevnightlyCanarystableLatestST Mondayst-monday-stableST Wednesdayst-wednesday-stableST Thursdayst-thursday-stable Released by: **TimKlense**Aug 20, 2026, 09:09 PM Automated ST snapshot [v2.0.0-preview.208](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview208 "View this release in the dbt v2 changelog")Known BadDevnightly Released by: **github-merge-queue**Aug 8, 2026, 08:59 AM Automated promotion [v2.0.0-preview.207](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview207 "View this release in the dbt v2 changelog")Known BadDevnightly Released by: **akbog**Aug 6, 2026, 12:33 AM Automated promotion [v2.0.0-preview.206](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview206 "View this release in the dbt v2 changelog")Known BadDevnightlyCanary Released by: **akbog**Aug 5, 2026, 01:11 AM Planned Promotion [v2.0.0-preview.205](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview205 "View this release in the dbt v2 changelog")GoodDevnightlyCanarystableLatestST Mondayst-monday-stableST Wednesdayst-wednesday-stableST Thursdayst-thursday-stable Released by: **akbog**Aug 14, 2026, 02:44 AM Incident Investigation [v2.0.0-preview.204](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview204 "View this release in the dbt v2 changelog")GoodDevnightlyCanarystableLatest Released by: **akbog**Aug 3, 2026, 08:18 PM Planned Promotion [v2.0.0-preview.203](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview203 "View this release in the dbt v2 changelog")GoodDevnightlyCanarystableLatestST Mondayst-monday-stableextendedst-wednesday-stableST Thursdayst-thursday-stablest-monday-extendedst-wednesday-extendedst-thursday-extendedfallbackst-monday-fallbackst-wednesday-fallbackst-thursday-fallback Released by: **johnchappelledbt**Sep 9, 2026, 10:19 PM Automated ST snapshot [v2.0.0-preview.202](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview202 "View this release in the dbt v2 changelog")GoodDevnightlyCanarystableLatestST Mondayst-monday-stableST Wednesdayst-wednesday-stableST Thursdayst-thursday-stable Released by: **TimKlense**Jul 30, 2026, 09:26 PM Automated ST snapshot [v2.0.0-preview.201](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview201 "View this release in the dbt v2 changelog")GoodDevnightly Released by: **mishamsk**Jul 20, 2026, 05:42 PM Automated promotion [v2.0.0-preview.200](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview200 "View this release in the dbt v2 changelog")GoodDevnightlyCanaryLateststableST Mondayst-monday-stableST Wednesdayst-wednesday-stableST Thursdayst-thursday-stable Released by: **TimKlense**Jul 23, 2026, 09:25 PM Automated ST snapshot [v2.0.0-preview.199](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview199 "View this release in the dbt v2 changelog")GoodDevCanarynightlystableLatestST Mondayst-monday-stableST Wednesdayst-wednesday-stableST Thursdayst-thursday-stable Released by: **jeremyhutt11**Jul 16, 2026, 10:12 PM Automated ST snapshot [v2.0.0-preview.198](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview198 "View this release in the dbt v2 changelog")GoodDev Released by: **kczimm**Jul 9, 2026, 09:36 PM Automated promotion [v2.0.0-preview.197](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview197 "View this release in the dbt v2 changelog")GoodDevCanarynightly Released by: **akbog**Jul 2, 2026, 07:20 PM Planned promotion [v2.0.0-preview.196](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview196 "View this release in the dbt v2 changelog")GoodDevCanaryLateststableST Mondayst-monday-stableST Wednesdayst-wednesday-stableST Thursdayst-thursday-stable Released by: **TimKlense**Jul 9, 2026, 09:31 PM Automated ST snapshot [v2.0.0-preview.195](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview195 "View this release in the dbt v2 changelog")GoodDev Released by: **aiguofer**Jun 30, 2026, 12:08 AM Automated promotion [v2.0.0-preview.194](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview194 "View this release in the dbt v2 changelog")GoodDevCanarystableextendedst-monday-extendedst-wednesday-extendedst-thursday-extendedfallbackst-monday-fallbackst-wednesday-fallbackst-thursday-fallback Released by: **TimKlense**Sep 3, 2026, 09:03 PM Automated ST snapshot [v2.0.0-preview.193](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview193 "View this release in the dbt v2 changelog")GoodDevCanaryLatestnightlystableST Mondayst-monday-stableST Wednesdayst-wednesday-stableST Thursdayst-thursday-stable Released by: **TimKlense**Jul 2, 2026, 09:26 PM Automated ST snapshot [v2.0.0-preview.192](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview192 "View this release in the dbt v2 changelog")Known BadDevCanaryLatestnightlystable Released by: **akbog**Jun 18, 2026, 06:22 PM Planned promotion [v2.0.0-preview.191](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview191 "View this release in the dbt v2 changelog")Known BadDevCanarynightly Released by: **akbog**Jun 12, 2026, 09:14 PM Planned Promotion [v2.0.0-preview.190](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview190 "View this release in the dbt v2 changelog")GoodDevCanarynightlystableLatestST Mondayst-monday-stableST Wednesdayst-wednesday-stableST Thursdayst-thursday-stable Released by: **TimKlense**Jun 25, 2026, 09:33 PM Automated ST snapshot [v2.0.0-preview.189](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview189 "View this release in the dbt v2 changelog")GoodDevCanaryLateststable Released by: **akbog**Jun 11, 2026, 03:15 AM Planned Promotion [v2.0.0-preview.188](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview188 "View this release in the dbt v2 changelog")Known BadDevCanarystablenightlyst-monday-stable Released by: **fa-assistant**Jun 8, 2026, 07:05 PM Automated ST snapshot [v2.0.0-preview.187](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview187 "View this release in the dbt v2 changelog")Known BadDev Released by: **akbog**Jun 6, 2026, 02:06 AM Automated promotion [v2.0.0-preview.186](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview186 "View this release in the dbt v2 changelog")GoodDevnightlyCanarystableLatestST Mondayst-monday-stableextendedST Thursdayst-thursday-stablest-monday-extendedst-wednesday-extendedst-thursday-extendedfallbackst-monday-fallbackst-wednesday-fallbackst-thursday-fallback Released by: **laconc**Aug 6, 2026, 10:00 PM Automated ST snapshot [v2.0.0-preview.185](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview185 "View this release in the dbt v2 changelog")GoodDev Released by: **akbog**Jun 3, 2026, 07:49 AM Automated promotion [v2.0.0-preview.184](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview184 "View this release in the dbt v2 changelog")GoodDevCanarystableLatestst-thursday-stable Released by: **fa-assistant**Jun 4, 2026, 09:34 PM Automated ST snapshot [v2.0.0-preview.183](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview183 "View this release in the dbt v2 changelog")GoodDevCanaryLateststablenightlyst-wednesday-stable Released by: **fa-assistant**Jun 3, 2026, 10:03 PM Automated ST snapshot [v2.0.0-preview.182](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview182 "View this release in the dbt v2 changelog")GoodDevCanaryLateststablenightlyST Mondayst-monday-stableST WednesdayST Thursday Released by: **fa-assistant**Jun 4, 2026, 09:32 PM Automated ST promotion [v2.0.0-preview.181](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview181 "View this release in the dbt v2 changelog")GoodDevCanarystableLatestnightly Released by: **akbog**Jun 1, 2026, 06:55 AM Planned Promotion [v2.0.0-preview.180](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview180 "View this release in the dbt v2 changelog")GoodDevCanarystableLatestnightly Released by: **akbog**May 30, 2026, 06:35 AM Planned promotion [v2.0.0-preview.179](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview179 "View this release in the dbt v2 changelog")GoodDev Released by: **kczimm**May 29, 2026, 05:01 PM Automated promotion [v2.0.0-preview.178](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview178 "View this release in the dbt v2 changelog")GoodDevCanaryLateststablenightlyst-wednesday-stablest-thursday-stable Released by: **fa-assistant**May 29, 2026, 01:12 PM Automated ST snapshot [v2.0.0-preview.177](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview177 "View this release in the dbt v2 changelog")GoodDevCanarystableLatestnightlyst-thursday-stableST Mondayst-monday-stableST WednesdayST Thursday Released by: **fa-assistant**May 29, 2026, 01:10 PM Automated ST promotion [v2.0.0-preview.176](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview176 "View this release in the dbt v2 changelog")GoodDevCanarynightlystableLatestst-wednesday-stablest-thursday-stableST MondayST WednesdayST Thursday Released by: **fa-assistant**May 21, 2026, 09:31 PM Automated ST promotion [v2.0.0-preview.175](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview175 "View this release in the dbt v2 changelog")GoodDevCanaryLateststablest-thursday-stablest-monday-stablest-wednesday-stableST MondayextendedfallbacknightlyST WednesdayST Thursdayst-monday-extendedst-monday-fallbackst-wednesday-extendedst-wednesday-fallbackst-thursday-extendedst-thursday-fallback Released by: **TimKlense**Jul 9, 2026, 09:31 PM Automated ST snapshot [v2.0.0-preview.174](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview174 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **ayshukla**Apr 29, 2026, 03:25 PM Planned orchestration promotion [v2.0.0-preview.173](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview173 "View this release in the dbt v2 changelog")release candidateGoodDevCanaryLatestST MondaynightlystableST WednesdayST Thursday Released by: **TimKlense**Apr 30, 2026, 11:02 PM manually updated for thursday [v2.0.0-preview.172](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview172 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **mikaylacrawford**Apr 15, 2026, 04:28 PM Planned orchestration promotion [v2.0.0-preview.171](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview171 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST MondayST WednesdayST Thursday Released by: **fa-assistant**Apr 16, 2026, 09:21 PM Automated ST promotion [v2.0.0-preview.170](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview170 "View this release in the dbt v2 changelog")GoodDevCanaryLatestextendedfallbackst-thursday-extendedst-thursday-fallbackst-monday-extendedst-monday-fallbackst-wednesday-extendedst-wednesday-fallback Released by: **agelber-dbt**May 1, 2026, 02:22 PM Preparing for GA [v2.0.0-preview.169](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview169 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **github-merge-queue**Apr 8, 2026, 12:06 PM Automated promotion [v2.0.0-preview.168](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview168 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **peter-bertuglia**Apr 8, 2026, 08:04 PM Planned orchestration promotion [v2.0.0-preview.167](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview167 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **github-merge-queue**Apr 7, 2026, 12:17 PM Automated promotion [v2.0.0-preview.166](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview166 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **peter-bertuglia**Apr 7, 2026, 04:34 PM Planned orchestration promotion [v2.0.0-preview.165](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview165 "View this release in the dbt v2 changelog")GoodDev Released by: **akbog**Apr 4, 2026, 03:16 AM Automated promotion [v2.0.0-preview.164](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview164 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST MondayST WednesdayST Thursday Released by: **fa-assistant**Apr 9, 2026, 09:21 PM Automated ST promotion [v2.0.0-preview.163](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview163 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **ayshukla**Mar 26, 2026, 03:46 PM Planned orchestration promotion [v2.0.0-preview.162](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview162 "View this release in the dbt v2 changelog")Known BadDevCanary Released by: **dataders**Mar 23, 2026, 04:19 PM Automated promotion [v2.0.0-preview.161](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview161 "View this release in the dbt v2 changelog")Known BadDevCanaryLatest Released by: **james-durand-dbt**Mar 24, 2026, 04:22 PM Planned orchestration promotion [v2.0.0-preview.160](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview160 "View this release in the dbt v2 changelog")Known BadDevCanary Released by: **dataders**Mar 22, 2026, 07:14 PM Automated promotion [v2.0.0-preview.159](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview159 "View this release in the dbt v2 changelog")Known BadDevCanary Released by: **github-merge-queue**Mar 20, 2026, 12:15 PM Automated promotion [v2.0.0-preview.158](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview158 "View this release in the dbt v2 changelog")Known BadDevCanaryLatest Released by: **kylepeirce**Mar 20, 2026, 03:50 PM Planned orchestration promotion [v2.0.0-preview.157](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview157 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **jasonlin45**Mar 18, 2026, 08:35 PM Automated promotion [v2.0.0-preview.156](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview156 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **kczimm**Mar 17, 2026, 08:55 PM Automated promotion [v2.0.0-preview.155](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview155 "View this release in the dbt v2 changelog")GoodDev Released by: **github-merge-queue**Mar 16, 2026, 12:20 PM Automated promotion [v2.0.0-preview.154](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview154 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST MondayST WednesdayST Thursday Released by: **fa-assistant**Mar 26, 2026, 09:14 PM Automated ST promotion [v2.0.0-preview.153](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview153 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **kylepeirce**Mar 12, 2026, 10:37 PM Bug fix for INC-5687 [v2.0.0-preview.152](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview152 "View this release in the dbt v2 changelog")GoodDev Released by: **github-merge-queue**Mar 12, 2026, 12:13 PM Automated promotion [v2.0.0-preview.151](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview151 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **kylepeirce**Mar 12, 2026, 06:52 PM Planned orchestration promotion [v2.0.0-preview.150](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview150 "View this release in the dbt v2 changelog")GoodDev Released by: **kczimm**Mar 10, 2026, 10:53 PM Automated promotion [v2.0.0-preview.149](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview149 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **mikaylacrawford**Mar 10, 2026, 03:29 PM Planned orchestration promotion [v2.0.0-preview.148](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview148 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST MondayST WednesdayST Thursday Released by: **fa-assistant**Mar 12, 2026, 10:10 PM Automated ST promotion [v2.0.0-preview.147](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview147 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **kczimm**Mar 7, 2026, 02:04 AM Automated promotion [v2.0.0-preview.146](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview146 "View this release in the dbt v2 changelog")Known BadDevCanary Released by: **kczimm**Mar 5, 2026, 11:58 PM Automated promotion [v2.0.0-preview.145](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview145 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **venkaa28**Mar 6, 2026, 04:22 AM INC-5500 [v2.0.0-preview.144](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview144 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **peter-bertuglia**Mar 4, 2026, 08:41 PM retrying scheduled promo for vscode ext [v2.0.0-preview.143](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview143 "View this release in the dbt v2 changelog")Known BadDevCanaryLatestST Monday Released by: **akbog**Mar 3, 2026, 03:43 PM inc-5454-minor-table-not-found-in-schema-errors [v2.0.0-preview.142](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview142 "View this release in the dbt v2 changelog")GoodDev Released by: **github-merge-queue**Feb 27, 2026, 12:02 PM Automated promotion [v2.0.0-preview.141](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview141 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **jasonlin45**Feb 26, 2026, 10:31 PM Automated promotion [v2.0.0-preview.139](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview139 "View this release in the dbt v2 changelog")GoodDev Released by: **jasonlin45**Feb 26, 2026, 06:47 PM Automated promotion [v2.0.0-preview.137](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview137 "View this release in the dbt v2 changelog")GoodDev Released by: **jasonlin45**Feb 26, 2026, 01:55 AM Automated promotion [v2.0.0-preview.135](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview135 "View this release in the dbt v2 changelog")GoodDev Released by: **chayac**Feb 25, 2026, 12:45 AM Automated promotion [v2.0.0-preview.134](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview134 "View this release in the dbt v2 changelog")GoodDev Released by: **chayac**Feb 24, 2026, 10:42 PM Automated promotion [v2.0.0-preview.127](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview127 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **github-merge-queue**Feb 20, 2026, 12:17 PM Automated promotion [v2.0.0-preview.126](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview126 "View this release in the dbt v2 changelog")Known BadDevCanaryLatest Released by: **mikaylacrawford**Feb 20, 2026, 07:38 PM Planned orchestration promotion [v2.0.0-preview.125](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview125 "View this release in the dbt v2 changelog")GoodDev Released by: **github-merge-queue\[bot]**Feb 19, 2026, 06:05 PM Automated promotion [v2.0.0-preview.123](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview123 "View this release in the dbt v2 changelog")GoodDev Released by: **github-merge-queue**Feb 18, 2026, 12:06 PM Automated promotion [v2.0.0-preview.121](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview121 "View this release in the dbt v2 changelog")GoodDev Released by: **github-merge-queue**Feb 16, 2026, 12:09 PM Automated promotion [v2.0.0-preview.120](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview120 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST MondayST WednesdayST Thursday Released by: **fa-assistant**Mar 5, 2026, 10:13 PM Automated ST promotion [v2.0.0-preview.119](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview119 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **dataders**Feb 13, 2026, 06:48 PM Automated promotion [v2.0.0-preview.118](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview118 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **github-merge-queue**Feb 13, 2026, 12:15 PM Automated promotion [v2.0.0-preview.117](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview117 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **jzhu13**Feb 13, 2026, 12:45 AM Automated promotion [v2.0.0-preview.116](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview116 "View this release in the dbt v2 changelog")GoodDev Released by: **jzhu13**Feb 12, 2026, 07:36 PM Automated promotion [v2.0.0-preview.114](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview114 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST MondayST WednesdayST Thursday Released by: **fa-assistant**Feb 26, 2026, 10:15 PM Automated ST promotion [v2.0.0-preview.110](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview110 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **peter-bertuglia**Feb 6, 2026, 07:34 PM INC-5104 [v2.0.0-preview.108](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview108 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **peter-bertuglia**Feb 5, 2026, 04:42 PM Planned orchestration promotion [v2.0.0-preview.105](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview105 "View this release in the dbt v2 changelog")GoodDev Released by: **github-merge-queue**Jan 30, 2026, 01:12 PM Automated promotion [v2.0.0-preview.104](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview104 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST MondayST WednesdayST Thursday Released by: **fa-assistant**Feb 12, 2026, 10:13 PM Automated ST promotion [v2.0.0-preview.103](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview103 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **github-merge-queue**Jan 27, 2026, 01:09 PM Automated promotion [v2.0.0-preview.102](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview102 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **ayshukla**Jan 27, 2026, 08:06 PM Planned orchestration promotion [v2.0.0-preview.101](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview101 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST MondayST WednesdayST Thursday Released by: **fa-assistant**Jan 29, 2026, 10:12 PM Automated ST promotion [v2.0.0-preview.100](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview100 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **github-merge-queue**Jan 21, 2026, 01:42 PM Automated promotion [v2.0.0-preview.99](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview99 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **akbog**Jan 21, 2026, 07:10 AM Automated promotion [v2.0.0-preview.98](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview98 "View this release in the dbt v2 changelog")Known BadDevCanary Released by: **github-merge-queue**Jan 19, 2026, 01:22 PM Automated promotion [v2.0.0-preview.97](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview97 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **github-merge-queue**Jan 16, 2026, 01:14 PM Automated promotion [2.0.0-preview.97](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview97 "View this release in the dbt v2 changelog")Known Bad [v2.0.0-preview.96](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview96 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST MondayST WednesdayST Thursday Released by: **fa-assistant**Jan 22, 2026, 10:10 PM Automated ST promotion [v2.0.0-preview.95](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview95 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **jasonlin45**Jan 15, 2026, 06:16 AM Automated promotion [v2.0.0-preview.94](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview94 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **mikaylacrawford**Jan 13, 2026, 07:26 PM Planned orchestration promotion [v2.0.0-preview.93](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview93 "View this release in the dbt v2 changelog")Known BadDevCanary Released by: **github-merge-queue**Jan 9, 2026, 01:24 PM Automated promotion [v2.0.0-preview.92](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview92 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST MondayST WednesdayST Thursday Released by: **fa-assistant**Jan 15, 2026, 10:10 PM Automated ST promotion [v2.0.0-preview.91](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview91 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **github-merge-queue**Dec 22, 2025, 01:09 PM Automated promotion [v2.0.0-preview.90](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview90 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **github-merge-queue**Dec 19, 2025, 01:04 PM Automated promotion [v2.0.0-preview.89](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview89 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST MondayST WednesdayST Thursday Released by: **laconc**Dec 24, 2025, 12:06 AM st release [v2.0.0-preview.88](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview88 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **ayshukla**Dec 18, 2025, 04:20 PM Planned orchestration promotion [v2.0.0-preview.87](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview87 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **github-merge-queue\[bot]**Dec 16, 2025, 01:54 PM Automated promotion [v2.0.0-preview.86](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview86 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **james-durand-dbt**Dec 15, 2025, 08:35 PM INC-4737 [v2.0.0-preview.85](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview85 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST MondayST WednesdayST Thursday Released by: **fa-assistant**Dec 18, 2025, 10:10 PM Automated ST promotion [v2.0.0-preview.84](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview84 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **james-durand-dbt**Dec 12, 2025, 06:04 PM Planned orchestration promotion [v2.0.0-preview.83](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview83 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **mikaylacrawford**Dec 10, 2025, 04:19 PM Planned orchestration promotion [v2.0.0-preview.82](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview82 "View this release in the dbt v2 changelog")Known BadDevCanary Released by: **xuliangs**Dec 9, 2025, 07:41 PM Automated promotion [v2.0.0-preview.81](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview81 "View this release in the dbt v2 changelog")Known BadDevCanary Released by: **ddk-dbt**Dec 9, 2025, 03:21 PM Automated promotion [v2.0.0-preview.80](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview80 "View this release in the dbt v2 changelog")Known BadDevCanary Released by: **venkaa28**Dec 8, 2025, 07:34 PM Automated promotion [v2.0.0-preview.79](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview79 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **ddk-dbt**Dec 5, 2025, 03:18 PM Automated promotion [v2.0.0-preview.78](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview78 "View this release in the dbt v2 changelog")Known BadDevCanary Released by: **ddk-dbt**Dec 4, 2025, 03:24 PM Automated promotion [v2.0.0-preview.77](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview77 "View this release in the dbt v2 changelog")Known BadDevCanary Released by: **ddk-dbt**Dec 3, 2025, 10:26 PM Automated promotion [v2.0.0-preview.76](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview76 "View this release in the dbt v2 changelog")Known BadDevCanaryLatest Released by: **venkaa28**Dec 4, 2025, 01:36 AM preview.77 causing issues for IA. rolling back for incident. [v2.0.0-preview.75](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview75 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **jasonlin45**Nov 22, 2025, 04:09 AM Automated promotion [v2.0.0-preview.74](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview74 "View this release in the dbt v2 changelog")GoodDev Released by: **ddk-dbt**Nov 21, 2025, 03:05 PM Automated promotion [v2.0.0-preview.73](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview73 "View this release in the dbt v2 changelog")GoodDev Released by: **ddk-dbt**Nov 20, 2025, 03:19 PM Automated promotion [v2.0.0-preview.72](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview72 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST MondayST WednesdayST Thursday Released by: **fa-assistant**Dec 11, 2025, 10:11 PM Automated ST promotion [v2.0.0-preview.71](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview71 "View this release in the dbt v2 changelog")GoodDev Released by: **venkaa28**Nov 19, 2025, 02:06 AM Automated promotion [v2.0.0-preview.70](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview70 "View this release in the dbt v2 changelog")GoodDev Released by: **ddk-dbt**Nov 17, 2025, 03:03 PM Automated promotion [v2.0.0-preview.69](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview69 "View this release in the dbt v2 changelog")GoodDev Released by: **ChenyuLInx**Nov 15, 2025, 06:57 AM Automated promotion [v2.0.0-preview.68](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview68 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **dbtlabs007**Nov 18, 2025, 12:44 AM Planned orchestration promotion [v2.0.0-preview.67](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview67 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST MondayST WednesdayST Thursday Released by: **laconc**Nov 20, 2025, 10:47 PM Thursday ST release, manual run [v2.0.0-preview.66](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview66 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **mishamsk**Nov 12, 2025, 02:09 PM Automated promotion [v2.0.0-preview.65](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview65 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **ayshukla**Nov 12, 2025, 10:00 PM Planned orchestration promotion [v2.0.0-preview.63](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview63 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST MondayST WednesdayST Thursday Released by: **ddk-dbt**Nov 13, 2025, 11:05 PM scheduled promotion [v2.0.0-preview.62](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview62 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **jzhu13**Nov 5, 2025, 11:18 PM Automated promotion [v2.0.0-preview.61](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview61 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **ddk-dbt**Nov 5, 2025, 02:44 PM Automated promotion [v2.0.0-preview.60](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview60 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **mikaylacrawford**Nov 5, 2025, 08:15 PM Planned orchestration promotion [v2.0.0-preview.59](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview59 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **mikaylacrawford**Nov 5, 2025, 02:50 PM Planned orchestration promotion [v2.0.0-preview.58](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview58 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **ddk-dbt**Nov 3, 2025, 02:39 PM Automated promotion [v2.0.0-preview.57](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview57 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST MondayST WednesdayST Thursday Released by: **fa-assistant**Nov 6, 2025, 10:09 PM Automated ST promotion [v2.0.0-preview.56](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview56 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **mikaylacrawford**Oct 31, 2025, 03:24 PM Planned orchestration promotion [v2.0.0-preview.55](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview55 "View this release in the dbt v2 changelog")GoodDev Released by: **dataders**Oct 30, 2025, 07:08 PM Automated promotion [v2.0.0-preview.54](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview54 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **ddk-dbt**Oct 30, 2025, 03:24 PM Automated promotion [v2.0.0-preview.53](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview53 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **mikaylacrawford**Oct 30, 2025, 04:47 PM Planned orchestration promotion [v2.0.0-preview.52](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview52 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **ddk-dbt**Oct 29, 2025, 02:37 PM Automated promotion [v2.0.0-preview.51](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview51 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **dataders**Oct 29, 2025, 01:13 AM Automated promotion [v2.0.0-preview.50](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview50 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **peter-bertuglia**Oct 28, 2025, 06:46 PM Planned orchestration promotion [v2.0.0-preview.49](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview49 "View this release in the dbt v2 changelog")GoodDev Released by: **jzhu13**Oct 22, 2025, 06:12 PM Automated promotion [v2.0.0-preview.48](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview48 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST Monday Released by: **fa-assistant**Oct 27, 2025, 06:13 PM Automated promotion [v2.0.0-preview.47](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview47 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **ayshukla**Oct 22, 2025, 09:27 PM Planned orchestration promotion [v2.0.0-preview.45](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview45 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST MondayST Wednesday Released by: **johnchappelledbt**Oct 22, 2025, 10:16 PM 10-22-25 Wed ST release [v2.0.0-preview.44](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview44 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **dbtlabs007**Oct 9, 2025, 07:39 PM inc-3966-informational-releasing-dbt-lsp-hotfix-to-fix-hang-coalesce-thaw [v2.0.0-preview.43](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview43 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **ayshukla**Oct 9, 2025, 03:56 PM fs deployment freeze promotion [v2.0.0-preview.42](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview42 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **github-merge-queue**Oct 8, 2025, 02:26 PM Automated promotion [v2.0.0-preview.41](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview41 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST MondayST WednesdayST Thursday Released by: **ddk-dbt**Oct 8, 2025, 09:45 PM pre-coalesce sync [v2.0.0-preview.40](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview40 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **davidharting**Oct 7, 2025, 09:56 PM Automated promotion [v2.0.0-preview.39](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview39 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **github-merge-queue**Oct 7, 2025, 02:33 PM Automated promotion [v2.0.0-preview.38](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview38 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **ayshukla**Oct 7, 2025, 04:19 PM orc release [v2.0.0-preview.37](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview37 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **github-merge-queue\[bot]**Oct 6, 2025, 02:40 PM Automated promotion [v2.0.0-preview.36](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview36 "View this release in the dbt v2 changelog")Known BadDevCanary Released by: **github-merge-queue**Oct 3, 2025, 02:21 PM Automated promotion [v2.0.0-preview.35](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview35 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST Monday Released by: **fa-assistant**Oct 6, 2025, 06:13 PM Automated promotion [v2.0.0-preview.34](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview34 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **ayshukla**Oct 3, 2025, 03:58 PM Planned orchestration promotion [v2.0.0-preview.33](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview33 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **ayshukla**Oct 2, 2025, 06:23 PM Planned orchestration promotion [v2.0.0-preview.32](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview32 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **mikaylacrawford**Oct 1, 2025, 03:21 PM Planned orchestration promotion [v2.0.0-preview.31](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview31 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **github-merge-queue\[bot]**Sep 30, 2025, 02:44 PM Automated promotion [v2.0.0-preview.30](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview30 "View this release in the dbt v2 changelog")GoodDevCanaryLatest Released by: **mikaylacrawford**Sep 30, 2025, 04:01 PM Planned orchestration promotion [v2.0.0-preview.29](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview29 "View this release in the dbt v2 changelog")GoodDevCanaryLatestST MondayST WednesdayST Thursday Released by: **ddk-dbt**Oct 3, 2025, 03:26 PM Automated promotion [v2.0.0-preview.28](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview28 "View this release in the dbt v2 changelog")GoodDev Sep 25, 2025, 02:08 PM [v2.0.0-preview.27](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview27 "View this release in the dbt v2 changelog")Known Bad [v2.0.0-preview.26](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview26 "View this release in the dbt v2 changelog")GoodCanary Sep 25, 2025, 03:16 PM [v2.0.0-preview.25](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview25 "View this release in the dbt v2 changelog")GoodST MondayST WednesdayLatestST Thursday Sep 25, 2025, 10:03 PM [v2.0.0-preview.23](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview23 "View this release in the dbt v2 changelog")Known Bad [v2.0.0-preview-nightly.176](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200-preview-nightly176 "View this release in the dbt v2 changelog")GoodDevCanary Released by: **akbog**May 12, 2026, 08:21 PM Rollback incorrect release of 2.0.0 [v2.0.0](https://github.com/dbt-labs/dbt/blob/main/CHANGELOG-fusion.md#200 "View this release in the dbt v2 changelog")GoodDevnightlyCanary Released by: **mishamsk**Sep 10, 2026, 07:30 PM planned --- ### Supported features Learn about the features supported by dbt v2, including requirements and limitations. When you install dbt, you get v2 by default. There's no separate feature set to choose between — v2 is just dbt, running faster, with more capability built in. #### Requirements To use v2 in your project you must: * Use a supported adapter and authentication method:  BigQuery[Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") * Service Account / User Token * Native OAuth * External OAuth * [Workload Identity Federation](https://docs.getdbt.com/docs/platform/manage-access/set-up-bigquery-oauth.md#set-up-bigquery-workload-identity-federation) (Microsoft Entra) * [Required permissions](https://docs.getdbt.com/docs/local/connect-data-platform/bigquery-setup.md#required-permissions)  Databricks[Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") * Service Account / User Token * Native OAuth  Redshift[Preview](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") * Username / Password * IAM profile  Snowflake * Username / Password * Native OAuth * External OAuth * Key pair using a modern PKCS#8 method * MFA  Apache Spark (CLI only)[Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") * Thrift * Simple Authentication and Security Layer (SASL) PLAIN * No SASL (NOSASL) * Livy * Basic authentication (username and password) * When deployed on Amazon Web Services (AWS): AWS Signature Version 4 * Supports authentication using single sign-on, service accounts, or user tokens  DuckDB (CLI only)[Beta](https://docs.getdbt.com/docs/dbt-versions/product-lifecycles "Go to https://docs.getdbt.com/docs/dbt-versions/product-lifecycles") DuckDB does not require authentication — it runs locally on your machine. *Note that adapter lifecycle may differ between the dbt platform and local development. An adapter can reach GA in the dbt platform before it reaches GA for local use.* * Be able to run your project on the latest version of v1 with no deprecation warnings or errors. * Migrate your Semantic Layer configurations to the [latest YAML spec](https://docs.getdbt.com/docs/build/latest-metrics-spec.md). #### Parity with dbt v1 dbt v2 supports nearly all of v1.x's capabilities today. Refer to [Limitations](#limitations) below for the small number of gaps that remain. v2 has also removed some deprecated features and introduced more rigorous validation of erroneous project code compared to v1.x. Refer to the [Upgrade guide](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md) for details. #### Features and capabilities dbt v2 gives your team faster development workflows with semantic and syntax error detection, a faster linter, column-level lineage, language server and VS Code integration, docs v2 (full), and data diff. The dbt VS Code extension adds editor features like IntelliSense, hover info, and inline errors on top, powered by the LSP. Most v2 features work right away, with no login required. A few more unlock once you sign in with a dbt platform account — free to create, no paid plan needed. For the full free-vs-login breakdown, refer to [v2 feature availability](https://docs.getdbt.com/docs/dbt/dbt-availability.md). For LSP features specifically, refer to [About dbt LSP](https://docs.getdbt.com/docs/about-dbt-lsp.md). tip dbt platform [features](https://docs.getdbt.com/docs/platform/about-platform/dbt-platform-features.md) (like [Advanced CI](https://docs.getdbt.com/docs/deploy/advanced-ci.md), [dbt Mesh](https://docs.getdbt.com/docs/mesh/about-mesh.md), and more) are the enterprise layer on top of v2 — available no matter how you run dbt, depending on your [dbt plan](https://www.getdbt.com/pricing). #### Limitations If your project uses any of the following, you can still use v2, but full migration may not be possible yet: * Models that rely on materialization features v2 doesn't fully support, or that need configurations it's still missing * Tooling that depends on v1.x's exact log output * Workflows built around dbt platform features v2 doesn't yet support, like model-level notifications * Using the dbt VS Code extension in Cursor's Agent mode — lineage visualization only renders in Editor mode, so switch there if you need the full lineage tab * [Linting using SQLFluff](https://docs.getdbt.com/docs/deploy/continuous-integration.md#to-configure-sqlfluff-linting) — You use SQLFluff for linting in CI or local development. SQLFluff is not natively compatible with dbt v2 but we provide a [SQLFluff compatible high performance alternative](https://docs.getdbt.com/reference/commands/lint.md?version=2.0) with dbt v2. Support varies by where you run it: * **dbt platform CI jobs**: CI jobs on a dbt v2 version invoke [`dbt lint`](https://docs.getdbt.com/reference/commands/lint.md?version=2.0) instead of SQLFluff, so results can differ. Refer to [Rule parity with SQLFluff](https://docs.getdbt.com/reference/commands/lint.md?version=2.0#rule-parity-with-sqlfluff). * **Studio IDE**: SQLFluff linting works, but uses the dbt v1 engine templater rather than dbt v2. * **Local development**: You can run SQLFluff locally using the standalone dbt v1 engine templater as a workaround. A native dbt v2 linter is available with [`dbt lint` command](https://docs.getdbt.com/reference/commands/lint.md?version=2.0) #### Package support To determine if a package is compatible with dbt v2, visit the [dbt package hub](https://hub.getdbt.com/) and look for the dbt v2-compatible badge, or review the package's [`require-dbt-version` configuration](https://docs.getdbt.com/reference/project-configs/require-dbt-version.md#pin-to-a-range). * Packages with a `require-dbt-version` that equals or contains `2.0.0` are compatible with dbt v2. For example, `require-dbt-version: ">=1.10.0,<3.0.0"`. Even if a package doesn't reflect compatibility in the package hub, it may still work with v2. Work with package maintainers to track updates, and [thoroughly test packages](https://docs.getdbt.com/guides/dbt-package-compat?step=5) that aren't clearly compatible before deploying. * Package maintainers who would like to make their package compatible with v2 can refer to the [dbt v2 package upgrade guide](https://docs.getdbt.com/guides/dbt-package-compat.md) for instructions. Fivetran package considerations: * The Fivetran `source` and `transformation` packages have been combined into a single package. * If you manually installed source packages like `fivetran/github_source`, you need to ensure `fivetran/github` is installed and deactivate the transformation models. ###### Package compatibility messages Inconsistent v2 warnings and `dbt-autofix` logs dbt v2 warnings and `dbt-autofix` logs may show different messages about package compatibility. If you use [`dbt-autofix`](https://github.com/dbt-labs/dbt-autofix) while upgrading to v2 in the Studio IDE or dbt VS Code extension, you may see different messages about package compatibility between `dbt-autofix` and v2 warnings. Here's why: * dbt v2 warnings are emitted based on a package's `require-dbt-version` and whether `require-dbt-version` contains `2.0.0`. * Some packages are already v2-compatible even though package maintainers haven't yet updated `require-dbt-version`. * `dbt-autofix` knows about these compatible packages and will not try to upgrade a package that it knows is already compatible. This means that even if you see a v2 warning for a package that `dbt-autofix` identifies as compatible, you don't need to change the package. The message discrepancy is temporary while we implement and roll out `dbt-autofix`'s enhanced compatibility detection to v2 warnings. Here's an example of a v2 warning in the Studio IDE that says a package isn't compatible with v2 but `dbt-autofix` indicates it is compatible: ```text dbt1065: Package 'dbt_utils' requires dbt version [>=1.30,<2.0.0], but current version is 2.0.0-preview.72. This package may not be compatible with your dbt version. dbt(1065) [Ln 1, Col 1] ``` #### More information about dbt v2 * [About the dbt extension](https://docs.getdbt.com/docs/about-dbt-extension.md) * [Supported features matrix](https://docs.getdbt.com/docs/dbt/supported-features.md) * [Install dbt](https://docs.getdbt.com/docs/local/install-dbt.md) * [Quickstart for dbt v2](https://docs.getdbt.com/guides/dbt.md?step=1) * [Upgrade guide](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md) * [dbt v2 license agreement](https://www.getdbt.com/dbt-fusion-engine-license-agreement) --- ## Getting Started ### dbt platform configuration checklist dbt platform So, you've created a new cloud-hosted dbt platform account, and you're ready to explore its lightning-fast and intuitive features. Welcome! Before you begin, let’s ensure your account is properly configured so that you can easily onboard new users and take advantage of all the integrations dbt has to offer. For most organizations, this will require some collaboration with IT and/or security teams. Depending on the features you're using, you may need some of the following admin personas to help you get set up: * Data warehouse (Snowflake, BigQuery, Databricks, etc.) * Access control (Okta, Entra ID, Google, SAML 2.0) * Git (GitHub, GitLab, Azure DevOps, etc.) This checklist ensures you have everything in the right place, allowing you to deploy quickly and without any bottlenecks. #### Data warehouse The dbt platform supports [global connections](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md#connection-management) for your data warehouses. This means that a single configured connection can be used across multiple projects and environments. The dbt platform supports multiple data warehouse connections, including (but not limited to) BigQuery, Databricks, Redshift, and Snowflake. One of the earliest account configuration steps you'll want to take is ensuring you have a working connection: * \[ ] Use the [connection set up documentation](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md) to configure the data warehouse connection of your choice. * \[ ] Verify that dbt developers have proper roles and access in your data warehouse(s). * \[ ] Be sure the data warehouse has real data you can reference. This can be production or development data. We have a sandbox e-commerce project called [The Jaffle Shop](https://github.com/dbt-labs/jaffle-shop) that you can use if you prefer. The Jaffle Shop includes mock data and ready-to-run models! * \[ ] Whether starting a brand new project or importing an existing dbt v1 project, you'll want to make sure you have the [proper structure configured](https://docs.getdbt.com/docs/build/projects.md). * \[ ] If you are migrating from dbt v1, there are some important things you'll need to know, so check out our [migration guide](https://docs.getdbt.com/guides/dbt-migration-2.md?step=1). * \[ ] Your users will need to [configure their credentials](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md#get-started-with-the-studio-ide) to connect to the development environment in the dbt Studio IDE. * \[ ] Ensure that all users who need access to work in the IDE have a [developer license](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md) assigned in your account. * \[ ] dbt models are primarily written as [SELECT statements](https://docs.getdbt.com/docs/build/sql-models.md), so an early step for measuring success is having a developer run a simple select statement in the IDE and validating the results. * \[ ] You can also verify the connection by running basic SQL queries using [dbt Insights](https://docs.getdbt.com/docs/explore/access-dbt-insights.md). * \[ ] Create a single model and ensure that you can [run it](https://docs.getdbt.com/reference/dbt-commands.md) successfully. * \[ ] For an easy to use drag-and-drop interface, try creating it with [dbt Canvas](https://docs.getdbt.com/docs/platform/canvas.md). * \[ ] Create a service account with proper access for your [production jobs](https://docs.getdbt.com/docs/deploy/jobs.md). #### Git configuration Git is, for many dbt environments, the backbone of your project. Git repositories are where your dbt files will live and where your developers will collaborate and manage version control of your project. * \[ ] Configure a [Git repository](https://docs.getdbt.com/docs/platform/git/configure-git.md) for your account. dbt supports integrations with: * [GitHub](https://docs.getdbt.com/docs/platform/git/connect-github.md) * [GitLab](https://docs.getdbt.com/docs/platform/git/connect-gitlab.md) * [Azure DevOps](https://docs.getdbt.com/docs/platform/git/connect-azure-devops.md) * Other providers using [Git clone](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md) * If you aren't ready to integrate with an existing Git solution, dbt can provide you with a [managed Git repository](https://docs.getdbt.com/docs/platform/git/managed-repository.md). * \[ ] Ensure developers can [checkout](https://docs.getdbt.com/docs/platform/git/version-control-basics.md#git-overview) a new branch in your repo. * \[ ] Ensure developers in the IDE can [commit changes](https://docs.getdbt.com/docs/platform/studio-ide/ide-user-interface.md#basic-layout). #### Environments and jobs [Environments](https://docs.getdbt.com/docs/environments-in-dbt.md) separate your development data from your production data. dbt supports two environment types: Development and Deployment. There are three types of deployment environments: * Production - One per project * Staging - One per project * General - Multiple per project Additionally, you will have only one `Development` environment per project, but each developer will have their own unique access to the IDE, separate from the work of other developers. [Jobs](https://docs.getdbt.com/docs/deploy/jobs.md) dictate which commands are run in your environments and can be triggered manually, on a schedule, by other jobs, by APIs, or when pull requests are committed or merged. Once you connect your data warehouse and complete the Git integration, you can configure environments and jobs: * \[ ] Start by creating a new [Development environment](https://docs.getdbt.com/docs/dbt-platform-environments.md#create-a-development-environment) for your project. * \[ ] Create a [Production Deployment environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md). * \[ ] (Optional) Create an additional Staging or General environment. * \[ ] [Create and schedule](https://docs.getdbt.com/docs/deploy/deploy-jobs.md#create-and-schedule-jobs) a deployment job. * \[ ] Validate the job by manually running it first. * \[ ] If needed, configure different databases for your environments. #### User access The dbt platform offers a variety of access control tools that you can leverage to grant or revoke user access, configure RBAC, and assign user licenses and permissions. * \[ ] Manually [invite users](https://docs.getdbt.com/docs/platform/manage-access/invite-users.md) to the dbt platform, and they can authenticate using [MFA (SMS or authenticator app)](https://docs.getdbt.com/docs/platform/manage-access/mfa.md). * \[ ] Configure [single sign-on or OAuth](https://docs.getdbt.com/docs/platform/manage-access/sso-overview.md) for advanced access control. [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") accounts only. * \[ ] Create [SSO mappings](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#sso-mappings-) for groups * Configure [System for Cross-Domain Identity Management (SCIM)](https://docs.getdbt.com/docs/platform/manage-access/scim.md) if available for your IdP. * \[ ] Ensure invited users are able to connect to the data warehouse from their personal profile. * \[ ] [Create groups](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#create-new-groups-) with granular permission sets assigned. * \[ ] Create [RBAC rules](https://docs.getdbt.com/docs/platform/manage-access/about-user-access#role-based-access-control-) to assign users to groups and permission sets upon sign in. [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") accounts only. * \[ ] Enforce SSO for all non-admin users, and MFA is enforced for all password-based logins. #### Continue the journey Once you've completed this checklist, you're ready to start your dbt platform journey, but that journey has only just begun. Explore these additional resources to support you along the way: * \[ ] Review the [guides](https://docs.getdbt.com/guides.md) for quickstarts to help you get started with projects and features. * \[ ] Take a [dbt Learn](https://learn.getdbt.com/catalog) hands-on course. * \[ ] Review our [best practices](https://docs.getdbt.com/best-practices.md) for practical advice on structuring and deploying your dbt projects. * \[ ] Become familiar with the [references](https://docs.getdbt.com/reference/references-overview.md), as they are the product dictionary and offer detailed implementation examples. --- ### dbt Quickstarts Begin your dbt journey by choosing how you want to develop: * [**dbt platform** ](#the-dbt-platform)— Develop in your browser (Studio IDE or Canvas) or use local tools (VS Code extension, dbt platform CLI) that connect to your platform account. The platform provides hosted CI/CD, documentation, and more. Supports both the [dbt v2](https://docs.getdbt.com/docs/introduction.md) and [dbt v1](https://docs.getdbt.com/docs/local/install-dbt.md) engines. * [**Self-hosted only**](#self-hosted-dbt-installations) — Use local tools like the [dbt VS Code extension](https://docs.getdbt.com/docs/about-dbt-extension.md) or [install dbt](https://docs.getdbt.com/docs/local/install-dbt.md) locally, to develop and run dbt on your own infrastructure. You can use local tools with or without a dbt platform account. * **Local + dbt platform** — Use the VS Code extension or dbt platform CLI with a dbt platform account to develop locally while leveraging platform features like CI/CD, documentation hosting, Insights, Canvas, and more. * [**dbt Wizard**](#dbt-wizard) — The AI agent for analytics engineering, available in the dbt platform and from your terminal. Grounded in your project's lineage, model health, and semantic definitions. #### The dbt platform dbt provides a fully managed environment to develop, run, and deploy dbt projects—with CI/CD, documentation hosting, and more. Learn more about [dbt features](https://docs.getdbt.com/docs/platform/about-platform/dbt-platform-features.md) and [start your free trial](https://www.getdbt.com/signup/) today. dbt v2 adds managed execution and a unified development experience so you can focus on building rather than infrastructure. Choose your warehouse to get started with a quickstart: [![](/img/icons/athena.svg)](https://docs.getdbt.com/guides/athena) ###### [Quickstart for dbt and Amazon Athena](https://docs.getdbt.com/guides/athena) [Integrate dbt with Amazon Athena for your data transformations.](https://docs.getdbt.com/guides/athena) [![](/img/icons/azure-synapse-analytics-2.svg)](https://docs.getdbt.com/guides/azure-synapse-analytics) ###### [Quickstart for dbt and Azure Synapse Analytics](https://docs.getdbt.com/guides/azure-synapse-analytics) [Discover how to integrate dbt with Azure Synapse Analytics for your data transformations.](https://docs.getdbt.com/guides/azure-synapse-analytics) [![](/img/icons/bigquery.svg)](https://docs.getdbt.com/guides/bigquery) ###### [Quickstart for dbt and BigQuery](https://docs.getdbt.com/guides/bigquery) [Discover how to leverage dbt with BigQuery to streamline your analytics workflows.](https://docs.getdbt.com/guides/bigquery) [![](/img/icons/databricks.svg)](https://docs.getdbt.com/guides/databricks) ###### [Quickstart for dbt and Databricks](https://docs.getdbt.com/guides/databricks) [Learn how to integrate dbt with Databricks for efficient data processing and analysis.](https://docs.getdbt.com/guides/databricks) [![](/img/icons/fabric.svg)](https://docs.getdbt.com/guides/microsoft-fabric) ###### [Quickstart for dbt and Microsoft Fabric](https://docs.getdbt.com/guides/microsoft-fabric) [Explore the synergy between dbt and Microsoft Fabric to optimize your data transformations.](https://docs.getdbt.com/guides/microsoft-fabric) [![](/img/icons/redshift.svg)](https://docs.getdbt.com/guides/redshift) ###### [Quickstart for dbt and Redshift](https://docs.getdbt.com/guides/redshift) [Learn how to connect dbt to Redshift for more agile data transformations.](https://docs.getdbt.com/guides/redshift) [![](/img/icons/snowflake.svg)](https://docs.getdbt.com/guides/snowflake) ###### [Quickstart for dbt and Snowflake](https://docs.getdbt.com/guides/snowflake) [Unlock the full potential of using dbt with Snowflake for your data transformations.](https://docs.getdbt.com/guides/snowflake) [![](/img/icons/starburst.svg)](https://docs.getdbt.com/guides/starburst-galaxy) ###### [Quickstart for dbt and Starburst Galaxy](https://docs.getdbt.com/guides/starburst-galaxy) [Leverage dbt with Starburst Galaxy to enhance your data transformation workflows.](https://docs.getdbt.com/guides/starburst-galaxy) [![](/img/icons/teradata.svg)](https://docs.getdbt.com/guides/teradata) ###### [Quickstart for dbt and Teradata](https://docs.getdbt.com/guides/teradata) [Discover and use dbt with Teradata to enhance your data transformation workflows.](https://docs.getdbt.com/guides/teradata) #### Self-hosted dbt installations When you install dbt into your Windows, macOS, or Linux environment, you get command-line tools and the VS Code extension that enable you to transform data using analytics engineering best practices. You can use self-hosted tools with or without a dbt platform account. With an account, the VS Code extension and dbt platform CLI sync with your platform project for CI/CD, documentation, and more. Without an account, you run dbt entirely on your own infrastructure. [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/guides/dbt.md?step=2) ###### [Self-hosted dbt v2 from a manual install](https://docs.getdbt.com/guides/dbt.md?step=2) [Learn how to install self-hosted v2 and set up a project.](https://docs.getdbt.com/guides/dbt.md?step=2) [![](/img/icons/dbt-bit.svg)](https://docs.getdbt.com/guides/manual-install.md) ###### [Self-hosted dbt v1 from a manual install](https://docs.getdbt.com/guides/manual-install.md) [Learn how to install self-hosted v1 and set up a project.](https://docs.getdbt.com/guides/manual-install.md) [![](/img/icons/duckdb-seeklogo.svg)](https://docs.getdbt.com/guides/duckdb.md?step=1) ###### [Quickstart for dbt with DuckDB](https://docs.getdbt.com/guides/duckdb.md?step=1) [Learn how to connect dbt to DuckDB.](https://docs.getdbt.com/guides/duckdb.md?step=1) #### dbt Wizard [dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-overview.md) is an AI agent purpose-built for analytics engineering. It uses dbt's [native metadata engine](https://docs.getdbt.com/docs/dbt-ai/about-dbt-ai.md) — a structured index of your project's lineage, model health, tests, and semantic definitions — to build, refactor, validate, and document your project grounded in full project context. [![](/img/icons/dbt-copilot.svg)](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) ###### [dbt Wizard in the dbt platform](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) [Use dbt Wizard in the Studio IDE or home app to build and refactor models from natural language, generate tests and docs, and validate changes against your warehouse.](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) [![](/img/icons/dbt-copilot.svg)](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) ###### [dbt Wizard from your terminal](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) [Install the dbt Wizard CLI to run the agent locally against any dbt project — with or without a dbt platform plan. Start with a free trial using dbt managed AI, or bring your own provider key.](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) #### Related docs Expand your dbt knowledge and expertise with these additional resources: * [Join the monthly demos](https://www.getdbt.com/resources/webinars/dbt-cloud-demos-with-experts) to see dbt in action and ask questions. * [dbt AWS marketplace](https://aws.amazon.com/marketplace/pp/prodview-tjpcf42nbnhko) contains information on how to deploy dbt on AWS, user reviews, and more. * [Best practices](https://docs.getdbt.com/best-practices.md) contains information on how dbt Labs approaches building projects through our current viewpoints on structure, style, and setup. * [dbt Learn](https://learn.getdbt.com) offers free online courses that cover dbt fundamentals, advanced topics, and more. * [Join the dbt Community](https://www.getdbt.com/community/join-the-community) to learn how other data practitioners globally are using dbt, share your own experiences, and get help with your dbt projects. --- ### What is dbt? dbt transforms raw warehouse data into trusted data products. You write simple SQL select statements, and dbt handles the heavy lifting by creating modular, maintainable data models that power analytics, operations, and AI -- replacing the need for complex and fragile transformation code. dbt is the industry standard for data transformation, helping teams work faster and produce higher-quality data. As you build in dbt, your project creates structured context — lineage, tests, contracts, metrics, and governance — that explains how your data connects, what it means, and what changes may affect. That context makes dbt especially powerful for AI and comes with features like [dbt Wizard](#dbt-wizard), which helps you investigate, build, validate, and ship with full project context and governance on by default. You can use dbt and its [framework](#dbt-framework) to: * Centralize and modularize your analytics code, while also providing your data team with guardrails typically found in software engineering workflows. * Collaborate on data models to safely deploy and monitor data transformations in production. * Apply software engineering best practices like version control, testing, modularity, CI/CD, and documentation to analytics workflows. * Build idempotent transformations that are safe to rerun and produce consistent results. Learn more about [Idempotence in dbt](https://docs.getdbt.com/best-practices/idempotence.md). Backed by a 100,000+ member [community](https://docs.getdbt.com/community/join.md), dbt helps teams build high-quality, trustworthy data pipelines faster. [![dbt works alongside your ingestion, visualization, and other data tools, so you can transform data directly in your cloud data platform.](/img/docs/platform-overview.jpg?v=2 "dbt works alongside your ingestion, visualization, and other data tools, so you can transform data directly in your cloud data platform.")](#)dbt works alongside your ingestion, visualization, and other data tools, so you can transform data directly in your cloud data platform. Read more about why we want to enable analysts to work more like software engineers in [The dbt Viewpoint](https://docs.getdbt.com/community/resources/viewpoint.md). Learn how other data practitioners around the world are using dbt by [joining the dbt Community](https://www.getdbt.com/community/join-the-community). #### dbt framework Use the dbt framework to quickly and collaboratively transform data and deploy analytics code following software engineering best practices like version control, modularity, portability, CI/CD, and documentation. This means anyone on the data team familiar with SQL can safely contribute to production-grade data pipelines. The dbt framework is composed of a *language* and an *engine*: * The *dbt language* is the code you write in your dbt project — SQL select statements, Jinja templating, YAML configs, tests, and more. It's the standard for the data industry and the foundation of the dbt framework. * The *dbt engine* compiles your project, executes your transformation graph, and produces metadata. Today, the current Rust-based version generation is v2. By default, installing dbt gives you SQL comprehension, editor features, and richer development workflows. ##### dbt versions dbt has two major versions: v1 and v2. **v2** is the current generation of dbt and the default when you [install it](https://docs.getdbt.com/docs/local/install-dbt.md). Installing dbt comes with richer developer tooling, linting, and more. Refer to the [Upgrade v2 guide](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md) for more info. | Version | What it is | | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **v2**<br />*Recommended* | The current Rust-based generation of dbt, built for the full modern development experience and powered by an open-source runtime.<br /><br />Many features work right away. Some advanced capabilities unlock with a free sign-in — see [v2 feature availability](https://docs.getdbt.com/docs/dbt/dbt-availability.md) for the full breakdown or [upgrade to v2](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v2.md). | | **v1** | The original Python-based generation of dbt, still maintained as dbt v1. | Refer to the [Licensing FAQs](https://www.getdbt.com/licenses-faq) for more info. #### The dbt engine The current generation of dbt is written in Rust with a native understanding of SQL across multiple engine dialects. That comprehension lets dbt catch errors before they reach your warehouse and powers editor features like autocomplete and inline errors as you type. v2 is the default experience when you [install dbt](https://docs.getdbt.com/docs/local/install-dbt.md). It builds on the Apache 2.0 open-source runtime CLI foundation for the dbt framework. dbt is free to use, with some capabilities unlocked when you sign in with any dbt platform account. ##### Enhance your development workflows As a developer, dbt can: * Immediately catch incorrect SQL in your dbt models, before they ever hit the warehouse * Give you autocomplete, hover info, and inline errors as you type * Preview inline CTEs for faster debugging * Trace model and column definitions across your entire project Get all of this, free, in the [dbt extension for VS Code](https://docs.getdbt.com/docs/about-dbt-extension.md), built to enhance the power of v2, but also available in v1. #### How to use dbt You can use dbt in different ways depending on your needs: * [With the dbt platform](#dbt-platform) (recommended for most users) * [Locally from your command line or code editor](#dbt-local-development) ##### dbt platform The dbt platform is the fastest way to run dbt: scheduling, CI/CD, documentation hosting, monitoring, and alerting, all in one place. It works with both v1 and v2, on every plan from Developer (free) through Enterprise+. Develop directly in the platform with the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) or connect from your local machine with the dbt VS Code extension or dbt platform CLI. Learn more about [dbt platform features](https://docs.getdbt.com/docs/platform/about-platform/dbt-platform-features.md), explore [plans and pricing](https://www.getdbt.com/pricing/), or try a [quickstart](https://docs.getdbt.com/guides.md). ##### dbt local development [Install dbt](https://docs.getdbt.com/docs/local/install-dbt.md) to run v2 locally from the command line, powered by an open-source runtime. For the best development experience, we recommend pairing v2 with the [dbt VS Code extension](https://docs.getdbt.com/docs/about-dbt-extension.md) for autocomplete, inline errors, and lineage as you work. You can also run [`dbt login`](https://docs.getdbt.com/reference/commands/login.md?version=2.0) to unlock additional capabilities and create a free dbt platform account. To get started quickly, try the [dbt quickstart](https://docs.getdbt.com/guides/dbt.md). Other ways to run self-hosted dbt: * [dbt v1](https://docs.getdbt.com/docs/local/install-dbt.md?version=1.0): The original Python-based CLI. * [dbt OSS](https://docs.getdbt.com/docs/local/install-dbt-v2.md): The free, fully open-source (Apache 2.0) distribution of the new Rust-based dbt engine. Typically for organizations with a strict requirement to use this OSS runtime. To contribute to the open-source project, refer to the [GitHub repo](https://github.com/dbt-labs/dbt). #### Why use dbt As a dbt user, your main focus will be on writing models (select queries) that reflect core business logic – there's no need to write boilerplate code to create tables and views, or to define the order of execution of your models. Instead, dbt handles turning these models into objects in your warehouse for you. * **No boilerplate**: Write business logic with just a SQL `select` statement or a Python DataFrame. dbt handles materialization, transactions, DDL, and schema changes. * **Modular and reusable**: Build data models that can be referenced in subsequent work. Change a model once and the change propagates to all its dependencies, so you can publish canonical business logic without reimplementing it. * **Fast builds**: Use [incremental models](https://docs.getdbt.com/docs/build/incremental-models.md) and leverage metadata to optimize long-running models. * **Tested and documented** — Write [data quality tests](https://docs.getdbt.com/docs/build/data-tests.md) on your underlying data and auto-generate [documentation](https://docs.getdbt.com/docs/build/documentation.md) alongside your code. * **Software engineering workflows**: Version control, branching, pull requests, CI/CD, and [package management](https://docs.getdbt.com/docs/build/packages.md) for your data pipelines. Write DRYer code with [macros](https://docs.getdbt.com/docs/build/jinja-macros.md) and [hooks](https://docs.getdbt.com/docs/build/hooks-operations.md). * **AI-powered development**: Use [dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-overview.md) to investigate, build, validate, and ship from natural language. dbt Wizard is grounded in your project's full context, validates its own work against lineage and tests, and includes governance and audit trails by default. #### Related docs * [Quickstarts for dbt](https://docs.getdbt.com/guides.md) * [Best practice guides](https://docs.getdbt.com/best-practices.md) * [What is a dbt project?](https://docs.getdbt.com/docs/build/projects.md) * [Supported features matrix](https://docs.getdbt.com/docs/dbt/supported-features.md) * [AI and agents](https://docs.getdbt.com/docs/dbt-ai/about-dbt-ai.md) * [Licensing](https://docs.getdbt.com/docs/dbt-licensing.md) * [v2 license agreement](https://www.getdbt.com/dbt-fusion-engine-license-agreement) --- ## Guides ### Airflow and dbt [Back to guides](https://docs.getdbt.com/guides.md) dbt platform Orchestration Intermediate [Menu ]() #### Introduction Many organizations already use [Airflow](https://airflow.apache.org/) to orchestrate their data workflows. dbt works great with Airflow, letting you execute your dbt code in dbt while keeping orchestration duties with Airflow. This ensures your project's metadata (important for tools like Catalog) is available and up-to-date, while still enabling you to use Airflow for general tasks such as: * Scheduling other processes outside of dbt runs * Ensuring that a [dbt job](https://docs.getdbt.com/docs/deploy/job-scheduler.md) kicks off before or after another process outside of dbt * Triggering a dbt job only after another has completed In this guide, you'll learn how to: 1. Create a working local Airflow environment 2. Invoke a dbt job with Airflow 3. Reuse tested and trusted Airflow code for your specific use cases You’ll also gain a better understanding of how this will: * Reduce the cognitive load when building and maintaining pipelines * Avoid dependency hell (think: `pip install` conflicts) * Define clearer handoff of workflows between data engineers and analytics engineers #### Prerequisites * [dbt platform Enterprise or Enterprise+ account](https://www.getdbt.com/pricing/) (with [admin access](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md)) in order to create a service token. Permissions for service tokens can be found [here](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md#permissions-for-service-account-tokens). * Service account token if you have [Developer license (with admins access)](https://docs.getdbt.com/docs/platform/manage-access/enterprise-permissions.md)). Permissions for service tokens can be found [here](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md#permissions-for-service-account-tokens) and [here](https://docs.getdbt.com/docs/platform/manage-access/seats-and-users.md?version=2.0). * [Personal access token (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens#create-a-personal-access-token) if you have developer, IT, or Read-only license. * A [free Docker account](https://hub.docker.com/signup) in order to sign in to Docker Desktop, which will be installed in the initial setup. * A local digital scratchpad for temporarily copy-pasting API keys and URLs 🙌 Let’s get started! 🙌 #### Install the Astro CLI Astro is a managed software service that includes key features for teams working with Airflow. In order to use Astro, we’ll install the Astro CLI, which will give us access to useful commands for working with Airflow locally. You can read more about Astro [here](https://docs.astronomer.io/astro/). In this example, we’re using Homebrew to install Astro CLI. Follow the instructions to install the Astro CLI for your own operating system [here](https://docs.astronomer.io/astro/install-cli). ```bash brew install astro ``` #### Install and start Docker Desktop Docker allows us to spin up an environment with all the apps and dependencies we need for this guide. Follow the instructions [here](https://docs.docker.com/desktop/) to install Docker desktop for your own operating system. Once Docker is installed, ensure you have it up and running for the next steps. #### Clone the airflow-dbt-cloud repository Open your terminal and clone the [airflow-dbt-cloud repository](https://github.com/dbt-labs/airflow-dbt-cloud). This contains example Airflow DAGs that you’ll use to orchestrate your dbt job. Once cloned, navigate into the `airflow-dbt-cloud` project. ```bash git clone https://github.com/dbt-labs/airflow-dbt-cloud.git cd airflow-dbt-cloud ``` For more information about cloning GitHub repositories, refer to "[Cloning a repository](https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository)" in the GitHub documentation. #### Start the Docker container 1. From the `airflow-dbt-cloud` directory you cloned and opened in the prior step, run the following command to start your local Airflow deployment: ```bash astro dev start ``` When this finishes, you should see a message similar to the following: ```bash Airflow is starting up! This might take a few minutes… Project is running! All components are now available. Airflow Webserver: http://localhost:8080 Postgres Database: localhost:5432/postgres The default Airflow UI credentials are: admin:admin The default Postgres DB credentials are: postgres:postgres ``` 2. Open the Airflow interface. Launch your web browser and navigate to the address for the **Airflow Webserver** from your output above (for us, `http://localhost:8080`). This will take you to your local instance of Airflow. You’ll need to log in with the **default credentials**: * Username: admin * Password: admin ![Airflow login screen](/assets/images/airflow-login-56d38c8b37cf6d5cfe9672e8274a2d19.png) #### Create a dbt service token [Create a service token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) with `Job Admin` privileges from within dbt. Ensure that you save a copy of the token, as you won’t be able to access this later. As an alternative, you can [create a PAT](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md#create-a-personal-access-token) if you have a Developer, IT, or Read-only license. #### Create a dbt job [Create a job in your dbt account](https://docs.getdbt.com/docs/deploy/deploy-jobs.md#create-and-schedule-jobs), paying special attention to the information in the bullets below. * Configure the job with the full commands that you want to include when this job kicks off. This sample code has Airflow triggering the dbt job and all of its commands, instead of explicitly identifying individual models to run from inside of Airflow. * Ensure that the schedule is turned **off** since we’ll be using Airflow to kick things off. * Once you hit `save` on the job, make sure you copy the URL and save it for referencing later. The url will look similar to this: ```html https://YOUR_ACCESS_URL/#/accounts/{account_id}/projects/{project_id}/jobs/{job_id}/ ``` #### Connect dbt to Airflow Now you have all the working pieces to get up and running with Airflow + dbt. It's time to **set up a connection** and **run a DAG in Airflow** that kicks off a dbt job. 1. From the Airflow interface, navigate to Admin and click on **Connections** ![Airflow connections menu](/assets/images/airflow-connections-menu-71e1784305b7249eba5892141dde3d98.png) 2. Click on the `+` sign to add a new connection, then click on the drop down to search for the dbt Connection Type. ![Connection type](/assets/images/connection-type-42874161269a9c455e01c5734c4aaf64.png) 3. Add in your connection details and your default dbt account id. This is found in your dbt URL after the accounts route section (`/accounts/{YOUR_ACCOUNT_ID}`), for example the account with id 16173 would see this in their URL: `https://YOUR_ACCESS_URL/#/accounts/16173/projects/36467/jobs/65767/` ![Connection type](/assets/images/connection-type-configured-f0be2b2ee60d8192f0cf0bbeac03528b.png) #### Update the placeholders in the sample code Add your `account_id` and `job_id` to the python file [run\_dbt\_cloud\_job.py](https://github.com/dbt-labs/airflow-dbt-cloud/blob/main/dags/run_dbt_cloud_job.py). Both IDs are included inside of the dbt job URL as shown in the following snippets: ```python # For the dbt Job URL https://YOUR_ACCESS_URL/#/accounts/16173/projects/36467/jobs/65767/ # The account_id is 16173 and the job_id is 65767 # Update lines 34 and 35 ACCOUNT_ID = "16173" JOB_ID = "65767" ``` #### Run the Airflow DAG Turn on the DAG and trigger it to run. Verify the job succeeded after running. ![Airflow DAG](/assets/images/airflow-dag-d7d6a6fe556ac6e8a7970ae7305a5bc3.png) Click **Monitor Job Run** to open the run details in dbt. ![Task run instance](/assets/images/task-run-instance-936ac2e4ef47727b434363656900a99d.png) #### Cleaning up At the end of this guide, make sure you shut down your docker container. When you’re done using Airflow, use the following command to stop the container: ```bash $ astrocloud dev stop [+] Running 3/3 ⠿ Container airflow-dbt-cloud_e3fe3c-webserver-1 Stopped 7.5s ⠿ Container airflow-dbt-cloud_e3fe3c-scheduler-1 Stopped 3.3s ⠿ Container airflow-dbt-cloud_e3fe3c-postgres-1 Stopped 0.3s ``` To verify that the deployment has stopped, use the following command: ```bash astrocloud dev ps ``` This should give you an output like this: ```bash Name State Ports airflow-dbt-cloud_e3fe3c-webserver-1 exited airflow-dbt-cloud_e3fe3c-scheduler-1 exited airflow-dbt-cloud_e3fe3c-postgres-1 exited ``` #### Frequently asked questions ##### How can we run specific subsections of the dbt DAG in Airflow? Because the Airflow DAG references dbt jobs, your analytics engineers can take responsibility for configuring the jobs in dbt. For example, to run some models hourly and others daily, there will be jobs like `Hourly Run` or `Daily Run` using the commands `dbt run --select tag:hourly` and `dbt run --select tag:daily` respectively. Once configured in dbt, these can be added as steps in an Airflow DAG as shown in this guide. Refer to our full [node selection syntax docs here](https://docs.getdbt.com/reference/node-selection/syntax.md). ##### How can I re-run models from the point of failure? You can trigger re-run from point of failure with the `rerun` API endpoint. See the docs on [retrying jobs](https://docs.getdbt.com/docs/deploy/retry-jobs.md) for more information. ##### Should Airflow run one big dbt job or many dbt jobs? dbt jobs are most effective when a build command contains as many models at once as is practical. This is because dbt manages the dependencies between models and coordinates running them in order, which ensures that your jobs can run in a highly parallelized fashion. It also streamlines the debugging process when a model fails and enables re-run from point of failure. As an explicit example, it's not recommended to have a dbt job for every single node in your DAG. Try combining your steps according to desired run frequency, or grouping by department (finance, marketing, customer success...) instead. ##### We want to kick off our dbt jobs after our ingestion tool (such as Fivetran) / data pipelines are done loading data. Any best practices around that? Astronomer's DAG registry has a sample workflow combining Fivetran, dbt and Census [here](https://registry.astronomer.io/dags/fivetran-dbt_cloud-census/versions/3.0.0). ##### How do you set up a CI/CD workflow with Airflow? Check out these two resources for accomplishing your own CI/CD pipeline: * [Continuous Integration with dbt](https://docs.getdbt.com/docs/deploy/continuous-integration.md) * [Astronomer's CI/CD Example](https://docs.astronomer.io/software/ci-cd/#example-cicd-workflow) ##### Can dbt dynamically create tasks in the DAG like Airflow can? As discussed above, we prefer to keep jobs bundled together and containing as many nodes as are necessary. If you must run nodes one at a time for some reason, then review [this article](https://www.astronomer.io/blog/airflow-dbt-1/) for some pointers. ##### Can you trigger notifications if a dbt job fails with Airflow? Yes, either through [Airflow's email/slack](https://www.astronomer.io/guides/error-notifications-in-airflow/) functionality, or [dbt's notifications](https://docs.getdbt.com/docs/deploy/job-notifications.md), which support email and Slack notifications. You could also create a [webhook](https://docs.getdbt.com/docs/deploy/webhooks.md). ##### How should I plan my dbt + Airflow implementation? Check out [this recording](https://www.youtube.com/watch?v=n7IIThR8hGk) of a dbt meetup for some tips. --- ### Analyze your data in dbt Start with a stakeholder question and analyze the data to answer that question without writing any SQL [Back to guides](https://docs.getdbt.com/guides.md) Analyst dbt platform Quickstart [Menu ]() #### Introduction As a data analyst, you play a key role in transforming complex data into trusted, actionable insights for your team. With dbt, you can use built-in, AI-powered tools to build governed data models, explore how they’re built, and even run your own analysis. In this quickstart, you’ll learn how to: * Use Catalog to browse and understand data models across both dbt and Snowflake data assets * Use Insights to run queries for exploring and validating your data * Use Canvas to visually build your own data models * Build confidence using dbt as your workspace enhanced with AI Here's more about the tools you will use on your journey: * Catalog: View your project's resources (such as models, tests, and metrics), their lineage, and query patterns to gain a better understanding of its latest production state. * Insights: Explore, validate, and query data with an intuitive, context-rich interface that bridges technical and business users by combining metadata, documentation, AI-assisted tools, and powerful querying capabilities. * Canvas: Quickly access and transform data through a visual, drag-and-drop experience and with a built-in AI for custom code generation. #### Prerequisites Before you begin, make sure: * You have access to and credentials configured for a dbt project * Your team has already run a successful dbt job, so models are built and ready * You have a a git provider connected and authenticated #### Analyst workflows Kimiko, an analyst at the Jaffle Shop, notices they've been doing a lot of new sales and wants to investigate the most critical data they have in their warehouse. **Question: A stakeholder is curious how many customers you've acquired month by month, in the last 12 months.** Kimiko wonders, "How do I find data in our project that will help me answer their question?" ##### Explore a stakeholder question She navigates to the data catalog, Catalog, by signing into dbt and clicking Catalog in the left panel. Because the question was about customers, Kimiko begins by searching for "customers" in Catalog: [![Catalog search for customers](/img/guides/analyst-qs/catalog-search.png?v=2 "Catalog search for customers")](#)Catalog search for customers She finds a "customers" model, which might be what she needs. She clicks **customers** to open the model. The description reads, “Customer overview data Mart offering key details for each unique customer, one row per customer.” Next, Kimiko selects **Columns** to see which columns this model uses. [![Columns in customers table](/img/guides/analyst-qs/columns.png?v=2 "Columns in customers table")](#)Columns in customers table She notices these columns: `customer_ID`, `customer_names`, and `first_ordered_at`. The `first_ordered_at` column stands out to Kimiko, and she wonders if she might use it to see how many customers they've acquired based on when they placed their first order. But first, she decides to interact with the data to learn more. ##### Query data in Insights From the **Customer model page** in Catalog, Kimiko selects **Analyze data** from the **Open in...** dropdown. This enables her to query data for the Customer model. Once opened, Insights contains a query poised and ready to run. [![Select Analyze data from Catalog](/img/guides/analyst-qs/catalog-analyze-data.png?v=2 "Select Analyze data from Catalog")](#)Select Analyze data from Catalog [![Open query in Insights](/img/guides/analyst-qs/query.png?v=2 "Open query in Insights")](#)Open query in Insights When Kimiko runs the query, she can look at the data underyling it. The same context she saw in Catalog she now sees in her SQL editing experience. As she looks through the data, she sees information about each customer. She also notices the `first_ordered_at` column. Kimiko wants to code the query but her SQL is a little rusty so she uses natural language in dbt Wizard: *How many new customers did we get in each month last year? I'd like to use my customer model and the first ordered at field to do this analysis.* dbt Wizard writes SQL that Kimiko decides to use: ```sql select date_trunc('month', first_ordered_at) as month, count(customer_id) as new_customers from {{ ref('customers') }} where date_part('year', first_ordered_at) = date_part('year', current_date) - 1 and customer_type = 'new' group by 1 order by 1; ``` Kimiko clicks **Replace** to move all of the SQL into her editor and replaces the original query. She runs the new query and reviews the data but decides to limit the dates using dbt Wizard once again: *Can we limit the dates to 2024?* She verifies the new filter for 2024 and reruns this query: ```sql select date_trunc('month', first_ordered_at) as month, count(customer_id) as new_customers from {{ ref('customers') }} where date_part('year', first_ordered_at) = 2024 and customer_type = 'new' group by 1 order by 1; ``` She's happy with the results and clicks **Details** to see the AI-generated report, which includes a title and description, supplied SQL, and the compiled SQL. [![Details report tabe](/img/guides/analyst-qs/details.png?v=2 "Details report tabe")](#)Details report tabe Once she's ready to get the insight to her stakeholder, she clicks **Chart** to view the chart prefilled with the data from the **Data** tab. She adds x- and y-axis labels, such as "Month of first order" and "Total new customers" to make it more comprehensible for the final report she'll share with her stakeholder. Next, she takes a screenshot to share with them. She often comes back to this data so Kimiko decides to bookmark the page by clicking **Bookmark** in the top right. She also exports it to a CSV file. ##### Visualize results Kimiko has a few conversations with teammates and she finds out they're running pretty similar one-off queries, so she decides to take her long running query that she previously bookmarked and turn it into a full-fledged dbt model using Canvas. She does this so she can share it with others, which de-duplicates work and helps her team become more efficient. To do this, she opens the query in Insights and clicks **Develop** then \***Develop in** Canvas. This opens the SQL query in a visual form, represented in a DAG. When she examines the model, she notes it's selecting from customers as expected, filtering for 2024, showing dates by month, and aggregating over that month. She runs it in her development environment and clicks **Commit** to submit a pull request. Now Kimiko's entire team, those who have the same access as her, can run this model and see the same results she does! What's more is they can help her improve the model as the stakeholder requests get more complicated, and she will benefit from their help. ##### The query becomes a model Going forward, Kimiko is able to return to her project in Catalog and run the model to get the most current results. From here, she can: * Manually run the model, which also runs tests and is versioned so Kimiko can track changes over time * Trigger a scheduled job to run the dbt model, like every Monday for her stakeholder report * Set up a Slack notification in case the job fails so she can recitfy any problems --- ### Browse our guides [Skip to main content](#__docusaurus_skipToContent_fallback) [Join us at dbt Summit, September 15–18 in Las Vegas. Come learn, swap ideas, and spend time with the dbt community as we shape the future of data and AI.](https://www.getdbt.com/dbt-summit/?utm_medium=internal\&utm_source=docs\&utm_campaign=q3-2027_dbt-summit-2026_aw\&utm_content=dbt-summit____\&utm_term=all_all__) [![dbt Logo](/img/dbt-logo.svg?v=2)![dbt Logo](/img/dbt-logo-light.svg?v=2)](https://docs.getdbt.com/index.md) * [v2]() * [v1]() * [1.12]() * [1.11]() * On dbt platform? <br /> v2 = [dbt v2 release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks?version=2.0#dbt-v2-release-tracks) <br /> v1 = [dbt v1 release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks?version=2.0#dbt-v1-release-tracks) [Docs](#) * [Product docs](https://docs.getdbt.com/docs/introduction.md) * [References](https://docs.getdbt.com/reference/references-overview.md) * [Best practices](https://docs.getdbt.com/best-practices.md) * [Developer blog](https://docs.getdbt.com/blog) [Guides](https://docs.getdbt.com/guides.md)[APIs](https://docs.getdbt.com/docs/dbt-apis/overview.md) [Help](#) * [Release notes](https://docs.getdbt.com/docs/dbt-versions/release-notes.md) * [FAQs](https://docs.getdbt.com/docs/faqs.md) * [Support and billing](https://docs.getdbt.com/docs/dbt-support.md) * [Licensing](https://docs.getdbt.com/docs/dbt-licensing.md) * [Courses](https://learn.getdbt.com) [Community](#) * [Join the dbt Community](https://docs.getdbt.com/community/join.md) * [Become a contributor](https://docs.getdbt.com/community/contribute.md) * [Community forum](https://discourse.getdbt.com/) * [Webinars](https://www.getdbt.com/resources/webinars) * [Events](https://www.getdbt.com/events) [Account](#) * [Log in to dbt](https://login.dbt.com/) * [Create a free account](https://www.getdbt.com/signup) [Get started with dbt](https://docs.getdbt.com/docs/dbt/get-started-dbt.md?version=2) Search The dbt platform is the fastest and most reliable way to deploy dbt for scalable data transformation, while dbt v1 powers open-source transformation workflows. Together, they provide a seamless analytics engineering experience. Explore our step-by-step guides, quickstart tutorials, and troubleshooting resources to get started with dbt and your data platform. Search guides... ##### Filter by ▼ Choose a topic \[ ]Adapter creation\[ ]AI\[ ]Amazon \+ View more (36)Select all Choose a level \[ ]Advanced\[ ]Beginner\[ ]Intermediate Select all Clear all ##### Get started with dbt v2 [![](/img/icons/zap.svg)](https://docs.getdbt.com/guides/dbt.md) ###### [Quickstart for dbt v2](https://docs.getdbt.com/guides/dbt.md) [Start ](https://docs.getdbt.com/guides/dbt.md) [![](/img/icons/zap.svg)](https://docs.getdbt.com/guides/prepare-v2-upgrade.md) ###### [Upgrade to dbt v2 part 1: Preparing to upgrade](https://docs.getdbt.com/guides/prepare-v2-upgrade.md) [Start ](https://docs.getdbt.com/guides/prepare-v2-upgrade.md) [![](/img/icons/zap.svg)](https://docs.getdbt.com/guides/upgrade-to-v2.md) ###### [Upgrade to dbt v2 part 2: Making the move](https://docs.getdbt.com/guides/upgrade-to-v2.md) [Start ](https://docs.getdbt.com/guides/upgrade-to-v2.md) [![](/img/icons/zap.svg)](https://docs.getdbt.com/guides/dbt-platform-local-workflow.md) ###### [Hybrid development with dbt platform and dbt v2](https://docs.getdbt.com/guides/dbt-platform-local-workflow.md) [Start ](https://docs.getdbt.com/guides/dbt-platform-local-workflow.md) [![](/img/icons/guides.svg)](https://docs.getdbt.com/guides/adapter-creation-v2.md) ###### [Contribute a dbt v2 adapter](https://docs.getdbt.com/guides/adapter-creation-v2.md) [Start ](https://docs.getdbt.com/guides/adapter-creation-v2.md) ##### Foundations [![](/img/icons/guides.svg)](https://docs.getdbt.com/guides/terminal-guide.md) ###### [Getting started with the terminal](https://docs.getdbt.com/guides/terminal-guide.md) [Start ](https://docs.getdbt.com/guides/terminal-guide.md) [![](/img/icons/guides.svg)](https://docs.getdbt.com/guides/clone-jaffle-shop.md) ###### [Clone the Jaffle Shop sample project](https://docs.getdbt.com/guides/clone-jaffle-shop.md) [Start ](https://docs.getdbt.com/guides/clone-jaffle-shop.md) ##### Popular [![](/img/icons/guides.svg)](https://docs.getdbt.com/guides/migrate-off-legacy-dbt-versions.md) ###### [Migrate off legacy dbt versions](https://docs.getdbt.com/guides/migrate-off-legacy-dbt-versions.md) [Start ](https://docs.getdbt.com/guides/migrate-off-legacy-dbt-versions.md) [![](/img/icons/snowflake.svg)](https://docs.getdbt.com/guides/snowflake.md) ###### [Quickstart for dbt and Snowflake](https://docs.getdbt.com/guides/snowflake.md) [dbt v2 compatible](https://docs.getdbt.com/guides/snowflake.md) [Start ](https://docs.getdbt.com/guides/snowflake.md) [![](/img/icons/databricks.svg)](https://docs.getdbt.com/guides/databricks.md) ###### [Quickstart for dbt and Databricks](https://docs.getdbt.com/guides/databricks.md) [dbt v2 compatible](https://docs.getdbt.com/guides/databricks.md) [Start ](https://docs.getdbt.com/guides/databricks.md) [![](/img/icons/bigquery.svg)](https://docs.getdbt.com/guides/bigquery.md) ###### [Quickstart for dbt and BigQuery](https://docs.getdbt.com/guides/bigquery.md) [dbt v2 compatible](https://docs.getdbt.com/guides/bigquery.md) [Start ](https://docs.getdbt.com/guides/bigquery.md) [![](/img/icons/redshift.svg)](https://docs.getdbt.com/guides/redshift.md) ###### [Quickstart for dbt and Redshift](https://docs.getdbt.com/guides/redshift.md) [dbt v2 compatible](https://docs.getdbt.com/guides/redshift.md) [Start ](https://docs.getdbt.com/guides/redshift.md) ##### Troubleshooting [![](/img/icons/guides.svg)](https://docs.getdbt.com/guides/debug-schema-names.md) ###### [Debug schema names](https://docs.getdbt.com/guides/debug-schema-names.md) [Start ](https://docs.getdbt.com/guides/debug-schema-names.md) [![](/img/icons/guides.svg)](https://docs.getdbt.com/guides/using-jinja.md) ###### [Use Jinja to improve your SQL code](https://docs.getdbt.com/guides/using-jinja.md) [Start ](https://docs.getdbt.com/guides/using-jinja.md) [![](/img/icons/guides.svg)](https://docs.getdbt.com/guides/debug-errors.md) ###### [Debug errors](https://docs.getdbt.com/guides/debug-errors.md) [Start ](https://docs.getdbt.com/guides/debug-errors.md) ##### Advanced use cases [![](/img/icons/zap.svg)](https://docs.getdbt.com/guides/iceberg.md) ###### [Get hands-on with Snowflake Horizon and Apache Iceberg](https://docs.getdbt.com/guides/iceberg.md) [60 minutesStart ](https://docs.getdbt.com/guides/iceberg.md) [![](/img/icons/zap.svg)](https://docs.getdbt.com/guides/dbt-package-compat.md) ###### [dbt v2 package upgrade guide](https://docs.getdbt.com/guides/dbt-package-compat.md) [Start ](https://docs.getdbt.com/guides/dbt-package-compat.md) [![](/img/icons/zap.svg)](https://docs.getdbt.com/guides/qs-mcp-oauth.md) ###### [Connect dbt MCP to dbt platform](https://docs.getdbt.com/guides/qs-mcp-oauth.md) [Start ](https://docs.getdbt.com/guides/qs-mcp-oauth.md) [![](/img/icons/zap.svg)](https://docs.getdbt.com/guides/qs-mcp-local.md) ###### [Run dbt MCP locally](https://docs.getdbt.com/guides/qs-mcp-local.md) [Start ](https://docs.getdbt.com/guides/qs-mcp-local.md) [![](/img/icons/guides.svg)](https://docs.getdbt.com/guides/airflow-and-dbt-cloud.md) ###### [Airflow and dbt](https://docs.getdbt.com/guides/airflow-and-dbt-cloud.md) [Start ](https://docs.getdbt.com/guides/airflow-and-dbt-cloud.md) [![](/img/icons/guides.svg)](https://docs.getdbt.com/guides/adapter-creation.md) ###### [Build, test, document, and promote adapters](https://docs.getdbt.com/guides/adapter-creation.md) [Start ](https://docs.getdbt.com/guides/adapter-creation.md) [![](/img/icons/guides.svg)](https://docs.getdbt.com/guides/adapter-creation-v2.md) ###### [Contribute a dbt v2 adapter](https://docs.getdbt.com/guides/adapter-creation-v2.md) [Start ](https://docs.getdbt.com/guides/adapter-creation-v2.md) [![](/img/icons/guides.svg)](https://docs.getdbt.com/guides/dbt-migration-1.md) ###### [Move to the dbt platform: Get started](https://docs.getdbt.com/guides/dbt-migration-1.md) [Total estimated time: 3-4 hoursStart ](https://docs.getdbt.com/guides/dbt-migration-1.md) [![](/img/icons/guides.svg)](https://docs.getdbt.com/guides/create-new-materializations.md) ###### [Create new materializations](https://docs.getdbt.com/guides/create-new-materializations.md) [Start ](https://docs.getdbt.com/guides/create-new-materializations.md) [![](/img/icons/guides.svg)](https://docs.getdbt.com/guides/customize-schema-alias.md) ###### [Customize dbt models database, schema, and alias](https://docs.getdbt.com/guides/customize-schema-alias.md) [Start ](https://docs.getdbt.com/guides/customize-schema-alias.md) [![](/img/icons/guides.svg)](https://docs.getdbt.com/guides/refactoring-legacy-sql.md) ###### [Refactoring legacy SQL to dbt](https://docs.getdbt.com/guides/refactoring-legacy-sql.md) [Start ](https://docs.getdbt.com/guides/refactoring-legacy-sql.md) Get started #### Start building with dbt. The free dbt VS Code extension is the best way to develop locally with dbt. [Install free extension](https://marketplace.visualstudio.com/items?itemName=dbtLabsInc.dbt) [Request your demo](https://www.getdbt.com/contact) [![dbt Labs](/img/dbt-logo-light.svg?v=2)](https://docs.getdbt.com/index.md) ###### Resources [VS Code Extension](https://docs.getdbt.com/docs/about-dbt-extension.md) [Resource Hub](https://www.getdbt.com/resources) [dbt Learn](https://www.getdbt.com/dbt-learn) [Certification](https://www.getdbt.com/dbt-certification) [Developer Blog](https://docs.getdbt.com/blog) ###### Community [Join the Community](https://docs.getdbt.com/community/join.md) [Become a Contributor](https://docs.getdbt.com/community/contribute.md) [Open Source dbt Packages](https://hub.getdbt.com/) [Community Forum](https://discourse.getdbt.com/) ###### Support [Contact Support](https://docs.getdbt.com/docs/dbt-support.md) [Professional Services](https://www.getdbt.com/services) [Find a Partner](https://www.getdbt.com/partner-directory) [System Status](https://status.getdbt.com/) ###### Connect with Us [](https://github.com/dbt-labs/docs.getdbt.com "GitHub") [](https://www.linkedin.com/company/dbtlabs/mycompany/ "LinkedIn") [](https://www.youtube.com/channel/UCVpBwKK-ecMEV75y1dYLE5w "YouTube") [](https://www.instagram.com/dbt_labs/ "Instagram") [](https://x.com/dbt_labs "X") [](https://bsky.app/profile/getdbt.com "Bluesky") [](https://www.getdbt.com/community/join-the-community/ "Community Slack") © 2026 dbt Labs, LLC. All Rights Reserved. [Terms of Service](https://www.getdbt.com/terms-of-use/) [Privacy Policy](https://www.getdbt.com/cloud/privacy-policy/) [Security](https://www.getdbt.com/security/) Cookie Settings --- ### Browse our guides [Back to guides](https://docs.getdbt.com/guides.md) [Menu ]() --- ### Build a data lakehouse with dbt v1 and Dremio Cloud [Back to guides](https://docs.getdbt.com/guides.md) Dremio dbt v1 Intermediate [Menu ]() #### Introduction This guide will demonstrate how to build a data lakehouse with dbt 1.5 or newer and Dremio Cloud. You can simplify and optimize your data infrastructure with dbt's robust transformation framework and Dremio’s open and easy data lakehouse. The integrated solution empowers companies to establish a strong data and analytics foundation, fostering self-service analytics and enhancing business insights while simplifying operations by eliminating the necessity to write complex Extract, Transform, and Load (ETL) pipelines. ##### Prerequisites * You must have a [Dremio Cloud](https://docs.dremio.com/cloud/) account. * You must have Python 3 installed. * You must have dbt v1.5 or newer [installed](//docs/local/install-dbt). * You must have the Dremio adapter 1.5.0 or newer [installed and configured](https://docs.getdbt.com/docs/local/connect-data-platform/dremio-setup.md) for Dremio Cloud. * You must have basic working knowledge of Git and the command line interface (CLI). #### Validate your environment Validate your environment by running the following commands in your CLI and verifying the results: ```shell $ python3 --version Python 3.11.4 # Must be Python 3 ``` ```shell $ dbt --version Core: - installed: 1.5.0 # Must be 1.5 or newer - latest: 1.6.3 - Update available! Your version of dbt-core is out of date! You can find instructions for upgrading here: https://docs.getdbt.com/docs/installation Plugins: - dremio: 1.5.0 - Up to date! # Must be 1.5 or newer ``` #### Getting started 1. Clone the Dremio dbt v1 sample project from the [GitHub repo](https://github.com/dremio-brock/DremioDBTSample/tree/master/dremioSamples). 2. In your integrated development environment (Studio IDE), open the relation.py file in the Dremio adapter directory: `$HOME/Library/Python/3.9/lib/python/site-packages/dbt/adapters/dremio/relation.py` 3. Find and update lines 51 and 52 to match the following syntax: ```python PATTERN = re.compile(r"""((?:[^."']|"[^"]*"|'[^']*')+)""") return ".".join(PATTERN.split(identifier)[1::2]) ``` The complete selection should look like this: ```python def quoted_by_component(self, identifier, componentName): if componentName == ComponentName.Schema: PATTERN = re.compile(r"""((?:[^."']|"[^"]*"|'[^']*')+)""") return ".".join(PATTERN.split(identifier)[1::2]) else: return self.quoted(identifier) ``` You need to update this pattern because the plugin doesn’t support schema names in Dremio containing dots and spaces. #### Build your pipeline 1. Create a `profiles.yml` file in the `$HOME/.dbt/profiles.yml` path and add the following configs: ```yaml dremioSamples: outputs: cloud_dev: dremio_space: dev dremio_space_folder: no_schema object_storage_path: dev object_storage_source: $scratch pat: <this_is_the_personal_access_token> cloud_host: api.dremio.cloud cloud_project_id: <id_of_project_you_belong_to> threads: 1 type: dremio use_ssl: true user: <your_username> target: dev ``` 2. Execute the transformation pipeline: ```shell $ dbt run -t cloud_dev ``` If the above configurations have been implemented, the output will look something like this: ```shell 17:24:16 Running with dbt=1.5.0 17:24:17 Found 5 models, 0 tests, 0 snapshots, 0 analyses, 348 macros, 0 operations, 0 seed files, 2 sources, 0 exposures, 0 metrics, 0 groups 17:24:17 17:24:29 Concurrency: 1 threads (target='cloud_dev') 17:24:29 17:24:29 1 of 5 START sql view model Preparation.trips .................................. [RUN] 17:24:31 1 of 5 OK created sql view model Preparation. trips ............................. [OK in 2.61s] 17:24:31 2 of 5 START sql view model Preparation.weather ................................ [RUN] 17:24:34 2 of 5 OK created sql view model Preparation.weather ........................... [OK in 2.15s] 17:24:34 3 of 5 START sql view model Business.Transportation.nyc_trips .................. [RUN] 17:24:36 3 of 5 OK created sql view model Business.Transportation.nyc_trips ............. [OK in 2.18s] 17:24:36 4 of 5 START sql view model Business.Weather.nyc_weather ....................... [RUN] 17:24:38 4 of 5 OK created sql view model Business.Weather.nyc_weather .................. [OK in 2.09s] 17:24:38 5 of 5 START sql view model Application.nyc_trips_with_weather ................. [RUN] 17:24:41 5 of 5 OK created sql view model Application.nyc_trips_with_weather ............ [OK in 2.74s] 17:24:41 17:24:41 Finished running 5 view models in 0 hours 0 minutes and 24.03 seconds (24.03s). 17:24:41 17:24:41 Completed successfully 17:24:41 17:24:41 Done. PASS=5 WARN=0 ERROR=0 SKIP=0 TOTAL=5 ``` Now that you have a running environment and a completed job, you can view the data in Dremio and expand your code. This is a snapshot of the project structure in an Studio IDE: [![Cloned repo in an IDE](/img/guides/dremio/dremio-cloned-repo.png?v=2 "Cloned repo in an IDE")](#)Cloned repo in an IDE #### About the schema.yml The `schema.yml` file defines Dremio sources and models to be used and what data models are in scope. In this guides sample project, there are two data sources: 1. The `NYC-weather.csv` stored in the **Samples** database and 2. The `sample_data` from the **Samples database**. The models correspond to both weather and trip data respectively and will be joined for analysis. The sources can be found by navigating to the **Object Storage** section of the Dremio Cloud UI. [![NYC-weather.csv location in Dremio Cloud](/img/guides/dremio/dremio-nyc-weather.png?v=2 "NYC-weather.csv location in Dremio Cloud")](#)NYC-weather.csv location in Dremio Cloud #### About the models **Preparation** — `preparation_trips.sql` and `preparation_weather.sql` are building views on top of the trips and weather data. **Business** — `business_transportation_nyc_trips.sql` applies some level of transformation on `preparation_trips.sql` view. `Business_weather_nyc.sql` has no transformation on the `preparation_weather.sql` view. **Application** — `application_nyc_trips_with_weather.sql` joins the output from the Business model. This is what your business users will consume. #### The Job output When you run the dbt job, it will create a **dev** space folder that has all the data assets created. This is what you will see in Dremio Cloud UI. Spaces in Dremio is a way to organize data assets which map to business units or data products. [![Dremio Cloud dev space](/img/guides/dremio/dremio-dev-space.png?v=2 "Dremio Cloud dev space")](#)Dremio Cloud dev space Open the **Application folder** and you will see the output of the simple transformation we did using dbt. [![Application folder transformation output](/img/guides/dremio/dremio-dev-application.png?v=2 "Application folder transformation output")](#)Application folder transformation output #### Query the data Now that you have run the job and completed the transformation, it's time to query your data. Click on the `nyc_trips_with_weather` view. That will take you to the SQL Runner page. Click **Show SQL Pane** on the upper right corner of the page. Run the following query: ```sql SELECT vendor_id, AVG(tip_amount) FROM dev.application."nyc_treips_with_weather" GROUP BY vendor_id ``` [![Sample output from SQL query](/img/guides/dremio/dremio-test-results.png?v=2 "Sample output from SQL query")](#)Sample output from SQL query This completes the integration setup and data is ready for business consumption. --- ### Build, test, document, and promote adapters [Back to guides](https://docs.getdbt.com/guides.md) Adapter creation Advanced [Menu ]() important dbt v1 This guide is for creating adapters for the Python-based dbt v1. For adapter creation on the Rust-based dbt v2, check out our [new guide](https://docs.getdbt.com/guides/adapter-creation-v2.md?step=1) #### Introduction Adapters are an essential component of dbt. At their most basic level, they are how dbt connects with the various supported data platforms. At a higher-level, dbt v1 adapters strive to give analytics engineers more transferrable skills as well as standardize how analytics projects are structured. Gone are the days where you have to learn a new language or flavor of SQL when you move to a new job that has a different data platform. That is the power of adapters in dbt v1. Navigating and developing around the nuances of different databases can be daunting, but you are not alone. Visit [#adapter-ecosystem](https://getdbt.slack.com/archives/C030A0UF5LM) Slack channel for additional help beyond the documentation. ##### All databases are not the same There's a tremendous amount of work that goes into creating a database. Here is a high-level list of typical database layers (from the outermost layer moving inwards): * SQL API * Client Library / Driver * Server Connection Manager * Query parser * Query optimizer * Runtime * Storage Access Layer * Storage There's a lot more there than just SQL as a language. Databases (and data warehouses) are so popular because you can abstract away a great deal of the complexity from your brain to the database itself. This enables you to focus more on the data. dbt allows for further abstraction and standardization of the outermost layers of a database (SQL API, client library, connection manager) into a framework that both: * Opens database technology to less technical users (a large swath of a DBA's role has been automated, similar to how the vast majority of folks with websites today no longer have to be "[webmasters](https://en.wikipedia.org/wiki/Webmaster)"). * Enables more meaningful conversations about how data warehousing should be done. This is where dbt adapters become critical. ##### What needs to be adapted? dbt adapters are responsible for *adapting* dbt's standard functionality to a particular database. Our prototypical database and adapter are PostgreSQL and dbt-postgres, and most of our adapters are somewhat based on the functionality described in dbt-postgres. Connecting dbt to a new database will require a new adapter to be built or an existing adapter to be extended. The outermost layers of a database map roughly to the areas in which the dbt adapter framework encapsulates inter-database differences. ##### SQL API Even amongst ANSI-compliant databases, there are differences in the SQL grammar. Here are some categories and examples of SQL statements that can be constructed differently: | Category | Area of differences | Examples | | -------------------------------------------- | ------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | Statement syntax | The use of `IF EXISTS` | \* `IF <TABLE> EXISTS, DROP TABLE`\* `DROP <TABLE> IF EXISTS` | | Workflow definition & semantics | Incremental updates | \* `MERGE`\* `DELETE; INSERT` | | Relation and column attributes/configuration | Database-specific materialization configs | \* `DIST = ROUND_ROBIN` (Synapse)\* `DIST = EVEN` (Redshift) | | Permissioning | Grant statements that can only take one grantee at a time vs those that accept lists of grantees | \* `grant SELECT on table dinner.corn to corn_kid, everyone`\* `grant SELECT on table dinner.corn to corn_kid; grant SELECT on table dinner.corn to everyone` | ##### Python Client Library & Connection Manager The other big category of inter-database differences comes with how the client connects to the database and executes queries against the connection. To integrate with dbt, a data platform must have a pre-existing python client library or support ODBC, using a generic python library like pyodbc. | Category | Area of differences | Examples | | ---------------------------- | ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | | Credentials & authentication | Authentication | \* Username & password\* MFA with `boto3` or Okta token | | Connection opening/closing | Create a new connection to db | \* `psycopg2.connect(connection_string)`\* `google.cloud.bigquery.Client(...)` | | Inserting local data | Load seed .`csv` files into Python memory | \* `google.cloud.bigquery.Client.load_table_from_file(...)` (BigQuery)\* `INSERT ... INTO VALUES ...` prepared statement (most other databases) | ##### How dbt encapsulates and abstracts these differences Differences between databases are encoded into discrete areas: | Components | Code Path | Function | | ---------------- | ------------------------------------------------- | --------------------------------------------------------------------------------------- | | Python classes | `adapters/<adapter_name>` | Configuration (Refer to \[Python classes]\(#python classes) | | Macros | `include/<adapter_name>/macros/adapters/` | SQL API & statement syntax (for example, how to create schema or how to get table info) | | Materializations | `include/<adapter_name>/macros/materializations/` | Table/view/snapshot/ workflow definitions | ###### Python classes These classes implement all the methods responsible for: * Connecting to a database and issuing queries. * Providing dbt with database-specific configuration information. | Class | Description | | ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | AdapterClass | High-level configuration type conversion and any database-specific python methods needed | | AdapterCredentials | Typed dictionary of possible profiles and associated methods | | AdapterConnectionManager | All the methods responsible for connecting to a database and issuing queries | | AdapterRelation | How relation names should be rendered, printed, and quoted. Do relation names use all three parts? `catalog.model_name` (two-part name) or `database.schema.model_name` (three-part name) | | AdapterColumn | How names should be rendered, and database-specific properties | ###### Macros A set of *macros* responsible for generating SQL that is compliant with the target database. ###### Materializations A set of *materializations* and their corresponding helper macros defined in dbt using Jinja and SQL. They codify for dbt how model files should be persisted into the database. ##### Adapter Architecture Below is a flow diagram illustrating how a `dbt run` command works with the `dbt-postgres` adapter. It shows the relationship between `dbt-core`, `dbt-adapters`, and individual adapters. [![Diagram of adapter architecture](/img/adapter-guide/adapter-architecture-diagram.png?v=2 "Diagram of adapter architecture")](#)Diagram of adapter architecture #### Prerequisites It is very important that you have the right skills, and understand the level of difficulty required to make an adapter for your data platform. The more you can answer Yes to the below questions, the easier your adapter development (and user-) experience will be. See the [New Adapter Information Sheet wiki](https://github.com/dbt-labs/dbt/wiki/New-Adapter-Information-Sheet) for even more specific questions. ##### Training * The developer (and any product managers) ideally will have substantial experience as an end-user of dbt. If not, it is highly advised that you at least take the [dbt Fundamentals](https://learn.getdbt.com/courses/dbt-fundamentals) and [Advanced Materializations](https://learn.getdbt.com/courses/advanced-materializations) course. ##### Database * Does the database complete transactions fast enough for interactive development? * Can you execute SQL against the data platform? * Is there a concept of schemas? * Does the data platform support ANSI SQL, or at least a subset? ##### Driver / Connection Library * Is there a Python-based driver for interacting with the database that is db API 2.0 compliant (e.g. Psycopg2 for Postgres, pyodbc for SQL Server) * Does it support: prepared statements, multiple statements, or single sign on token authorization to the data platform? ##### Open source software * Does your organization have an established process for publishing open source software? It is easiest to build an adapter for dbt when the following the data warehouse/platform in question has: * a conventional ANSI-SQL interface (or as close to it as possible), * a mature connection library/SDK that uses ODBC or Python DB 2 API, and * a way to enable developers to iterate rapidly with both quick reads and writes ##### Maintaining your new adapter When your adapter becomes more popular, and people start using it, you may quickly become the maintainer of an increasingly popular open source project. With this new role, comes some unexpected responsibilities that not only include code maintenance, but also working with a community of users and contributors. To help people understand what to expect of your project, you should communicate your intentions early and often in your adapter documentation or README. Answer questions like, Is this experimental work that people should use at their own risk? Or is this production-grade code that you're committed to maintaining into the future? ###### Keeping the code compatible with dbt v1 An adapter is compatible with dbt v1 if it has correctly implemented the interface defined in [dbt-adapters](https://github.com/dbt-labs/dbt-adapters/) and is tested by [dbt-tests-adapters](https://github.com/dbt-labs/dbt-adapters/tree/main/dbt-tests-adapter). Prior to dbt version 1.8, this interface was contained in `dbt-core`. New minor version releases of `dbt-adapters` may include changes to the Python interface for adapter plugins, as well as new or updated test cases. The maintainers of `dbt-adapters` will clearly communicate these changes in documentation and release notes, and they will aim for backwards compatibility whenever possible. Patch releases of `dbt-adapters` will *not* include breaking changes or new features to adapter-facing code. ###### Versioning and releasing your adapter dbt Labs strongly recommends you to adopt the following approach when versioning and releasing your plugin. * Declare major version compatibility with `dbt-adapters` and only set a boundary on the minor version if there is some known reason. * Do not import or rely on code from `dbt-core`. * Aim to release a new minor version of your plugin as you add substantial new features. Typically, this will be triggered by adding support for new features released in `dbt-adapters` or by changes to the data platform itself. * While your plugin is new and you're iterating on features, aim to offer backwards compatibility and deprecation notices for at least one minor version. As your plugin matures, aim to leave backwards compatibility and deprecation notices in place until the next major version (v2). * Release patch versions of your plugins whenever needed. These patch releases should only contain fixes. note Prior to dbt version 1.8, we recommended that the minor version of your plugin should match the minor version in `dbt-core` (for example, 1.1.x). #### Build a new adapter This step will walk you through the first creating the necessary adapter classes and macros, and provide some resources to help you validate that your new adapter is working correctly. Make sure you've familiarized yourself with the previous steps in this guide. Once the adapter is passing most of the functional tests in the previous "Testing a new adapter" step, please let the community know that is available to use by adding the adapter to the ["Supported Data Platforms"](https://docs.getdbt.com/docs/supported-data-platforms.md) page by following the steps given in "Documenting your adapter. For any questions you may have, don't hesitate to ask in the [#adapter-ecosystem](https://getdbt.slack.com/archives/C030A0UF5LM) Slack channel. The community is very helpful and likely has experienced a similar issue as you. ##### Scaffolding a new adapter To create a new adapter plugin from scratch, you can use the [dbt-database-adapter-scaffold](https://github.com/dbt-labs/dbt-database-adapter-scaffold) to trigger an interactive session which will generate a scaffolding for you to build upon. Example usage: ```text $ cookiecutter gh:dbt-labs/dbt-database-adapter-scaffold ``` The generated boilerplate starting project will include a basic adapter plugin file structure, examples of macros, high level method descriptions, etc. One of the most important choices you will make during the cookiecutter generation will revolve around the field for `is_sql_adapter` which is a boolean used to correctly apply imports for either a `SQLAdapter` or `BaseAdapter`. Knowing which you will need requires a deeper knowledge of your selected database but a few good guides for the choice are. * Does your database have a complete SQL API? Can it perform tasks using SQL such as creating schemas, dropping schemas, querying an `information_schema` for metadata calls? If so, it is more likely to be a SQLAdapter where you set `is_sql_adapter` to `True`. * Most adapters do fall under SQL adapters which is why we chose it as the default `True` value. * It is very possible to build out a fully functional `BaseAdapter`. This will require a little more ground work as it doesn't come with some prebuilt methods the `SQLAdapter` class provides. See `dbt-bigquery` as a good guide. ##### Implementation details Regardless if you decide to use the cookiecutter template or manually create the plugin, this section will go over each method that is required to be implemented. The following table provides a high-level overview of the classes, methods, and macros you may have to define for your data platform. | File | Component | Purpose | | ------------------------------------------------- | ------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `./setup.py` | `setup()` function | adapter meta-data (package name, version, author, homepage, etc) | | `myadapter/dbt/adapters/myadapter/__init__.py` | `AdapterPlugin` | bundle all the information below into a dbt plugin | | `myadapter/dbt/adapters/myadapter/connections.py` | `MyAdapterCredentials` class | parameters to connect to and configure the database, via a the chosen Python driver | | `myadapter/dbt/adapters/myadapter/connections.py` | `MyAdapterConnectionManager` class | telling dbt how to interact with the database w.r.t opening/closing connections, executing queries, and fetching data. Effectively a wrapper around the db API or driver. | | `myadapter/dbt/include/bigquery/` | a dbt project of macro "overrides" in the format of "myadapter\_\_" | any differences in SQL syntax for regular db operations will be modified here from the global\_project (e.g. "Create Table As Select", "Get all relations in the current schema", etc) | | `myadapter/dbt/adapters/myadapter/impl.py` | `MyAdapterConfig` | database- and relation-level configs and | | `myadapter/dbt/adapters/myadapter/impl.py` | `MyAdapterAdapter` | for changing *how* dbt performs operations like macros and other needed Python functionality | | `myadapter/dbt/adapters/myadapter/column.py` | `MyAdapterColumn` | for defining database-specific column such as datatype mappings | ##### Editing `setup.py` Edit the file at `myadapter/setup.py` and fill in the missing information. You can skip this step if you passed the arguments for `email`, `url`, `author`, and `dependencies` to the cookiecutter template script. If you plan on having nested macro folder structures, you may need to add entries to `package_data` so your macro source files get installed. ##### Editing the connection manager Edit the connection manager at `myadapter/dbt/adapters/myadapter/connections.py`. This file is defined in the sections below. ###### The Credentials class The credentials class defines all of the database-specific credentials (e.g. `username` and `password`) that users will need in the [connection profile](https://docs.getdbt.com/docs/supported-data-platforms.md) for your new adapter. Each credentials contract should subclass dbt.adapters.base.Credentials, and be implemented as a python dataclass. Note that the base class includes required database and schema fields, as dbt uses those values internally. For example, if your adapter requires a host, integer port, username string, and password string, but host is the only required field, you'd add definitions for those new properties to the class as types, like this: connections.py ```python from dataclasses import dataclass from typing import Optional from dbt.adapters.base import Credentials @dataclass class MyAdapterCredentials(Credentials): host: str port: int = 1337 username: Optional[str] = None password: Optional[str] = None @property def type(self): return 'myadapter' @property def unique_field(self): """ Hashed and included in anonymous telemetry to track adapter adoption. Pick a field that can uniquely identify one team/organization building with this adapter """ return self.host def _connection_keys(self): """ List of keys to display in the `dbt debug` output. """ return ('host', 'port', 'database', 'username') ``` There are a few things you can do to make it easier for users when connecting to your database: * Be sure to implement the Credentials' `_connection_keys` method shown above. This method will return the keys that should be displayed in the output of the `dbt debug` command. As a general rule, it's good to return all the arguments used in connecting to the actual database except the password (even optional arguments). * Create a `profile_template.yml` to enable configuration prompts for a brand-new user setting up a connection profile via the [`dbt init` command](https://docs.getdbt.com/reference/commands/init.md). You will find more details in the following steps. * You may also want to define an `ALIASES` mapping on your Credentials class to include any config names you want users to be able to use in place of 'database' or 'schema'. For example if everyone using the MyAdapter database calls their databases "collections", you might do: connections.py ```python @dataclass class MyAdapterCredentials(Credentials): host: str port: int = 1337 username: Optional[str] = None password: Optional[str] = None ALIASES = { 'collection': 'database', } ``` Then users can use `collection` OR `database` in their `profiles.yml`, `dbt_project.yml`, or `config()` calls to set the database. ###### `ConnectionManager` class methods Once credentials are configured, you'll need to implement some connection-oriented methods. They are enumerated in the SQLConnectionManager docstring, but an overview will also be provided here. **Methods to implement:** * `open` * `get_response` * `cancel` * `exception_handler` * `standardize_grants_dict` ###### `open(cls, connection)` `open()` is a classmethod that gets a connection object (which could be in any state, but will have a `Credentials` object with the attributes you defined above) and moves it to the 'open' state. Generally this means doing the following: * if the connection is open already, log and return it. * If a database needed changes to the underlying connection before re-use, that would happen here * create a connection handle using the underlying database library using the credentials * on success: * set connection.state to `'open'` * set connection.handle to the handle object * this is what must have a `cursor()` method that returns a cursor! * on error: * set connection.state to `'fail'` * set connection.handle to `None` * raise a `dbt.exceptions.FailedToConnectException` with the error and any other relevant information For example: connections.py ```python @classmethod def open(cls, connection): if connection.state == 'open': logger.debug('Connection is already open, skipping open.') return connection credentials = connection.credentials try: handle = myadapter_library.connect( host=credentials.host, port=credentials.port, username=credentials.username, password=credentials.password, catalog=credentials.database ) connection.state = 'open' connection.handle = handle return connection ``` ###### `get_response(cls, cursor)` `get_response` is a classmethod that gets a cursor object and returns adapter-specific information about the last executed command. The return value should be an `AdapterResponse` object that includes items such as `code`, `rows_affected`, `bytes_processed`, and a summary `_message` for logging to stdout. connections.py ```python @classmethod def get_response(cls, cursor) -> AdapterResponse: code = cursor.sqlstate or "OK" rows = cursor.rowcount status_message = f"{code} {rows}" return AdapterResponse( _message=status_message, code=code, rows_affected=rows ) ``` ###### `cancel(self, connection)` `cancel` is an instance method that gets a connection object and attempts to cancel any ongoing queries, which is database dependent. Some databases don't support the concept of cancellation, they can simply implement it via 'pass' and their adapter classes should implement an `is_cancelable` that returns False - On ctrl+c connections may remain running. This method must be implemented carefully, as the affected connection will likely be in use in a different thread. connections.py ```python def cancel(self, connection): tid = connection.handle.transaction_id() sql = 'select cancel_transaction({})'.format(tid) logger.debug("Cancelling query '{}' ({})".format(connection_name, pid)) _, cursor = self.add_query(sql, 'master') res = cursor.fetchone() logger.debug("Canceled query '{}': {}".format(connection_name, res)) ``` ###### `exception_handler(self, sql, connection_name='master')` `exception_handler` is an instance method that returns a context manager that will handle exceptions raised by running queries, catch them, log appropriately, and then raise exceptions dbt knows how to handle. If you use the (highly recommended) `@contextmanager` decorator, you only have to wrap a `yield` inside a `try` block, like so: connections.py ```python @contextmanager def exception_handler(self, sql: str): try: yield except myadapter_library.DatabaseError as exc: self.release(connection_name) logger.debug('myadapter error: {}'.format(str(e))) raise dbt.exceptions.DatabaseException(str(exc)) except Exception as exc: logger.debug("Error running SQL: {}".format(sql)) logger.debug("Rolling back transaction.") self.release(connection_name) raise dbt.exceptions.RuntimeException(str(exc)) ``` ###### `standardize_grants_dict(self, grants_table: agate.Table) -> dict` `standardize_grants_dict` is an method that returns the dbt-standardized grants dictionary that matches how users configure grants now in dbt. The input is the result of `SHOW GRANTS ON {{model}}` call loaded into an agate table. If there's any massaging of agate table containing the results, of `SHOW GRANTS ON {{model}}`, that can't easily be accomplished in SQL, it can be done here. For example, the SQL to show grants *should* filter OUT any grants TO the current user/role (e.g. OWNERSHIP). If that's not possible in SQL, it can be done in this method instead. impl.py ```python @available def standardize_grants_dict(self, grants_table: agate.Table) -> dict: """ :param grants_table: An agate table containing the query result of the SQL returned by get_show_grant_sql :return: A standardized dictionary matching the `grants` config :rtype: dict """ grants_dict: Dict[str, List[str]] = {} for row in grants_table: grantee = row["grantee"] privilege = row["privilege_type"] if privilege in grants_dict.keys(): grants_dict[privilege].append(grantee) else: grants_dict.update({privilege: [grantee]}) return grants_dict ``` ##### Editing the adapter implementation Edit the connection manager at `myadapter/dbt/adapters/myadapter/impl.py` Very little is required to implement the adapter itself. On some adapters, you will not need to override anything. On others, you'll likely need to override some of the `convert_*` classmethods, or override the `is_cancelable` classmethod on others to return `False`. ###### `datenow()` This classmethod provides the adapter's canonical date function. This is not used but is required– anyway on all adapters. impl.py ```python @classmethod def date_function(cls): return 'datenow()' ``` ##### Editing SQL logic dbt implements specific SQL operations using Jinja macros. While reasonable defaults are provided for many such operations (like `create_schema`, `drop_schema`, `create_table`, etc), you may need to override one or more of macros when building a new adapter. ###### Required macros The following macros must be implemented, but you can override their behavior for your adapter using the "dispatch" pattern described below. Macros marked (required) do not have a valid default implementation, and are required for dbt to operate. * `alter_column_type` ([source](https://github.com/dbt-labs/dbt/blob/f988f76fccc1878aaf8d8631c05be3e9104b3b9a/core/dbt/include/global_project/macros/adapters/columns.sql#L37-L55)) * `check_schema_exists` ([source](https://github.com/dbt-labs/dbt/blob/f988f76fccc1878aaf8d8631c05be3e9104b3b9a/core/dbt/include/global_project/macros/adapters/metadata.sql#L43-L55)) * `create_schema` ([source](https://github.com/dbt-labs/dbt/blob/f988f76fccc1878aaf8d8631c05be3e9104b3b9a/core/dbt/include/global_project/macros/adapters/schema.sql#L1-L9)) * `drop_relation` ([source](https://github.com/dbt-labs/dbt/blob/f988f76fccc1878aaf8d8631c05be3e9104b3b9a/core/dbt/include/global_project/macros/adapters/relation.sql#L34-L42)) * `drop_schema` ([source](https://github.com/dbt-labs/dbt/blob/f988f76fccc1878aaf8d8631c05be3e9104b3b9a/core/dbt/include/global_project/macros/adapters/schema.sql#L12-L20)) * `get_columns_in_relation` ([source](https://github.com/dbt-labs/dbt/blob/f988f76fccc1878aaf8d8631c05be3e9104b3b9a/core/dbt/include/global_project/macros/adapters/columns.sql#L1-L8)) (required) * `list_relations_without_caching` ([source](https://github.com/dbt-labs/dbt/blob/f988f76fccc1878aaf8d8631c05be3e9104b3b9a/core/dbt/include/global_project/macros/adapters/metadata.sql#L58-L65)) (required) * `list_schemas` ([source](https://github.com/dbt-labs/dbt/blob/f988f76fccc1878aaf8d8631c05be3e9104b3b9a/core/dbt/include/global_project/macros/adapters/metadata.sql#L29-L40)) * `rename_relation` ([source](https://github.com/dbt-labs/dbt/blob/f988f76fccc1878aaf8d8631c05be3e9104b3b9a/core/dbt/include/global_project/macros/adapters/relation.sql#L56-L65)) * `truncate_relation` ([source](https://github.com/dbt-labs/dbt/blob/f988f76fccc1878aaf8d8631c05be3e9104b3b9a/core/dbt/include/global_project/macros/adapters/relation.sql#L45-L53)) * `current_timestamp` ([source](https://github.com/dbt-labs/dbt/blob/f988f76fccc1878aaf8d8631c05be3e9104b3b9a/core/dbt/include/global_project/macros/adapters/freshness.sql#L1-L8)) (required) * `copy_grants` ###### Adapter dispatch Most modern databases support a majority of the standard SQL spec. There are some databases that *do not* support critical aspects of the SQL spec however, or they provide their own nonstandard mechanisms for implementing the same functionality. To account for these variations in SQL support, dbt provides a mechanism called [multiple dispatch](https://en.wikipedia.org/wiki/Multiple_dispatch) for macros. With this feature, macros can be overridden for specific adapters. This makes it possible to implement high-level methods (like "create table") in a database-specific way. adapters.sql ```jinja2 {# dbt will call this macro by name, providing any arguments #} {% macro create_table_as(temporary, relation, sql) -%} {# dbt will dispatch the macro call to the relevant macro #} {{ return( adapter.dispatch('create_table_as')(temporary, relation, sql) ) }} {%- endmacro %} {# If no macro matches the specified adapter, "default" will be used #} {% macro default__create_table_as(temporary, relation, sql) -%} ... {%- endmacro %} {# Example which defines special logic for Redshift #} {% macro redshift__create_table_as(temporary, relation, sql) -%} ... {%- endmacro %} {# Example which defines special logic for BigQuery #} {% macro bigquery__create_table_as(temporary, relation, sql) -%} ... {%- endmacro %} ``` The `adapter.dispatch()` macro takes a second argument, `packages`, which represents a set of "search namespaces" in which to find potential implementations of a dispatched macro. This allows users of community-supported adapters to extend or "shim" dispatched macros from common packages, such as `dbt-utils`, with adapter-specific versions in their own project or other installed packages. See: * "Shim" package examples: [`spark-utils`](https://github.com/dbt-labs/spark-utils), [`tsql-utils`](https://github.com/dbt-msft/tsql-utils) * [`adapter.dispatch` docs](https://docs.getdbt.com/reference/dbt-jinja-functions/dispatch.md) ###### Overriding adapter methods While much of dbt's adapter-specific functionality can be modified in adapter macros, it can also make sense to override adapter methods directly. In this example, assume that a database does not support a `cascade` parameter to `drop schema`. Instead, we can implement an approximation where we drop each relation and then drop the schema. impl.py ```python def drop_schema(self, relation: BaseRelation): relations = self.list_relations( database=relation.database, schema=relation.schema ) for relation in relations: self.drop_relation(relation) super().drop_schema(relation) ``` ###### Grants Macros See [this GitHub discussion](https://github.com/dbt-labs/dbt/discussions/5468) for information on the macros required for `GRANT` statements: ##### Behavior change flags Starting in `dbt-adapters==1.5.0` and `dbt-core==1.8.7`, adapter maintainers can implement their own behavior change flags. Refer to [Behavior changes](https://docs.getdbt.com/reference/global-configs/behavior-changes.md) for more information. Behavior Flags are not intended to be long-living feature flags. They should be implemented with the expectation that the behavior will be the default within an expected period of time. To implement a behavior change flag, you must provide a name for the flag, a default setting (`True` / `False`), an optional source, and a description and/or a link to the flag's documentation on docs.getdbt.com. We recommend having a description and documentation link whenever possible. The description and/or docs should provide end users context for why the flag exists, why they may see a warning, and why they may want to utilize the behavior flag. Behavior change flags can be implemented by overwriting `_behavior_flags()` on the adapter in `impl.py`: impl.py ```python class ABCAdapter(BaseAdapter): ... @property def _behavior_flags(self) -> List[BehaviorFlag]: return [ { "name": "enable_new_functionality_requiring_higher_permissions", "default": False, "source": "dbt-abc", "description": ( "The dbt-abc adapter is implementing a new method for sourcing metadata. " "This is a more performant way for dbt to source metadata but requires higher permissions on the platform. " "Enabling this without granting the requisite permissions will result in an error. " "This feature is expected to be required by Spring 2025." ), "docs_url": "https://docs.getdbt.com/reference/global-configs/behavior-changes#abc-enable_new_functionality_requiring_higher_permissions", } ] ``` Once a behavior change flag has been implemented, it can be referenced on the adapter both in `impl.py` and in Jinja macros: impl.py ```python class ABCAdapter(BaseAdapter): ... def some_method(self, *args, **kwargs): if self.behavior.enable_new_functionality_requiring_higher_permissions: # do the new thing else: # do the old thing ``` adapters.sql ```sql {% macro some_macro(**kwargs) %} {% if adapter.behavior.enable_new_functionality_requiring_higher_permissions %} {# do the new thing #} {% else %} {# do the old thing #} {% endif %} {% endmacro %} ``` Every time the behavior flag evaluates to `False,` it warns the user, informing them that a change will occur in the future. This warning doesn't display when the flag evaluates to `True` as the user is already in the new experience. Recognizing that the warnings can be disruptive and are not always necessary, you can evaluate the flag without triggering the warning. Simply append `.no_warn` to the end of the flag. impl.py ```python class ABCAdapter(BaseAdapter): ... def some_method(self, *args, **kwargs): if self.behavior.enable_new_functionality_requiring_higher_permissions.no_warn: # do the new thing else: # do the old thing ``` adapters.sql ```sql {% macro some_macro(**kwargs) %} {% if adapter.behavior.enable_new_functionality_requiring_higher_permissions.no_warn %} {# do the new thing #} {% else %} {# do the old thing #} {% endif %} {% endmacro %} ``` It's best practice to evaluate a behavior flag as few times as possible. This will make it easier to remove once the behavior change has matured. As a result, evaluating the flag earlier in the logic flow is easier. Then, take either the old or the new path. While this may create some duplication in code, using behavior flags in this way provides a safer way to implement a change, which we are already admitting is risky or even breaking in nature. ##### Other files ###### `profile_template.yml` In order to enable the [`dbt init` command](https://docs.getdbt.com/reference/commands/init.md) to prompt users when setting up a new project and connection profile, you should include a **profile template**. The filepath needs to be `dbt/include/<adapter_name>/profile_template.yml`. It's possible to provide hints, default values, and conditional prompts based on connection methods that require different supporting attributes. Users will also be able to include custom versions of this file in their own projects, with fixed values specific to their organization, to support their colleagues when using your dbt adapter for the first time. See examples: * [dbt-postgres](https://github.com/dbt-labs/dbt-postgres/blob/main/dbt/include/postgres/profile_template.yml) * [dbt-redshift](https://github.com/dbt-labs/dbt-redshift/blob/main/dbt/include/redshift/profile_template.yml) * [dbt-snowflake](https://github.com/dbt-labs/dbt-snowflake/blob/main/dbt/include/snowflake/profile_template.yml) * [dbt-bigquery](https://github.com/dbt-labs/dbt-bigquery/blob/main/dbt/include/bigquery/profile_template.yml) ###### `__version__.py` To assure that `dbt --version` provides the latest v1 version the adapter supports, be sure include a `__version__.py` file. The filepath will be `dbt/adapters/<adapter_name>/__version__.py`. We recommend using the latest v1 version and as the adapter is made compatible with later versions, this file will need to be updated. For a sample file, check out this [example](https://github.com/dbt-labs/dbt-snowflake/blob/main/dbt/adapters/snowflake/__version__.py). It should be noted that both of these files are included in the bootstrapped output of the `dbt-database-adapter-scaffold` so when using the scaffolding, these files will be included. #### Test your adapter This document has two sections: 1. Refer to "About the testing framework" for a description of the standard framework that we maintain for using pytest together with dbt. It includes an example that shows the anatomy of a simple test case. 2. Refer to "Testing your adapter" for a step-by-step guide for using our out-of-the-box suite of "basic" tests, which will validate that your adapter meets a baseline of dbt functionality. ##### Testing prerequisites * Your adapter must be compatible with dbt **v1.1** or newer * You should be familiar with **pytest**: <https://docs.pytest.org> ##### About the testing framework [dbt-adapters-tests](https://github.com/dbt-labs/dbt-adapters/tree/main/dbt-tests-adapter) offers a standard framework for running prebuilt functional tests, and for defining your own tests. The core testing framework is built using `pytest`, a mature and standard library for testing Python projects. It includes basic utilities for setting up pytest + dbt. These are used by all "prebuilt" functional tests, and make it possible to quickly write your own tests. Those utilities allow you to do three basic things: 1. **Quickly set up a dbt "project."** Define project resources via methods such as `models()` and `seeds()`. Use `project_config_update()` to pass configurations into `dbt_project.yml`. 2. **Define a sequence of dbt commands.** The most important utility is `run_dbt()`, which returns the [results](https://docs.getdbt.com/reference/dbt-classes.md#result-objects) of each dbt command. It takes a list of CLI specifiers (subcommand + flags), as well as an optional second argument, `expect_pass=False`, for cases where you expect the command to fail. 3. **Validate the results of those dbt commands.** For example, `check_relations_equal()` asserts that two database objects have the same structure and content. You can also write your own `assert` statements, by inspecting the results of a dbt command, or querying arbitrary database objects with `project.run_sql()`. You can see the full suite of utilities, with arguments and annotations, in [`util.py`](https://github.com/dbt-labs/dbt/blob/1.latest/core/dbt/tests/util.py). You'll also see them crop up across a number of test cases. While all utilities are intended to be reusable, you won't need all of them for every test. In the example below, we'll show a simple test case that uses only a few utilities. ###### Example: a simple test case This example will show you the anatomy of a test case using dbt + pytest. We will create reusable components, combine them to form a dbt "project", and define a sequence of dbt commands. Then, we'll use Python `assert` statements to ensure those commands succeed (or fail) as we expect. In ["Getting started running basic tests,"](#getting-started-running-basic-tests) we'll offer step-by-step instructions for installing and configuring `pytest`, so that you can run it on your own machine. For now, it's more important to see how the pieces of a test case fit together. This example includes a seed, a model, and two tests—one of which will fail. 1. Define Python strings that will represent the file contents in your dbt project. Defining these in a separate file enables you to reuse the same components across different test cases. The pytest name for this type of reusable component is "fixture." tests/functional/example/fixtures.py ```python # seeds/my_seed.csv my_seed_csv = """ id,name,some_date 1,Easton,1981-05-20T06:46:51 2,Lillian,1978-09-03T18:10:33 3,Jeremiah,1982-03-11T03:59:51 4,Nolan,1976-05-06T20:21:35 """.lstrip() # models/my_model.sql my_model_sql = """ select * from {{ ref('my_seed') }} union all select null as id, null as name, null as some_date """ # models/my_model.yml my_model_yml = """ version: 2 models: - name: my_model columns: - name: id data_tests: - unique - not_null # this test will fail """ ``` 2. Use the "fixtures" to define the project for your test case. These fixtures are always scoped to the **class**, where the class represents one test case—that is, one dbt project or scenario. (The same test case can be used for one or more actual tests, which we'll see in step 3.) Following the default pytest configurations, the file name must begin with `test_`, and the class name must begin with `Test`. tests/functional/example/test\_example\_failing\_test.py ```python import pytest from dbt.tests.util import run_dbt # our file contents from tests.functional.example.fixtures import ( my_seed_csv, my_model_sql, my_model_yml, ) # class must begin with 'Test' class TestExample: """ Methods in this class will be of two types: 1. Fixtures defining the dbt "project" for this test case. These are scoped to the class, and reused for all tests in the class. 2. Actual tests, whose names begin with 'test_'. These define sequences of dbt commands and 'assert' statements. """ # configuration in dbt_project.yml @pytest.fixture(scope="class") def project_config_update(self): return { "name": "example", "models": {"+materialized": "view"} } # everything that goes in the "seeds" directory @pytest.fixture(scope="class") def seeds(self): return { "my_seed.csv": my_seed_csv, } # everything that goes in the "models" directory @pytest.fixture(scope="class") def models(self): return { "my_model.sql": my_model_sql, "my_model.yml": my_model_yml, } # continues below ``` 3. Now that we've set up our project, it's time to define a sequence of dbt commands and assertions. We define one or more methods in the same file, on the same class (`TestExampleFailingTest`), whose names begin with `test_`. These methods share the same setup (project scenario) from above, but they can be run independently by pytest—so they shouldn't depend on each other in any way. tests/functional/example/test\_example\_failing\_test.py ```python # continued from above # The actual sequence of dbt commands and assertions # pytest will take care of all "setup" + "teardown" def test_run_seed_test(self, project): """ Seed, then run, then test. We expect one of the tests to fail An alternative pattern is to use pytest "xfail" (see below) """ # seed seeds results = run_dbt(["seed"]) assert len(results) == 1 # run models results = run_dbt(["run"]) assert len(results) == 1 # test tests results = run_dbt(["test"], expect_pass = False) # expect failing test assert len(results) == 2 # validate that the results include one pass and one failure result_statuses = sorted(r.status for r in results) assert result_statuses == ["fail", "pass"] @pytest.mark.xfail def test_build(self, project): """Expect a failing test""" # do it all results = run_dbt(["build"]) ``` 3. Our test is ready to run! The last step is to invoke `pytest` from your command line. We'll walk through the actual setup and configuration of `pytest` in the next section. terminal ```sh $ python3 -m pytest tests/functional/test_example.py =========================== test session starts ============================ platform ... -- Python ..., pytest-..., pluggy-... rootdir: ... plugins: ... tests/functional/test_example.py .X [100%] ======================= 1 passed, 1 xpassed in 1.38s ======================= ``` You can find more ways to run tests, along with a full command reference, in the [pytest usage docs](https://docs.pytest.org/how-to/usage.html). We've found the `-s` flag (or `--capture=no`) helpful to print logs from the underlying dbt invocations, and to step into an interactive debugger if you've added one. You can also use environment variables to set [global dbt configs](https://docs.getdbt.com/reference/global-configs/about-global-configs.md), such as (Applies to dbt v1.11 and later) `DBT_ENGINE_DEBUG` (to show debug-level logs). ##### Testing this adapter Anyone who installs `dbt-core`, and wishes to define their own test cases, can use the framework presented in the first section. The framework is especially useful for testing standard dbt behavior across different databases. To that end, we have built and made available a [package of reusable adapter test cases](https://github.com/dbt-labs/dbt-adapters/tree/main/dbt-tests-adapter), for creators and maintainers of adapter plugins. These test cases cover basic expected functionality, as well as functionality that frequently requires different implementations across databases. For the time being, this package is also located within the `dbt-labs/dbt` repository, but separate from the `dbt-core` Python package. ##### Categories of tests In the course of creating and maintaining your adapter, it's likely that you will end up implementing tests that fall into three broad categories: 1. **Basic tests** that every adapter plugin is expected to pass. These are defined in `tests.adapter.basic`. Given differences across data platforms, these may require slight modification or reimplementation. Significantly overriding or disabling these tests should be with good reason, since each represents basic functionality expected by dbt users. For example, if your adapter does not support incremental models, you should disable the test, [by marking it with `skip` or `xfail`](https://docs.pytest.org/en/latest/how-to/skipping.html), as well as noting that limitation in any documentation, READMEs, and usage guides that accompany your adapter. 2. **Optional tests**, for second-order functionality that is common across plugins, but not required for basic use. Your plugin can opt into these test cases by inheriting existing ones, or reimplementing them with adjustments. For now, this category includes all tests located outside the `basic` subdirectory. More tests will be added as we convert older tests defined on dbt-core and mature plugins to use the standard framework. 3. **Custom tests**, for behavior that is specific to your adapter / data platform. Each data warehouse has its own specialties and idiosyncracies. We encourage you to use the same `pytest`-based framework, utilities, and fixtures to write your own custom tests for functionality that is unique to your adapter. If you run into an issue with the core framework, or the basic/optional test cases—or if you've written a custom test that you believe would be relevant and useful for other adapter plugin developers—please open an issue or PR in the `dbt-labs/dbt` repository on GitHub. ##### Getting started running basic tests In this section, we'll walk through the three steps to start running our basic test cases on your adapter plugin: 1. Install dependencies 2. Set up and configure pytest 3. Define test cases ##### Install dependencies You should already have a virtual environment with `dbt-core` and your adapter plugin installed. You'll also need to install: * [`pytest`](https://pypi.org/project/pytest/) * [`dbt-tests-adapter`](https://pypi.org/project/dbt-tests-adapter/), the set of common test cases * (optional) [`pytest` plugins](https://docs.pytest.org/en/7.0.x/reference/plugin_list.html)--we'll use `pytest-dotenv` below Or specify all dependencies in a requirements file like: dev\_requirements.txt ```txt pytest pytest-dotenv dbt-tests-adapter ``` ```sh python -m pip install -r dev_requirements.txt ``` ##### Set up and configure pytest First, set yourself up to run `pytest` by creating a file named `pytest.ini` at the root of your repository: pytest.ini ```python [pytest] filterwarnings = ignore:.*'soft_unicode' has been renamed to 'soft_str'*:DeprecationWarning ignore:unclosed file .*:ResourceWarning env_files = test.env # uses pytest-dotenv plugin # this allows you to store env vars for database connection in a file named test.env # rather than passing them in every CLI command, or setting in `PYTEST_ADDOPTS` # be sure to add "test.env" to .gitignore as well! testpaths = tests/functional # name per convention ``` Then, create a configuration file within your tests directory. In it, you'll want to define all necessary profile configuration for connecting to your data platform in local development and continuous integration. We recommend setting these values with environment variables, since this file will be checked into version control. tests/conftest.py ```python import pytest import os # Import the standard functional fixtures as a plugin # Note: fixtures with session scope need to be local pytest_plugins = ["dbt.tests.fixtures.project"] # The profile dictionary, used to write out profiles.yml # dbt will supply a unique schema per test, so we do not specify 'schema' here @pytest.fixture(scope="class") def dbt_profile_target(): return { 'type': '<myadapter>', 'threads': 1, 'host': os.getenv('HOST_ENV_VAR_NAME'), 'user': os.getenv('USER_ENV_VAR_NAME'), ... } ``` ##### Define test cases As in the example above, each test case is defined as a class, and has its own "project" setup. To get started, you can import all basic test cases and try running them without changes. tests/functional/adapter/test\_basic.py ```python import pytest from dbt.tests.adapter.basic.test_base import BaseSimpleMaterializations from dbt.tests.adapter.basic.test_singular_tests import BaseSingularTests from dbt.tests.adapter.basic.test_singular_tests_ephemeral import BaseSingularTestsEphemeral from dbt.tests.adapter.basic.test_empty import BaseEmpty from dbt.tests.adapter.basic.test_ephemeral import BaseEphemeral from dbt.tests.adapter.basic.test_incremental import BaseIncremental from dbt.tests.adapter.basic.test_generic_tests import BaseGenericTests from dbt.tests.adapter.basic.test_snapshot_check_cols import BaseSnapshotCheckCols from dbt.tests.adapter.basic.test_snapshot_timestamp import BaseSnapshotTimestamp from dbt.tests.adapter.basic.test_adapter_methods import BaseAdapterMethod class TestSimpleMaterializationsMyAdapter(BaseSimpleMaterializations): pass class TestSingularTestsMyAdapter(BaseSingularTests): pass class TestSingularTestsEphemeralMyAdapter(BaseSingularTestsEphemeral): pass class TestEmptyMyAdapter(BaseEmpty): pass class TestEphemeralMyAdapter(BaseEphemeral): pass class TestIncrementalMyAdapter(BaseIncremental): pass class TestGenericTestsMyAdapter(BaseGenericTests): pass class TestSnapshotCheckColsMyAdapter(BaseSnapshotCheckCols): pass class TestSnapshotTimestampMyAdapter(BaseSnapshotTimestamp): pass class TestBaseAdapterMethod(BaseAdapterMethod): pass ``` Finally, run pytest: ```sh python3 -m pytest tests/functional ``` ##### Modifying test cases You may need to make slight modifications in a specific test case to get it passing on your adapter. The mechanism to do this is simple: rather than simply inheriting the "base" test with `pass`, you can redefine any of its fixtures or test methods. For instance, on Redshift, we need to explicitly cast a column in the fixture input seed to use data type `varchar(64)`: tests/functional/adapter/test\_basic.py ```python import pytest from dbt.tests.adapter.basic.files import seeds_base_csv, seeds_added_csv, seeds_newcolumns_csv from dbt.tests.adapter.basic.test_snapshot_check_cols import BaseSnapshotCheckCols # set the datatype of the name column in the 'added' seed so it # can hold the '_update' that's added schema_seed_added_yml = """ version: 2 seeds: - name: added config: column_types: name: varchar(64) """ class TestSnapshotCheckColsRedshift(BaseSnapshotCheckCols): # Redshift defines the 'name' column such that it's not big enough # to hold the '_update' added in the test. @pytest.fixture(scope="class") def models(self): return { "base.csv": seeds_base_csv, "added.csv": seeds_added_csv, "seeds.yml": schema_seed_added_yml, } ``` As another example, the `dbt-bigquery` adapter asks users to "authorize" replacing a table with a view by supplying the `--full-refresh` flag. The reason: In the table materialization logic, a view by the same name must first be dropped; if the table query fails, the model will be missing. Knowing this possibility, the "base" test case offers a `require_full_refresh` switch on the `test_config` fixture class. For BigQuery, we'll switch it on: tests/functional/adapter/test\_basic.py ```python import pytest from dbt.tests.adapter.basic.test_base import BaseSimpleMaterializations class TestSimpleMaterializationsBigQuery(BaseSimpleMaterializations): @pytest.fixture(scope="class") def test_config(self): # effect: add '--full-refresh' flag in requisite 'dbt run' step return {"require_full_refresh": True} ``` It's always worth asking whether the required modifications represent gaps in perceived or expected dbt functionality. Are these simple implementation details, which any user of this database would understand? Are they limitations worth documenting? If, on the other hand, they represent poor assumptions in the "basic" test cases, which fail to account for a common pattern in other types of databases-—please open an issue or PR in the `dbt-labs/dbt` repository on GitHub. ##### Running with multiple profiles Some databases support multiple connection methods, which map to actually different functionality behind the scenes. For instance, the `dbt-spark` adapter supports connections to Apache Spark clusters *and* Databricks runtimes, which supports additional functionality out of the box, enabled by the Delta file format. tests/conftest.py ```python def pytest_addoption(parser): parser.addoption("--profile", action="store", default="apache_spark", type=str) # Using @pytest.mark.skip_profile('apache_spark') uses the 'skip_by_profile_type' # autouse fixture below def pytest_configure(config): config.addinivalue_line( "markers", "skip_profile(profile): skip test for the given profile", ) @pytest.fixture(scope="session") def dbt_profile_target(request): profile_type = request.config.getoption("--profile") elif profile_type == "databricks_sql_endpoint": target = databricks_sql_endpoint_target() elif profile_type == "apache_spark": target = apache_spark_target() else: raise ValueError(f"Invalid profile type '{profile_type}'") return target def apache_spark_target(): return { "type": "spark", "host": "localhost", ... } def databricks_sql_endpoint_target(): return { "type": "spark", "host": os.getenv("DBT_DATABRICKS_HOST_NAME"), ... } @pytest.fixture(autouse=True) def skip_by_profile_type(request): profile_type = request.config.getoption("--profile") if request.node.get_closest_marker("skip_profile"): for skip_profile_type in request.node.get_closest_marker("skip_profile").args: if skip_profile_type == profile_type: pytest.skip("skipped on '{profile_type}' profile") ``` If there are tests that *shouldn't* run for a given profile: tests/functional/adapter/basic.py ```python # Snapshots require access to the Delta file format, available on our Databricks connection, # so let's skip on Apache Spark @pytest.mark.skip_profile('apache_spark') class TestSnapshotCheckColsSpark(BaseSnapshotCheckCols): @pytest.fixture(scope="class") def project_config_update(self): return { "seeds": { "+file_format": "delta", }, "snapshots": { "+file_format": "delta", } } ``` Finally: ```sh python3 -m pytest tests/functional --profile apache_spark python3 -m pytest tests/functional --profile databricks_sql_endpoint ``` #### Document a new adapter If you've already built, and tested your adapter, it's time to document it so the dbt community will know that it exists and how to use it. ##### Making your adapter available Many community members maintain their adapter plugins under open source licenses. If you're interested in doing this, we recommend: * Hosting on a public git provider (for example, GitHub or Gitlab) * Publishing to [PyPI](https://pypi.org/) * Adding to the list of ["Supported Data Platforms"](https://docs.getdbt.com/docs/supported-data-platforms.md#community-supported) (more info below) ##### General Guidelines To best inform the dbt community of the new adapter, you should contribute to the dbt's open-source documentation site, which uses the [Docusaurus project](https://docusaurus.io/). This is the site you're currently on! ##### Conventions Each `.md` file you create needs a header as shown below. The document id will also need to be added to the config file: `website/sidebars.js`. ```md --- title: "Documenting a new adapter" id: "documenting-a-new-adapter" --- ``` ##### Single Source of Truth We ask our adapter maintainers to use the [docs.getdbt.com repo](https://github.com/dbt-labs/docs.getdbt.com) (i.e. this site) as the single-source-of-truth for documentation rather than having to maintain the same set of information in three different places. The adapter repo's `README.md` and the data platform's documentation pages should simply link to the corresponding page on this docs site. Keep reading for more information on what should and shouldn't be included on the dbt docs site. ##### Assumed Knowledge To simplify things, assume the reader of this documentation already knows how both dbt and your data platform works. There's already great material for how to learn dbt and the data platform out there. The documentation we're asking you to add should be what a user who is already profiecient in both dbt and your data platform would need to know in order to use both. Effectively that boils down to two things: how to connect, and how to configure. ##### Topics and Pages to Cover The following subjects need to be addressed across three pages of this docs site to have your data platform be listed on our documentation. After the corresponding pull request is merged, we ask that you link to these pages from your adapter repo's `README` as well as from your product documentation. To contribute, all you will have to do make the changes listed in the table below. | How To... | File to change within `/website/docs/` | Action | Info to include | | -------------------- | --------------------------------------------------------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Connect | `/docs/local/connect-data-platform/{MY-DATA-PLATFORM}-setup.md` | Create | Give all information needed to define a target in `~/.dbt/profiles.yml` and get `dbt debug` to connect to the database successfully. All possible configurations should be mentioned. | | Configure | `reference/resource-configs/{MY-DATA-PLATFORM}-configs.md` | Create | What options and configuration specific to your data platform do users need to know? e.g. table distribution and indexing options, column\_quoting policy, which incremental strategies are supported | | Discover and Install | `docs/supported-data-platforms.md` | Modify | Is it a vendor- or community- supported adapter? How to install Python adapter package? Ideally with pip and PyPI hosted package, but can also use `git+` link to GitHub Repo | | Add link to sidebar | `website/sidebars.js` | Modify | Add the document id to the correct location in the sidebar menu | For example say I want to document my new adapter: `dbt-ders`. For the "Connect" page, I will make a new Markdown file, `ders-setup.md` and add it to the `/website/docs/local/connect-data-platform/` directory. ##### Example PRs to add new adapter documentation Below are some recent pull requests made by partners to document their data platform's adapter: * [TiDB](https://github.com/dbt-labs/docs.getdbt.com/pull/1309) * [SingleStore](https://github.com/dbt-labs/docs.getdbt.com/pull/1044) * [Firebolt](https://github.com/dbt-labs/docs.getdbt.com/pull/941) Note — Use the following re-usable component to auto-fill the frontmatter content on your new page: ```markdown import SetUpPages from '/snippets/_setup-pages-intro.md'; <SetUpPages meta={frontMatter.meta} /> ``` #### Promote a new adapter The most important thing here is recognizing that people are successful in the community when they join, first and foremost, to engage authentically. What does authentic engagement look like? It’s challenging to define explicit rules. One good rule of thumb is to treat people with dignity and respect. Contributors to the community should think of contribution *as the end itself,* not a means toward other business KPIs (leads, community members, etc.). [We are a mission-driven company.](https://www.getdbt.com/dbt-labs/values/) Some ways to know if you’re authentically engaging: * Is an engagement’s *primary* purpose of sharing knowledge and resources or building brand engagement? * Imagine you didn’t work at the org you do — can you imagine yourself still writing this? * Is it written in formal / marketing language, or does it sound like you, the human? ##### Who should join the dbt community slack? * People who have insight into what it means to do hands-on [analytics engineering](https://www.getdbt.com/analytics-engineering/) work The dbt Community Slack workspace is fundamentally a place for analytics practitioners to interact with each other — the closer the users are in the community to actual data/analytics engineering work, the more natural their engagement will be (leading to better outcomes for partners and the community). * DevRel practitioners with strong focus DevRel practitioners often have a strong analytics background and a good understanding of the community. It’s essential to be sure they are focused on *contributing,* not on driving community metrics for partner org (such as signing people up for their slack or events). The metrics will rise naturally through authentic engagement. * Founder and executives who are interested in directly engaging with the community This is either incredibly successful or not at all depending on the profile of the founder. Typically, this works best when the founder has a practitioner-level of technical understanding and is interested in joining not to promote, but to learn and hear from users. * Software Engineers at partner products that are building and supporting integrations with either dbt v1 or the dbt platform This is successful when the engineers are familiar with dbt as a product or at least have taken our training course. The Slack is often a place where end-user questions and feedback is initially shared, so it is recommended that someone technical from the team be present. There are also a handful of channels aimed at those building integrations, which tend to be a font of knowledge. ##### Who might struggle in the dbt community * People in marketing roles dbt Slack is not a marketing channel. Attempts to use it as such invariably fall flat and can even lead to people having a negative view of a product. This doesn’t mean that dbt can’t serve marketing objectives, but a long-term commitment to engagement is the only proven method to do this sustainably. * People in product roles The dbt Community can be an invaluable source of feedback on a product. There are two primary ways this can happen — organically (community members proactively suggesting a new feature) and via direct calls for feedback and user research. Immediate calls for engagement must be done in your dedicated #tools channel. Direct calls should be used sparingly, as they can overwhelm more organic discussions and feedback. ##### Who is the audience for an adapter release? A new adapter is likely to drive huge community interest from several groups of people: * People who are currently using the database that the adapter is supporting * People who may be adopting the database in the near future. * People who are interested in dbt development in general. The database users will be your primary audience and the most helpful in achieving success. Engage them directly in the adapter’s dedicated Slack channel. If one does not exist already, reach out in #channel-requests, and we will get one made for you and include it in an announcement about new channels. The final group is where non-slack community engagement becomes important. Twitter and LinkedIn are both great places to interact with a broad audience. A well-orchestrated adapter release can generate impactful and authentic engagement. ##### How to message the initial rollout and follow-up content Tell a story that engages dbt users and the community. Highlight new use cases and functionality unlocked by the adapter in a way that will resonate with each segment. * Existing users of your technology who are new to dbt * Provide a general overview of the value dbt will deliver to your users. This can lean on dbt's messaging and talking points which are laid out in the [dbt viewpoint.](https://docs.getdbt.com/community/resources/viewpoint.md) * Give examples of a rollout that speaks to the overall value of dbt and your product. * Users who are already familiar with dbt and the community * Consider unique use cases or advantages your adapter provide over existing adapters. Who will be excited for this? * Contribute to the dbt Community and ensure that dbt users on your adapter are well supported (tutorial content, packages, documentation, etc). * Example of a rollout that is compelling for those familiar with dbt: [Firebolt](https://www.linkedin.com/feed/update/urn:li:activity:6879090752459182080/) ##### Tactically manage distribution of content about new or existing adapters There are tactical pieces on how and where to share that help ensure success. * On slack: * \#i-made-this channel — this channel has a policy against “marketing” and “content marketing” posts, but it should be successful if you write your content with the above guidelines in mind. Even with that, it’s important to post here sparingly. * Your own database / tool channel — this is where the people who have opted in to receive communications from you and always a great place to share things that are relevant to them. * On social media: * Twitter * LinkedIn * Social media posts *from the author* or an individual connected to the project tend to have better engagement than posts from a company or organization account. * Ask your partner representative about: * Retweets and shares from the official dbt Labs accounts. * Flagging posts internally at dbt Labs to get individual employees to share. ###### Measuring engagement You don’t need 1000 people in a channel to succeed, but you need at least a few active participants who can make it feel lived in. If you’re comfortable working in public, this could be members of your team, or it can be a few people who you know that are highly engaged and would be interested in participating. Having even 2 or 3 regulars hanging out in a channel is all that’s needed for a successful start and is, in fact, much more impactful than 250 people that never post. ##### How to announce a new adapter We’d recommend *against* boilerplate announcements and encourage finding a unique voice. That being said, there are a couple of things that we’d want to include: * A summary of the value prop of your database / technology for users who aren’t familiar. * The personas that might be interested in this news. * A description of what the adapter *is*. For example: > With the release of our new dbt adapter, you’ll be able to to use dbt to model and transform your data in \[name-of-your-org] * Particular or unique use cases or functionality unlocked by the adapter. * Plans for future / ongoing support / development. * The link to the documentation for using the adapter on the dbt Labs docs site. * An announcement blog. ###### Announcing new release versions of existing adapters This can vary substantially depending on the nature of the release but a good baseline is the types of release messages that [we put out in the #dbt-releases](https://getdbt.slack.com/archives/C37J8BQEL/p1651242161526509) channel. ![Full Release Post](/assets/images/0-full-release-notes-1cc8cb263cb178df48deda1f69875c99.png) Breaking this down: * Visually distinctive announcement - make it clear this is a release [![title](/img/adapter-guide/1-announcement.png?v=2 "title")](#)title * Short written description of what is in the release [![description](/img/adapter-guide/2-short-description.png?v=2 "description")](#)description * Links to additional resources [![more resources](/img/adapter-guide/3-additional-resources.png?v=2 "more resources")](#)more resources * Implementation instructions: [![more installation](/img/adapter-guide/4-installation.png?v=2 "more installation")](#)more installation * Contributor recognition (if applicable) [![thank yous](/img/adapter-guide/6-thank-contribs.png?v=2 "thank yous")](#)thank yous #### Build a trusted adapter The Trusted Adapter Program exists to allow adapter maintainers to demonstrate to the dbt community that your adapter is trusted to be used in production. The very first data platform dbt supported was Redshift followed quickly by Postgres ([dbt-labs/dbt#174](https://github.com/dbt-labs/dbt/pull/174)). In 2017, back when dbt Labs (née Fishtown Analytics) was still a data consultancy, we added support for Snowflake and BigQuery. We also turned dbt's database support into an adapter framework ([dbt-labs/dbt#259](https://github.com/dbt-labs/dbt/pull/259/)), and a plugin system a few years later. For years, dbt Labs specialized in those four data platforms and became experts in them. However, the surface area of all possible databases, their respective nuances, and keeping them up-to-date and bug-free is a Herculean and/or Sisyphean task that couldn't be done by a single person or even a single team! Enter the dbt community which enables dbt v1 to work on more than 30 different databases (32 as of Sep '22)! Free and open-source tools for the data professional are increasingly abundant. This is by-and-large a *good thing*, however it requires due diligence that wasn't required in a paid-license, closed-source software world. Before taking a dependency on an open-source project is is important to determine the answer to the following questions: 1. Does it work? 2. Does it meet my team's specific use case? 3. Does anyone "own" the code, or is anyone liable for ensuring it works? 4. Do bugs get fixed quickly? 5. Does it stay up-to-date with new dbt features? 6. Is the usage substantial enough to self-sustain? 7. What risks do I take on by taking a dependency on this library? These are valid, important questions to answer—especially given that `dbt-core` itself only put out its first stable release (major version v1.0) in December 2021! Indeed, up until now, the majority of new user questions in database-specific channels are some form of: * "How mature is `dbt-<ADAPTER>`? Any gotchas I should be aware of before I start exploring?" * "has anyone here used `dbt-<ADAPTER>` for production models?" * "I've been playing with `dbt-<ADAPTER>` -- I was able to install and run my initial experiments. I noticed that there are certain features mentioned on the documentation that are marked as 'not ok' or 'not tested'. What are the risks? I'd love to make a statement on my team to adopt dbt, but I'm pretty sure questions will be asked around the possible limitations of the adapter or if there are other companies out there using dbt with Oracle DB in production, etc." There has been a tendency to trust the dbt Labs-maintained adapters over community- and vendor-supported adapters, but repo ownership is only one among many indicators of software quality. We aim to help our users feel well-informed as to the caliber of an adapter with a new program. ##### What it means to be trusted By opting into the below, you agree to this, and we take you at your word. dbt Labs reserves the right to remove an adapter from the trusted adapter list at any time, should any of the below guidelines not be met. ##### Feature Completeness To be considered for the Trusted Adapter Program, the adapter must cover the essential functionality of dbt v1 given below, with best effort given to support the entire feature set. Essential functionality includes (but is not limited to the following features): * table, view, and seed materializations * dbt tests The adapter should have the required documentation for connecting and configuring the adapter. The dbt docs site should be the single source of truth for this information. These docs should be kept up-to-date. Proceed to the "Document a new adapter" step for more information. ##### Release cadence Keeping an adapter up-to-date with the latest features of dbt, as defined in [dbt-adapters](https://github.com/dbt-labs/dbt-adapters), is an integral part of being a trusted adapter. We encourage adapter maintainers to keep track of new dbt-adapter releases and support new features relevant to their platform, ensuring users have the best version of dbt. Before [dbt v1.8](https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/upgrading-to-v1.8.md#new-dbt-core-adapter-installation-procedure), adapter versions needed to match the semantic versioning of dbt v1. After v1.8, this is no longer required. This means users can use an adapter on v1.8+ with a different version of dbt v1.8+. For example, a user could use dbt-core v1.9 with dbt-postgres v1.8. ##### Community responsiveness On a best effort basis, active participation and engagement with the dbt Community across the following forums: * Being responsive to feedback and supporting user enablement in dbt Community’s Slack workspace * Responding with comments to issues raised in public dbt adapter code repository * Merging in code contributions from community members as deemed appropriate ##### Security Practices Trusted adapters will not do any of the following: * Output to logs or file either access credentials information to or data from the underlying data platform itself. * Make API calls other than those expressly required for using dbt features (adapters may not add additional logging) * Obfuscate code and/or functionality so as to avoid detection Additionally, to avoid supply-chain attacks: * Use an automated service to keep Python dependencies up-to-date (such as Dependabot or similar), * Publish directly to PyPI from the dbt adapter code repository by using trusted CI/CD process (such as GitHub actions) * Restrict admin access to both the respective code (GitHub) and package (PyPI) repositories * Identify and mitigate security vulnerabilities by use of a static code analyzing tool (such as Snyk) as part of a CI/CD process ##### Other considerations The adapter repository is: * open-souce licensed, * published to PyPI, and * automatically tests the codebase against dbt Lab's provided adapter test suite ##### How to get an adapter on the trusted list Open an issue on the [docs.getdbt.com GitHub repository](https://github.com/dbt-labs/docs.getdbt.com) using the "Add adapter to Trusted list" template. In addition to contact information, it will ask confirm that you agree to the following. 1. my adapter meet the guidelines given above 2. I will make best reasonable effort that this continues to be so 3. checkbox: I acknowledge that dbt Labs reserves the right to remove an adapter from the trusted adapter list at any time, should any of the above guidelines not be met. The approval workflow is as follows: 1. create and populate the template-created issue 2. dbt Labs will respond as quickly as possible (maximally four weeks, though likely faster) 3. If approved, dbt Labs will create and merge a Pull request to formally add the adapter to the list. ##### Getting help for my trusted adapter Ask your question in #adapter-ecosystem channel of the dbt community Slack. --- ### Building dbt packages [Back to guides](https://docs.getdbt.com/guides.md) Advanced [Menu ]() #### Introduction Creating packages is an **advanced use of dbt**. If you're new to the tool, we recommend that you first use the product for your own analytics before attempting to create a package for others. ##### Prerequisites A strong understanding of: * [packages](https://docs.getdbt.com/docs/build/packages.md) * administering a repository on GitHub * [semantic versioning](https://semver.org/) ##### Assess whether a package is the right solution Packages typically contain either: * macros that solve a particular analytics engineering problem — for example, [auditing the results of a query](https://hub.getdbt.com/dbt-labs/audit_helper/latest/), [generating code](https://hub.getdbt.com/dbt-labs/codegen/latest/), or [adding additional schema tests to a dbt project](https://hub.getdbt.com/calogica/dbt_expectations/latest/). * models for a common dataset — for example a dataset for software products like [MailChimp](https://hub.getdbt.com/fivetran/mailchimp/latest/) or [Snowplow](https://hub.getdbt.com/dbt-labs/snowplow/latest/), or even models for metadata about your data stack like [Snowflake query spend](https://hub.getdbt.com/gitlabhq/snowflake_spend/latest/) and [the artifacts produced by `dbt run`](https://hub.getdbt.com/tailsdotcom/dbt_artifacts/latest/). In general, there should be a shared set of industry-standard metrics that you can model (e.g. email open rate). We also recommend ensuring your package is compatible with [dbt v2](https://docs.getdbt.com/docs/introduction.md) and [dbt v1](https://docs.getdbt.com/docs/local/install-dbt.md). To ensure dbt v2 compatibility, you can follow the steps in the [v2 package upgrade guide](https://docs.getdbt.com/guides/dbt-package-compat.md). Packages are *not* a good fit for sharing models that contain business-specific logic, for example, writing code for marketing attribution, or monthly recurring revenue. Instead, consider sharing a blog post and a link to a sample repo, rather than bundling this code as a package (here's our blog post on [marketing attribution](https://blog.getdbt.com/modeling-marketing-attribution/) as an example). #### Create your new project Using the command line for package development We tend to use the command line interface for package development. The development workflow often involves installing a local copy of your package in another dbt project — at present dbt is not designed for this workflow. 1. Use the [dbt init](https://docs.getdbt.com/reference/commands/init.md) command to create a new dbt project, which will be your package: ```shell $ dbt init [package_name] ``` 2. Create a public GitHub¹ repo, named `dbt-<package-name>`, e.g. `dbt-mailchimp`. Follow the GitHub instructions to link this to the dbt project you just created. 3. Update the `name:` of the project in `dbt_project.yml` to your package name, e.g. `mailchimp`. 4. Define the allowed dbt versions by using the [`require-dbt-version` config](https://docs.getdbt.com/reference/project-configs/require-dbt-version.md). ¹Currently, our package registry only supports packages that are hosted in GitHub. #### Develop your package We recommend that first-time package authors first develop macros and models for use in their own dbt project. Once your new package is created, you can get to work on moving them across, implementing some additional package-specific design patterns along the way. When working on your package, we often find it useful to install a local copy of the package in another dbt project — this workflow is described [here](https://discourse.getdbt.com/t/contributing-to-an-external-dbt-package/657). ##### Ensure dbt v2 compatibility If you're building a package, we recommend you ensure it's compatible with [dbt v2](https://docs.getdbt.com/docs/introduction.md) and [dbt v1](https://docs.getdbt.com/docs/local/install-dbt.md). To ensure dbt v2 compatibility, you can follow the steps in the [v2 package upgrade guide](https://docs.getdbt.com/guides/dbt-package-compat.md). Doing so will ensure your package is compatible with dbt v2 (and dbt v1), but will be displayed with a dbt v2-compatible badge in dbt package hub. ##### Follow best practices *Modeling packages only* Use our [dbt coding conventions](https://github.com/dbt-labs/corp/blob/main/dbt_style_guide.md), our article on [how we structure our dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md), and our [best practices](https://docs.getdbt.com/best-practices.md) for all of our advice on how to build your dbt project. This is where it comes in especially handy to have worked on your own dbt project previously. ##### Make the location of raw data configurable *Modeling packages only* Not every user of your package is going to store their Mailchimp data in a schema named `mailchimp`. As such, you'll need to make the location of raw data configurable. We recommend using [sources](https://docs.getdbt.com/docs/build/sources.md) and [variables](https://docs.getdbt.com/docs/build/project-variables.md) to achieve this. Check out [this package](https://github.com/fivetran/dbt_facebook_ads_source/blob/main/models/src_facebook_ads.yml#L5-L6) for an example — notably, the README [includes instructions](https://github.com/fivetran/dbt_facebook_ads_source#configuration) on how to override the default schema from a `dbt_project.yml` file. ##### Install upstream packages from hub.getdbt.com If your package relies on another package (for example, you use some of the cross-database macros from [dbt-utils](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/)), we recommend you install the package from [hub.getdbt.com](https://hub.getdbt.com), specifying a version range like so: packages.yml ```yaml packages: - package: dbt-labs/dbt_utils version: [">0.6.5", "0.7.0"] ``` When packages are installed from hub.getdbt.com, dbt is able to handle duplicate dependencies. ##### Implement cross-database compatibility Many SQL functions are specific to a particular database. For example, the function name and order of arguments to calculate the difference between two dates varies between Redshift, Snowflake and BigQuery, and no similar function exists on Postgres! If you wish to support multiple warehouses, we have a number of tricks up our sleeve: * We've written a number of macros that compile to valid SQL snippets on each of the original four adapters. Where possible, leverage these macros. * If you need to implement cross-database compatibility for one of your macros, use the [`adapter.dispatch` macro](https://docs.getdbt.com/reference/dbt-jinja-functions/dispatch.md) to achieve this. Check out the cross-database macros in dbt-utils for examples. * If you're working on a modeling package, you may notice that you need write different models for each warehouse (for example, if the EL tool you are working with stores data differently on each warehouse). In this case, you can write different versions of each model, and use the [`enabled` config](https://docs.getdbt.com/reference/resource-configs/enabled.md), in combination with [`target.type`](https://docs.getdbt.com/reference/dbt-jinja-functions/target.md) to enable the correct models — check out [this package](https://github.com/fivetran/dbt_facebook_ads_creative_history/blob/main/dbt_project.yml#L11-L16) as an example. If your package has only been written to work for one data warehouse, make sure you document this in your package README. ##### Use specific model names *Modeling packages only* Many datasets have a concept of a "user" or "account" or "session". To make sure things are unambiguous in dbt, prefix all of your models with `[package_name]_`. For example, `mailchimp_campaigns.sql` is a good name for a model, whereas `campaigns.sql` is not. ##### Default to views *Modeling packages only* dbt makes it possible for users of your package to override your model materialization settings. In general, default to materializing models as `view`s instead of `table`s. The major exception to this is when working with data sources that benefit from incremental modeling (for example, web page views). Implementing incremental logic on behalf of your end users is likely to be helpful in this case. ##### Test and document your package It's critical that you [test](https://docs.getdbt.com/docs/build/data-tests.md) your models and sources. This will give your end users confidence that your package is actually working on top of their dataset as intended. Further, adding [documentation](https://docs.getdbt.com/docs/build/documentation.md) via descriptions will help communicate your package to end users, and benefit their stakeholders that use the outputs of this package. ##### Include useful GitHub artifacts Over time, we've developed a set of useful GitHub artifacts that make administering our packages easier for us. In particular, we ensure that we include: * A useful README, that has: * installation instructions that refer to the latest version of the package on hub.getdbt.com, and includes any configurations requires ([example](https://github.com/dbt-labs/segment)) * Usage examples for any macros ([example](https://github.com/dbt-labs/dbt-audit-helper#macros)) * Descriptions of the main models included in the package ([example](https://github.com/dbt-labs/snowplow)) * GitHub templates, including PR templates and issue templates ([example](https://github.com/dbt-labs/dbt-audit-helper/tree/master/.github)) #### Add integration tests *Optional* We recommend that you implement integration tests to confirm that the package works as expected — this is an even *more* advanced step, so you may find that you build up to this. This pattern can be seen most packages, including the [`audit-helper`](https://github.com/dbt-labs/dbt-audit-helper/tree/master/integration_tests) and [`snowplow`](https://github.com/dbt-labs/snowplow/tree/master/integration_tests) packages. As a rough guide: 1. Create a subdirectory named `integration_tests` 2. In this subdirectory, create a new dbt project — you can use the `dbt init` command to do this. However, our preferred method is to copy the files from an existing `integration_tests` project, like the ones [here](https://github.com/dbt-labs/dbt-codegen/tree/HEAD/integration_tests) (removing the contents of the `macros`, `models` and `tests` folders since they are project-specific) 3. Install the package in the `integration_tests` subdirectory by using the `local` syntax, and then running `dbt deps` packages.yml ```yml packages: - local: ../ # this means "one directory above the current directory" ``` 4. Add resources to the package (seeds, models, tests) so that you can successfully run your project, and compare the output with what you expect. The exact approach here will vary depending on your packages. In general you will find that you need to: * Add mock data via a [seed](https://docs.getdbt.com/docs/build/seeds.md) with a few sample (anonymized) records. Configure the `integration_tests` project to point to the seeds instead of raw data tables. * Add more seeds that represent the expected output of your models, and use the [dbt\_utils.equality](https://github.com/dbt-labs/dbt-utils#equality-source) test to confirm the output of your package, and the expected output matches. 5. Confirm that you can run `dbt run` and `dbt test` from your command line successfully. 6. (Optional) Use a CI tool, like CircleCI or GitHub Actions, to automate running your dbt project when you open a new Pull Request. For inspiration, check out one of our [CircleCI configs](https://github.com/dbt-labs/snowplow/blob/main/.circleci/config.yml), which runs tests against our four main warehouses. Note: this is an advanced step — if you are going down this path, you may find it useful to say hi on [dbt Slack](https://community.getdbt.com/). #### Deploy the docs for your package *Optional* A dbt docs site can help a prospective user of your package understand the code you've written. As such, we recommend that you deploy the site generated by `dbt docs generate` and link to the deployed site from your package. The easiest way we've found to do this is to use [GitHub Pages](https://pages.github.com/). 1. On a new git branch, run `dbt docs generate`. If you have integration tests set up (above), use the integration-test project to do this. 2. Move the following files into a directory named `docs` ([example](https://github.com/fivetran/dbt_ad_reporting/tree/HEAD/docs)): `catalog.json`, `index.html`, `manifest.json`, `run_results.json`. 3. Merge these changes into the main branch 4. Enable GitHub pages on the repo in the settings tab, and point it to the “docs” subdirectory 5. GitHub should then deploy the docs at `<org-name>.github.io/<repo-name>`, like so: [fivetran.github.io/dbt\_ad\_reporting](https://fivetran.github.io/dbt_ad_reporting/) #### Release your package Create a new [release](https://docs.github.com/en/github/administering-a-repository/managing-releases-in-a-repository) once you are ready for others to use your work! Be sure to use [semantic versioning](https://semver.org/) when naming your release. In particular, if new changes will cause errors for users of earlier versions of the package, be sure to use *at least* a minor release (e.g. go from `0.1.1` to `0.2.0`). The release notes should contain an overview of the changes introduced in the new version. Be sure to call out any changes that break the existing interface! #### Add the package to hub.getdbt.com Our package registry, [hub.getdbt.com](https://hub.getdbt.com/), gets updated by the [hubcap script](https://github.com/dbt-labs/hubcap). To add your package to hub.getdbt.com, create a PR on the [hubcap repository](https://github.com/dbt-labs/hubcap) to include it in the `hub.json` file. --- ### Clone the Jaffle Shop sample project [Back to guides](https://docs.getdbt.com/guides.md) dbt Beginner Beginner [Menu ]() #### What is Jaffle Shop? [Jaffle Shop](https://github.com/dbt-labs/jaffle-shop) is dbt Labs' canonical sample dbt project for a fictional cafe business. This guide shows you how to clone the project using Git from GitHub or GitLab. Cloning downloads the dbt project files to your machine: models, seeds, tests, and configuration. It doesn't install dbt or set up a database. If you only need the project files, follow the [Clone the repository](#clone-the-repository) and [Verify the clone](#verify-the-clone) sections. To run dbt commands like `dbt seed`, `dbt run`, and `dbt test`, you also need dbt installed and a database to connect to. Refer to [Next steps](#next-steps) for links to warehouse quickstarts. ##### Related content * [Install dbt v1](https://docs.getdbt.com/docs/local/install-dbt.md) * [About dbt projects](https://docs.getdbt.com/docs/build/projects.md) * [Example dbt projects](https://docs.getdbt.com/faqs/Project/example-projects.md) #### Prerequisites * [Git](https://git-scm.com/downloads) installed * A terminal * A [GitHub](https://github.com/join) or [GitLab](https://gitlab.com/users/sign_up) account (only if you plan to fork the repo, push changes, or clone from a private repository) Verify Git is installed: ```bash git --version ``` #### Clone the repository ##### GitHub 1. Open your terminal and navigate to where you keep projects: ```bash cd ~/Documents/Github ``` 2. Clone the repository: ```bash git clone https://github.com/dbt-labs/jaffle-shop.git ``` 3. Change into the project directory: ```bash cd jaffle-shop ``` ##### GitLab If your organization hosts Jaffle Shop on GitLab, or you've forked the repo there, clone from your GitLab URL instead: 1. Open your terminal and navigate to where you keep projects: ```bash cd ~/Documents/Github ``` 2. Clone the repository (replace `YOUR_USERNAME` with your GitLab username or group): ```bash git clone https://gitlab.com/YOUR_USERNAME/jaffle-shop.git ``` 3. Change into the project directory: ```bash cd jaffle-shop ``` ##### Other platforms If your organization has mirrored or forked Jaffle Shop on Bitbucket, Azure DevOps, or another Git host, use the clone URL from that platform. The `git clone` command works the same way: 1. Open your terminal and navigate to where you keep projects: ```bash cd ~/Documents/Github ``` 2. Clone the repository: ```bash git clone <your-repo-clone-url> ``` 3. Change into the project directory: ```bash cd jaffle-shop ``` #### Verify the clone Confirm the project files are present: ```bash ls ``` You should see files and folders including: * `dbt_project.yml` * `models/` * `seeds/` * `packages.yml` Cloning is complete! 🎉 You now have the Jaffle Shop project files on your machine. #### Explore the project structure Before you install dbt or connect a database, take a quick look at what you cloned. Jaffle Shop is a fictional cafe business. The project contains 13 models that transform cafe data about customers, locations (stores), products, supplies, and orders. It also includes: * **Seeds:** Six CSV files under `seeds/jaffle-data/` that provide the raw sample data. * **Data tests and unit tests:** YAML alongside the models that check uniqueness, not-null values, and a few unit-test cases. * **Macros:** Small helper macros in `macros/`. * **Packages:** Dependencies listed in `packages.yml` (install later with `dbt deps`). A simplified view of the project looks like this: ```text jaffle-shop/ ├── dbt_project.yml ├── packages.yml ├── models/ │ ├── staging/ # Clean and rename raw tables │ │ ├── stg_customers.sql │ │ ├── stg_orders.sql │ │ ├── stg_order_items.sql │ │ ├── stg_products.sql │ │ ├── stg_locations.sql │ │ ├── stg_supplies.sql │ │ └── __sources.yml │ └── marts/ # Business-ready tables for the cafe │ ├── customers.sql │ ├── orders.sql │ ├── order_items.sql │ ├── products.sql │ ├── locations.sql │ ├── supplies.sql │ └── metricflow_time_spine.sql ├── seeds/ │ └── jaffle-data/ # Sample CSV data (customers, orders, and more) ├── macros/ ├── analyses/ └── data-tests/ ``` Staging models sit closest to the raw seed data. Marts models join and shape that data into the tables you use for analysis. You don't need to read every file yet. This layout follows a standard [dbt project](https://docs.getdbt.com/docs/build/projects.md) pattern that appears in many real-world projects. #### Next steps To run or develop the project, you need dbt installed and a database connected. These links can help: * **[Install self-hosted dbt](https://docs.getdbt.com/docs/local/install-dbt.md):** Cloning doesn't install dbt. You need it to run commands. * **[Set up a virtual environment](https://docs.getdbt.com/docs/local/install-dbt.md):** Keeps dbt separate from other Python projects on your machine. * **[About dbt deps command](https://docs.getdbt.com/reference/commands/deps.md):** The repo lists packages in `packages.yml`. Run `dbt deps` after you install dbt. * **[About dbt projects](https://docs.getdbt.com/docs/build/projects.md):** Learn the project structure before editing models. * **[dbt Learn](https://learn.getdbt.com/):** Interactive courses for new users. ##### Warehouse quickstarts To run the project, you need a database and adapter configured in `profiles.yml`. Choose the quickstart for your warehouse or local setup: * [Quickstart for dbt and Snowflake](https://docs.getdbt.com/guides/snowflake.md) * [Quickstart for dbt and BigQuery](https://docs.getdbt.com/guides/bigquery.md) * [Quickstart for dbt and Databricks](https://docs.getdbt.com/guides/databricks.md) * [Quickstart for dbt and Redshift](https://docs.getdbt.com/guides/redshift.md) * [Quickstart for dbt v1 using DuckDB](https://docs.getdbt.com/guides/duckdb.md): Clone [`jaffle_shop_duckdb`](https://github.com/dbt-labs/jaffle_shop_duckdb) and follow the guide. You can also browse [all guides](https://docs.getdbt.com/guides.md) or other [example dbt projects](https://docs.getdbt.com/faqs/Project/example-projects.md). #### Optional cleanup If you cloned the repo only to test these steps and don't need the project anymore, you can remove the folder: ```bash rm -rf ~/Documents/Github/jaffle-shop ``` This deletes the cloned project from your machine. It doesn't affect the GitHub or GitLab repository. --- ### Connect dbt MCP to dbt platform [Back to guides](https://docs.getdbt.com/guides.md) dbt MCP AI Quickstart Beginner [Menu ]() This quickstart uses the local MCP server: it runs on your machine using `uvx dbt-mcp`, connects to your dbt platform for Semantic Layer, Discovery, and SQL, and optionally runs local dbt. For self-hosted CLI only (with or without a dbt platform account), see [Run self-hosted dbt](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-cli.md) or [Run self-hosted dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md). To configure or disable specific tools, see the [Environment variables reference](https://docs.getdbt.com/docs/dbt-ai/mcp-environment-variables.md). #### Prerequisites * [Install uv](https://docs.astral.sh/uv/getting-started/installation/) * A [dbt platform account](https://www.getdbt.com/signup) * For OAuth connections: * MCP OAuth is available for Starter, Enterprise, and Enterprise+ plans. #### Step 1: Choose your auth method and configure ##### OAuth *MCP OAuth is available in public beta for Starter, Enterprise, and Enterprise+ accounts.* OAuth is the fastest setup for dbt platform accounts, no tokens to copy or manage. A browser window opens to authenticate the first time you connect. For OAuth *without* a self-hosted installation, use the [remote MCP server](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-remote.md). Remote MCP OAuth is available in public beta for Starter, Enterprise, and Enterprise+ accounts. If your client does not support OAuth or you need token-based access, use [token-based authentication](https://docs.getdbt.com/docs/dbt-ai/setup-remote-mcp.md#token-based-authentication). Static subdomains required Only accounts with static subdomains (for example, `abc123` in `abc123.us1.dbt.com`) can use OAuth with MCP servers. Follow [these](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) instructions to find your account subdomain. If your account does not have a subdomain, contact support for more information. ###### Find your Access URL 1. Log in to your dbt platform account. 2. Go to **Account settings** and copy your **Access URL** (for example, `abc123.us1.dbt.com`). Multi-cell and DBT\_HOST format * The `DBT_HOST` field accepts both `abc123.us1.dbt.com` and `https://abc123.us1.dbt.com`. * If your Access URL is `abc123.us1.dbt.com`, split it into two variables: * `DBT_HOST=us1.dbt.com` * `MULTICELL_ACCOUNT_PREFIX=abc123` Don't include the account prefix in `DBT_HOST`. For more details, see [multi-cell configuration examples](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#api-and-sql-tool-settings). ###### Add the config to your MCP client ##### Claude Desktop **Option A: Quick install (recommended)** 1. Go to the [latest dbt MCP release](https://github.com/dbt-labs/dbt-mcp/releases/latest) and download `dbt-mcp.mcpb`. 2. Double-click the file to open it in Claude Desktop. 3. Enter your **Access URL** as the dbt platform Host. 4. Enable the server. **Option B: Manual config** 1. In Claude Desktop, go to **Settings** → **Developer** tab → **Edit Config**. 2. Paste the following configuration, replacing `YOUR-ACCESS-URL` with your Access URL: ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "YOUR-ACCESS-URL" } } } } ``` 3. Save and restart Claude Desktop. Config file location: * macOS: `~/Library/Application Support/Claude/claude_desktop_config.json` * Windows: `%APPDATA%\Claude\claude_desktop_config.json` ##### Claude Code Run this command, replacing `YOUR-ACCESS-URL` with your Access URL: ```shell claude mcp add dbt \ -e DBT_HOST=YOUR-ACCESS-URL \ -- uvx dbt-mcp ``` For example, if your Access URL is `abc123.us1.dbt.com`: ```shell claude mcp add dbt \ -e DBT_HOST=abc123.us1.dbt.com \ -- uvx dbt-mcp ``` ##### Cursor Click a link below with Cursor open to auto-configure, then replace the placeholder with your Access URL: * [dbt platform only (OAuth)](cursor://anysphere.cursor-deeplink/mcp/install?name=dbt\&config=eyJlbnYiOnsiREJUX0hPU1QiOiJZT1VSLUFDQ0VTUy1VUkwiLCJESVNBQkxFX0RCVF9DTEkiOiJ0cnVlIn0sImNvbW1hbmQiOiJ1dngiLCJhcmdzIjpbImRidC1tY3AiXX0%3D) — platform features only, no CLI * [dbt platform + CLI (OAuth)](cursor://anysphere.cursor-deeplink/mcp/install?name=dbt\&config=eyJlbnYiOnsiREJUX0hPU1QiOiJZT1VSLUFDQ0VTUy1VUkwiLCJEQlRfUFJPSkVDVF9ESVIiOiIvcGF0aC90by9wcm9qZWN0IiwiREJUX1BBVEgiOiJwYXRoL3RvL2RidC9leGVjdXRhYmxlIn0sImNvbW1hbmQiOiJ1dngiLCJhcmdzIjpbImRidC1tY3AiXX0%3D) — platform features + self-hosted dbt CLI commands After clicking, replace `YOUR-ACCESS-URL` with your actual Access URL (for example, `abc123.us1.dbt.com`) and save. ##### VS Code 1. Open **Settings** → **Features** → **Chat** and ensure **MCP** is enabled. 2. Open the Command Palette (`Ctrl/Cmd + Shift + P`) and select **MCP: Open User Configuration**. 3. Add the following configuration to `mcp.json`: VS Code uses `"servers"`, not `"mcpServers"` ```json { "servers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "YOUR-ACCESS-URL" } } } } ``` Replace `YOUR-ACCESS-URL` with your Access URL (for example, `abc123.us1.dbt.com`) and save. ###### Optional: Add self-hosted dbt CLI commands To also run dbt platform CLI commands (`dbt run`, `dbt build`, `dbt test`, and more), add these two variables to your `env` block: ```json "DBT_PROJECT_DIR": "/path/to/your/dbt/project", "DBT_PATH": "/path/to/your/dbt/executable" ``` Find `DBT_PATH` by running `which dbt` (macOS/Linux) or `where dbt` (Windows). `DBT_PROJECT_DIR` is the folder containing your `dbt_project.yml`. ##### Tokens Token-based auth gives you more control and is better for shared or team setups. You'll need a service token or Personal Access Token (PAT). Which token should I use? * **PAT (Personal Access Token):** Required if you want to use `execute_sql`. Tied to your user account. * **Service token:** Works for all other platform toolsets. Better for shared or team setups. See [Choosing an auth method](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#choose-your-auth-method) for full guidance. ##### Find your paths and IDs You need the following values. See [Finding your IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md) for step-by-step instructions. | Variable | Where to find it | | ----------------- | -------------------------------------------------------------------------- | | `DBT_HOST` | Your dbt platform hostname, found in **Account settings** → **Access URL** | | `DBT_TOKEN` | A service token or PAT from **Account settings** → **API tokens** | | `DBT_PROD_ENV_ID` | Your production environment ID, found in **Deploy** → **Environments** | | `DBT_DEV_ENV_ID` | Your development environment ID (required for `execute_sql`) | | `DBT_USER_ID` | Your numeric user ID (required for `execute_sql`) | | `DBT_ACCOUNT_ID` | Your account ID (required for Admin API tools) | Use values only, not full URLs These variables expect hostnames or numeric IDs — not full URLs: ```bash # ✅ Correct DBT_HOST=cloud.getdbt.com # https://cloud.getdbt.com also works DBT_PROD_ENV_ID=54321 DBT_USER_ID=123 # ❌ Wrong — IDs must be numeric, not full URLs DBT_PROD_ENV_ID=https://cloud.getdbt.com/deploy/12345/projects/67890/environments/54321 DBT_USER_ID=https://cloud.getdbt.com/settings/profile ``` Multi-cell accounts If your Access URL is `abc123.us1.dbt.com`, split it into two variables: * `DBT_HOST=us1.dbt.com` * `MULTICELL_ACCOUNT_PREFIX=abc123` Don't include the account prefix in `DBT_HOST`. For more details, see [multi-cell configuration examples](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#api-and-sql-tool-settings). ##### Add the config to your MCP client Use the configuration below, replacing the placeholder values with your IDs from above. Include only the variables you need: ##### Claude Desktop 1. In Claude Desktop, go to **Settings** → **Developer** tab → **Edit Config**. 2. Paste the following configuration, replacing the placeholder values: ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "cloud.getdbt.com", "DBT_TOKEN": "your-token-here", "DBT_PROD_ENV_ID": "12345", "DBT_DEV_ENV_ID": "67890", "DBT_USER_ID": "123", "DBT_ACCOUNT_ID": "99999" } } } } ``` 3. Save and restart Claude Desktop. Config file location: * macOS: `~/Library/Application Support/Claude/claude_desktop_config.json` * Windows: `%APPDATA%\Claude\claude_desktop_config.json` ##### Claude Code Run this command, replacing the placeholders with your actual values: ```bash claude mcp add dbt \ -e DBT_HOST=cloud.getdbt.com \ -e DBT_TOKEN=your-token-here \ -e DBT_PROD_ENV_ID=12345 \ -- uvx dbt-mcp ``` Add `-e DBT_DEV_ENV_ID=...` and `-e DBT_USER_ID=...` if you use `execute_sql`; add `-e DBT_ACCOUNT_ID=...` for Admin API. ##### Cursor 1. In Cursor, open **Settings** → **MCP** → **Edit config** (or your config file). 2. Paste the following configuration, replacing the placeholder values: ```json { "mcpServers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "cloud.getdbt.com", "DBT_TOKEN": "your-token-here", "DBT_PROD_ENV_ID": "12345", "DBT_DEV_ENV_ID": "67890", "DBT_USER_ID": "123", "DBT_ACCOUNT_ID": "99999" } } } } ``` 3. Save the configuration. ##### VS Code 1. Open **Settings** → **Features** → **Chat** and ensure **MCP** is enabled. 2. Open the Command Palette (`Ctrl/Cmd + Shift + P`) and select **MCP: Open User Configuration**. 3. Add the following configuration to `mcp.json`: VS Code uses `"servers"`, not `"mcpServers"` ```json { "servers": { "dbt": { "command": "uvx", "args": ["dbt-mcp"], "env": { "DBT_HOST": "cloud.getdbt.com", "DBT_TOKEN": "your-token-here", "DBT_PROD_ENV_ID": "12345", "DBT_DEV_ENV_ID": "67890", "DBT_USER_ID": "123", "DBT_ACCOUNT_ID": "99999" } } } } ``` 4. Save `mcp.json` and restart VS Code.  Optional: add self-hosted dbt CLI commands To also run dbt commands (`dbt run`, `dbt build`, `dbt test`, and more), add these two variables to your `env` block: ```json "DBT_PROJECT_DIR": "/path/to/your/dbt/project", "DBT_PATH": "/path/to/your/dbt/executable" ``` Find `DBT_PATH` by running `which dbt` (macOS/Linux) or `where dbt` (Windows). `DBT_PROJECT_DIR` is the folder containing your `dbt_project.yml`. #### Step 2: Authenticate ##### OAuth The first time you connect, dbt MCP opens a browser window to complete OAuth. After signing in, your session is saved and future connections are automatic. If authentication doesn't start, close your client and run: * macOS/Linux: `rm -f ~/.dbt/mcp.yml ~/.dbt/mcp.lock` * Windows: `Remove-Item -Force $env:USERPROFILE\.dbt\mcp.yml, $env:USERPROFILE\.dbt\mcp.lock` Then restart your client. ##### Tokens No additional authentication step is needed — your token is already in the configuration. The server connects automatically when your MCP client starts. #### Step 3: Test your setup Ask your AI assistant a data-related question (for example, *"What models are in my dbt project?"* or *"What metrics are defined in my Semantic Layer?"*). If dbt MCP is working, the response will use your dbt metadata. #### What's available With the platform setup, your AI assistant can use: * Semantic Layer queries * Metadata Discovery (model lineage, test results, source freshness) * Admin API (trigger jobs, list runs, get artifacts) * SQL execution and text-to-SQL (requires a [PAT](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md)) * All dbt commands if you added `DBT_PROJECT_DIR` and `DBT_PATH` For the complete tool list, see [Available tools](https://docs.getdbt.com/docs/dbt-ai/mcp-available-tools.md). Looking for self-hosted dbt CLI only? If you only need to run dbt commands locally (with or without a dbt platform account), see [Run self-hosted dbt](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-cli.md). #### Troubleshooting  Can't find the uvx executable  Can't find the uvx executable **Symptoms:** Error messages like `Could not connect to MCP server dbt-mcp`, `Error: spawn uvx ENOENT`, or `spawn uvx ENOENT` in your MCP client. **Cause:** Your MCP client (like Claude desktop) can't find `uvx` in its PATH because it starts with a limited environment. **Solution:** Use the full path to `uvx` in your configuration. 1. Find the full path: * macOS/Linux: Run `which uvx` in Terminal. * Windows: Run `where uvx` in Command Prompt or PowerShell. 2. Replace `"command": "uvx"` with the full path: ```json { "mcpServers": { "dbt": { "command": "/full/path/to/uvx", "args": ["dbt-mcp"], "env": { } } } } ``` Example on macOS with Homebrew: `"command": "/opt/homebrew/bin/uvx"` For VS Code (`mcp.json`), the same fix applies — replace `uvx` with its full path in the `command` field.  OAuth login not initiating  OAuth login not initiating **Symptoms:** The OAuth browser window never opens, or authentication appears to hang. **Cause:** dbt MCP uses a lock file to avoid repeated authentication. If a previous session left the lock file in place, it can block new authentication attempts. **Solution:** 1. Close your MCP client (Claude Desktop, Cursor, VS Code, etc.). 2. Delete the self-hosted dbt MCP config files: * macOS/Linux: `rm -f ~/.dbt/mcp.yml ~/.dbt/mcp.lock` * Windows: `Remove-Item -Force $env:USERPROFILE\.dbt\mcp.yml, $env:USERPROFILE\.dbt\mcp.lock` 3. Restart your client and try connecting again. If these steps don't resolve the issue, confirm that AI features are enabled on your account. An account admin can enable them in **Account settings** → **Edit** → toggle on **Enable account access to dbt Wizard features**. Refer to [Enable dbt Wizard](https://docs.getdbt.com/docs/platform/manage-dbt-ai.md).  Server not starting  Server not starting **Symptoms:** The MCP server shows as disconnected or unavailable in your client. **Diagnosis:** Check the server logs: * **VS Code:** Open the Command Palette (`Ctrl/Cmd + Shift + P`) → `MCP: List Servers` → click the dbt server to see detailed logs. * **Claude Desktop:** Check `~/Library/Logs/Claude` (macOS) or `%APPDATA%\Claude\logs` (Windows). * **All clients:** Set `DBT_MCP_LOG_LEVEL=DEBUG` in your environment variables to get more verbose output. **Common causes:** * Missing or incorrect `DBT_PROJECT_DIR` or `DBT_PATH` — verify the paths exist and are absolute paths. * Invalid or expired authentication tokens — generate a new token and update your config. * Missing required environment variables for the toolset you're trying to use — see [Tool requirements](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#tool-requirements-at-a-glance).  execute\_sql tool not working  execute\_sql tool not working **Symptoms:** The `execute_sql` tool returns an authentication error or is unavailable. **Cause:** How you fix this depends on how you connected: * **Remote MCP with OAuth:** Sign in through your MCP client when prompted. You don't need a personal access token. * **Remote MCP with token-based auth:** You need a [personal access token (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md). Service tokens won't work. You also need `x-dbt-dev-environment-id` and `x-dbt-user-id` headers. The user ID must match the user who created the PAT. * **Local MCP:** Set `DBT_TOKEN` to a personal access token (PAT). Service tokens won't work. **Solution:** 1. If you use OAuth for remote MCP, confirm you completed sign-in and consent in your MCP client. 2. If you use token-based auth or local MCP, create a [personal access token (PAT)](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) in **Account settings** → **API tokens** → **Personal tokens**. 3. For token-based remote MCP, add the PAT to the `Authorization` header as `Token YOUR_PAT`. For local MCP, set the PAT as your `DBT_TOKEN` value. 4. Also ensure `DBT_DEV_ENV_ID` and `DBT_USER_ID` are set (local MCP) or `x-dbt-dev-environment-id` and `x-dbt-user-id` headers are set (remote MCP). Refer to [Finding your IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md) for details.  Toolset unavailable or showing as disabled  Toolset unavailable or showing as disabled **Symptoms:** A toolset (Semantic Layer, Discovery, Admin API) is not available in your AI client even though you've configured credentials. **Cause:** Either the required variables are missing, or the toolset has been explicitly disabled. **Solution:** 1. Check that all required variables for the toolset are set — see [Tool requirements](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#tool-requirements-at-a-glance). 2. Check whether you have any `DISABLE_*` variables set to `true` that might be turning off the toolset. 3. If you're using enable mode (`DBT_MCP_ENABLE_*`), make sure the toolset you need is listed. 4. Set `DBT_MCP_LOG_LEVEL=DEBUG` to see which toolsets are active at startup.  Pasting full URLs instead of IDs  Pasting full URLs instead of IDs **Symptoms:** Authentication errors, unexpected behavior, or the server failing to connect to the right environment. **Cause:** Environment variables like `DBT_PROD_ENV_ID`, `DBT_USER_ID`, and `DBT_ACCOUNT_ID` expect numeric integers, not full browser URLs. **Solution:** ```bash # ✅ Correct DBT_HOST=cloud.getdbt.com # https://cloud.getdbt.com also works DBT_PROD_ENV_ID=54321 DBT_USER_ID=123 # ❌ Wrong — IDs must be numeric, not full URLs DBT_PROD_ENV_ID=https://cloud.getdbt.com/deploy/12345/projects/67890/environments/54321 DBT_USER_ID=https://cloud.getdbt.com/settings/profile ``` See [Finding your IDs](https://docs.getdbt.com/docs/dbt-ai/mcp-find-ids.md) for step-by-step instructions.  Multi-cell account connection issues  Multi-cell or static subdomain account connection issues **Symptoms:** Connection errors when your account URL includes a prefix (for example, `abc123.us1.dbt.com`). **Solution (as of v1.14.0):** Set `DBT_HOST` to the full hostname including the prefix. If you're using PAT-based auth, also set `DBT_ACCOUNT_ID`. ```bash # ✅ Correct DBT_HOST=abc123.us1.dbt.com DBT_ACCOUNT_ID=12345 # required for PAT-based auth ``` You no longer need to set `MULTICELL_ACCOUNT_PREFIX` or `DBT_HOST_PREFIX`. If you have these set from an older configuration, remove them. For all troubleshooting topics, see [MCP troubleshooting](https://docs.getdbt.com/docs/dbt-ai/mcp-troubleshooting.md). #### Next steps * Run dbt commands locally: see [Run self-hosted dbt](https://docs.getdbt.com/docs/dbt-ai/mcp-quickstart-cli.md) * Configure specific toolsets: see the [Environment variables reference](https://docs.getdbt.com/docs/dbt-ai/mcp-environment-variables.md) * Understand toolset requirements: see [Set up self-hosted MCP](https://docs.getdbt.com/docs/dbt-ai/setup-local-mcp.md#tool-requirements-at-a-glance) --- ### Contribute a dbt v2 adapter [Back to guides](https://docs.getdbt.com/guides.md) Adapter creation Advanced [Menu ]() info This guide is for dbt v2. For dbt v1, see [Build, test, document, and promote adapters](https://docs.getdbt.com/guides/adapter-creation.md). #### Step 1: Introduction dbt v2 adapters work very differently from v1, and it's worth understanding why before diving in. In dbt v1, every adapter was a **standalone Python package**, independently maintained by the community. That model didn't scale: maintenance cost grew with every new warehouse added, bugs had to be fixed separately in each package, and shared improvements rarely made it back across the ecosystem. dbt v2 flips this entirely. Adapters now live **inside a single Rust monorepo**, organized by feature area rather than by warehouse. A fix in the auth module benefits every adapter at once. When you contribute a new adapter, you're extending shared code, not building a whole new package from scratch. In Rust terms: adding arms to *existing* match expressions. The more adapters that exist, the easier each new adapter becomes. Your contribution makes the whole ecosystem better. Another advantage of dbt v2: connection management lives in its own lane outside of the adapter. In v1, each adapter had to own its connection logic, wrapping vendor SDKs or implementing the Python DB API spec. **In v2, ADBC drivers handle that responsibility**: pre-compiled binaries that you register but don't write. For a full explanation of ADBC and how dbt v2 uses it, see [ADBC in dbt v2](https://docs.getdbt.com/docs/dbt/adbc.md?version=2.0). What you need to contribute is the exact logic that varies by warehouse: credentials, relation naming, macros, and catalog queries. A complete community adapter touches ~13 files; that's the bar we're aiming for. This guide walks you through the architecture, crate structure, and how to contribute a dbt v2 adapter step by step. I’m Hope Watson, product manager for dbt v2 adapters at dbt Labs, and I put this together to help design, document, and shepherd this process for the community in the v2 world. The Exasol adapter, contributed by [Marco Nätlitz](https://www.linkedin.com/in/marco-naetlitz/), is used as the reference example throughout. The full file breakdown in the [reference section](#reference-file-by-file-implementation-guide) defines exactly what you're building. ##### How v1 vs. v2 adapters differ | Topic | dbt v1 | dbt v2 | | ------------------------- | ---------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | | **Package structure** | Separate Python packages (e.g. `dbt-snowflake`, `dbt-bigquery`), each inheriting from `dbt-adapters` | Part of the `dbt-labs/dbt` monorepo, contributed as PRs | | **Who maintains them** | dbt Labs maintains base adapter and 6 major adapters; others community-maintained independently | dbt Labs maintains the monorepo; community contributes via PRs | | **Language** | Python | Rust (with Jinja SQL macros still used for SQL logic) | | **How they load** | Dynamically loaded at runtime via Python's plugin system | Compiled into the monorepo binary | | **Connection management** | Each adapter implements its own connection logic, built on vendor Python SDK connectors/drivers | ADBC drivers handle connection management: a unified driver interface that abstracts away wire protocol details | | **Code organization** | Organized by warehouse, each adapter is its own isolated codebase | Organized by feature area (vertical): a fix in one crate benefits all adapters | ##### What you're building As a community contributor, you're building the foundation: connect your warehouse, run dbt macros, and support basic materializations. Once your adapter is in place, `dbt build` and `dbt run` should work. The ~13 files in the [reference section](#reference-file-by-file-implementation-guide) define the complete scope. Think of your adapter contribution as the bridge between dbt v2 and your warehouse: everything dbt needs to talk to it, nothing more. The architecture diagram below shows where your work fits in the stack. [![dbt v2 adapter architecture diagram showing the community contribution zone within the monorepo, the ADBC C ABI boundary, the ADBC driver, and the warehouse](/img/adapter-guide/adapter-creation-v2-architecture.svg?v=2 "dbt v2 adapter architecture diagram showing the community contribution zone within the monorepo, the ADBC C ABI boundary, the ADBC driver, and the warehouse")](#)dbt v2 adapter architecture diagram showing the community contribution zone within the monorepo, the ADBC C ABI boundary, the ADBC driver, and the warehouse #### Step 2: Prerequisites **Rust familiarity**: You don't need to be a Rust expert. You need to read Rust, understand enum match expressions, and make sense of compiler errors. If Rust is new to you, skim [The Rust Book](https://doc.rust-lang.org/book/) Chapter 6 (enums and pattern matching). Those are the concepts you'll encounter most. **dbt fundamentals**: Understand how [`profiles.yml`](https://docs.getdbt.com/docs/local/profiles.yml.md) works, what [materializations](https://docs.getdbt.com/docs/build/materializations.md) are, and what [adapter dispatch](https://docs.getdbt.com/reference/dbt-jinja-functions/dispatch.md) macros do. **Deep knowledge of your warehouse**: As you work through this guide, you'll need to know how your warehouse behaves: * What character does it use for quoting identifiers? * Does it support two-part or three-part relation names? * What connection parameters does it require? * How does it handle transactions? * What datatypes map to standard SQL types like string, timestamp, boolean? * What's the most efficient way to list columns in a schema? **A note on AI-assisted development**: dbt v2's architecture is naturally legible to AI coding tools. Match arms are explicit, compiler errors are precise, and the scope of each file is narrow. You don't need to use an LLM, but if you do, the structure works in your favor. ##### Your ADBC driver *Do you have an ADBC driver for your warehouse? If no, stop here.* The ADBC driver is a firm requirement before creating a dbt v2 adapter. Check these sources to find out if one exists for your warehouse: * [apache/arrow-adbc](https://github.com/apache/arrow-adbc): upstream community repo * [dbt-labs/arrow-adbc](https://github.com/dbt-labs/arrow-adbc): the dbt Labs fork * [ADBC driver foundry](https://github.com/adbc-drivers): primary community home for ADBC drivers, with active drivers and Rust/Go frameworks for building new ones * Your warehouse vendor's own GitHub org If no driver exists yet, building one is a separate project that comes before the adapter contribution. This is outside the scope of dbt Labs. [Columnar](https://columnar.tech/) specializes in building ADBC drivers and may be a useful resource. Long-term: signed driver distribution dbt Labs is working toward a model where community drivers can be distributed and loaded safely via signed binaries, similar to how operating systems verify installers. The goal is to make third-party driver installation as seamless and trustworthy as first-party. This isn't in place yet, but dbt Labs is considering this direction. #### Step 3: Understanding the codebase shape before you build ##### The vertical model dbt v2 organizes adapter logic by *capability*, not by warehouse. There's no `SnowflakeAdapter` class, no `BigQueryAdapter` class. Instead, a single shared codebase handles all warehouses, with warehouse-specific behavior driven by `match adapter_type()` expressions. When you add a new warehouse, you're adding arms to those expressions across a set of feature areas, auth, relations, catalog introspection, Jinja macros, rather than creating a new top-level package. The structural diagram below shows how the layers fit together. Your work lives in the **ADAPTER** layer: the crates that handle warehouse identity, credential resolution, relation logic, and Jinja dispatch. XDBC (driver loading and connection pooling) sits just below and has a small registration step too. [![dbt v2 system layers diagram showing USER, CLI, PIPELINE, TASK RUNNER, ADAPTER (highlighted as the contribution zone), XDBC, and WAREHOUSE layers](/img/adapter-guide/adapter-creation-v2-structural.svg?v=2 "dbt v2 system layers diagram showing USER, CLI, PIPELINE, TASK RUNNER, ADAPTER (highlighted as the contribution zone), XDBC, and WAREHOUSE layers")](#)dbt v2 system layers diagram showing USER, CLI, PIPELINE, TASK RUNNER, ADAPTER (highlighted as the contribution zone), XDBC, and WAREHOUSE layers ##### Crate map In Rust, a **crate** is a package: the unit of compilation, roughly equivalent to a "library" or "module" in other languages. The `dbt-labs/dbt` repo is a monorepo of multiple crates, each responsible for one vertical slice of functionality across all warehouses. This is a quick-reference map of the six crates you'll touch to build your adapter, in the order you'll work through them in Step 5. | Crate | Location | What you touch | | ------------------ | -------------------------- | ------------------------------------------------------------------------------------------ | | `dbt-adapter-core` | `crates/dbt-adapter-core/` | `AdapterType` enum variant, `quote_char` arm | | `dbt-xdbc` | `crates/dbt-xdbc/` | ADBC driver registration (`Backend` enum, library name) | | `dbt-schemas` | `crates/dbt-schemas/` | `profiles.yml` config struct (`DbConfig` variant) | | `dbt-auth` | `crates/dbt-auth/` | Credential resolution, connection URI construction | | `dbt-adapter` | `crates/dbt-adapter/` | Relation quoting, metadata catalog queries, adapter match arms, column builder, sql\_types | | `dbt-loader` | `crates/dbt-loader/` | Jinja SQL macros (`dbt_macro_assets/dbt-<adapter>/`) | ##### Where your adapter fits in the execution model By the time your adapter is invoked, all the dbt work is already done. dbt has resolved `ref()` and `source()` calls, rendered every Jinja template, and produced dialect-specific SQL. Your adapter receives a finished string and executes it. The adapter never sees un-rendered SQL dbt resolves `ref()`, renders Jinja, and hands your adapter a finished string. You execute it; you don't template it. While you're coding, the compiler enforces completeness. When you add `AdapterType::MyWarehouse` to the enum, every `match adapter_type()` block that doesn't handle it fails at that exact line. Your to-do list is always visible. For example: ```rust // You add your variant to the enum in dbt-adapter-core: pub enum AdapterType { Snowflake, BigQuery, DuckDb, Trino, MyWarehouse, // ← you added this } // Now every match block in the codebase must handle it. // This one in adapter_impl.rs doesn't yet: match self.adapter_type() { AdapterType::Snowflake => { ... } AdapterType::BigQuery => { ... } AdapterType::DuckDb => { ... } AdapterType::Trino => { ... } // missing MyWarehouse → compile error } ``` ```text error[E0004]: non-exhaustive patterns: `AdapterType::MyWarehouse` not covered --> crates/dbt-adapter/src/adapter/adapter_impl.rs:142:18 | | match self.adapter_type() { | ^^^^^^^^^^^^^^^^^^^ pattern `AdapterType::MyWarehouse` not covered ``` Each failure is a concrete implementation task. ##### Are you porting an existing v1 adapter? No v1 adapter? Skip to [Step 4](#step-4-development-machine-setup). If you're porting, use this map to see exactly what transfers, what needs rethinking, and what dbt Labs now owns for you in v2. [![Diagram showing which v1 adapter components map directly to v2 (green), require rethinking (purple), or are absorbed by dbt Labs or the ADBC driver (red)](/img/adapter-guide/adapter-creation-v2-blocks.svg?v=2 "Diagram showing which v1 adapter components map directly to v2 (green), require rethinking (purple), or are absorbed by dbt Labs or the ADBC driver (red)")](#)Diagram showing which v1 adapter components map directly to v2 (green), require rethinking (purple), or are absorbed by dbt Labs or the ADBC driver (red) Most community contributors aren't building from scratch, they're porting an adapter they already maintain or use in Python for dbt v1. If that's you, **you have a significant head start**, and this is the most realistic path for the vast majority of contributors. Before writing any Rust, check a couple of things: 1. Does a v1 adapter already exist for your warehouse? Check the [trusted adapters](https://docs.getdbt.com/docs/trusted-adapters.md) and [community adapters](https://docs.getdbt.com/docs/community-adapters.md) lists. If one exists, find its GitHub repo: the macro SQL and connection logic are almost directly reusable. 2. Did dbt Labs already add a placeholder for your warehouse? Some warehouses already appear in v2's `AdapterType` enum but aren't fully implemented yet: the enum variant exists, which means the boilerplate is partially in place. When you add the remaining code, the compiler shows you exactly what's still missing. The warehouses in this state are: | Warehouse | v1 adapter | dbt v2 status | | --------- | ----------------------------------------------------------------- | ----------------------------------------------------------------- | | Athena | [dbt-athena](https://github.com/dbt-athena/dbt-athena) (trusted) | `AdapterType::Athena` exists: needs auth, macros, adapter arms | | Trino | [dbt-trino](https://github.com/starburstdata/dbt-trino) (trusted) | `AdapterType::Trino` exists: needs auth, macros, adapter arms | | Starburst | [dbt-trino](https://github.com/starburstdata/dbt-trino) (trusted) | `AdapterType::Starburst` exists: needs auth, macros, adapter arms | | Dremio | [dbt-dremio](https://github.com/dremio/dbt-dremio) (trusted) | `AdapterType::Dremio` exists: needs auth, macros, adapter arms | | Oracle | [dbt-oracle](https://github.com/oracle/dbt-oracle) (trusted) | `AdapterType::Oracle` exists: needs auth, macros, adapter arms | For warehouses not yet in `AdapterType` at all (MySQL, Hive, Vertica, SQL Server, Teradata, etc.), you start from Step 5.1 by adding the `AdapterType` variant. ###### What transfers from v1 to v2 | v1 component | Where it goes in v2 | Notes | | ----------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | | `macros/adapters.sql` | `dbt-loader/.../dbt-<wh>/macros/adapters.sql` | Mostly a direct port, same `<wh>__` dispatch prefix, same macro names, same Jinja patterns | | `macros/catalog.sql` | Same location | The catalog SQL (`list_relations_without_caching`, `get_catalog`) transfers almost verbatim | | Custom materializations and adapter-overrides | Same location | Look out for Jinja that might not yet be supported in dbt v2; that will need to be addressed separately | | Profile fields in `credentials.py` / `profile_template.yml` | `DbConfig` struct in `dbt-schemas` | Each profile field becomes a struct field: optional fields use `Option<T>` | | Connection URI / DSN construction in `connections.py` | `dbt-auth/src/<wh>/mod.rs` | The URI building logic maps cleanly to the auth module pattern | | `BaseRelation.quote_policy` / identifier casing behavior | `include_policy()` in `crates/dbt-adapter/src/relation/relation_impl.rs`, `Policy` is a type alias for `ResolvedQuoting` defined in `crates/dbt-schemas/src/schemas/relations/base.rs` | The 3-part vs. 2-part name structure and quote flags map 1:1 to the dbt v2 `Policy` struct | | Catalog introspection SQL in macros and `adapter.py` | `get_relation.rs` and Jinja macros | The system catalog table names and queries you already know transfer directly | ###### What doesn't transfer by design Four categories of v1 code simply don't exist in v2, which means less code to port: * **Connection management**: In v1 you owned the full connection lifecycle against the Python DB API. In v2 the ADBC driver handles this entirely; skip anything related to cursors, retries, or connection pooling. * **Query execution**: Shared Rust infrastructure handles raw query plumbing (`execute`, `add_query`). You don't implement it. * **Adapter class hierarchy**: The Python subclass structure is replaced by shared Rust match expressions. You register your adapter; you don't implement execution logic. * **Packaging and distribution**: No `setup.py`, no PyPI release. dbt Labs ships your adapter once the PR is merged. #### Step 4: Development machine setup ```shell # Rust rustup # install from https://rustup.rs rustup show # verify # Go (needed for driver builds) go version # verify # Clone the repo git clone https://github.com/dbt-labs/dbt cd dbt # Verify you can build cargo build --bin dbt ``` If you hit Z3 errors: ```shell brew install pkg-config z3 ``` If disk fills during build: ```shell cargo clean # frees old build artifacts; you'll do this often ``` ##### Development workflow The core development loop is the same whether you write code manually or with an AI assistant: 1. Add `AdapterType::MyWarehouse` to the enum (Step 5.1) 2. Run `cargo build -p <crate>`: the compiler lists every match arm missing for your new variant 3. Use the compiler output + the file you're editing + the equivalent reference file (from the breakdown at the bottom) to write each arm. If using an AI assistant, paste all three in as context. * *Example:* to fill in the `quote_char` arm, paste the error listing `AdapterType::MyWarehouse` as missing, the `quote_char` match block from `dbt-adapter-core/src/lib.rs`, and the Exasol line (`Exasol => '"'`) as the pattern to follow. 4. Fill in the arm; verify with `cargo build` again 5. Repeat for each crate until error-free Every missing case is a compile error, so an AI assistant always has a precise specification to work from. What context helps for each arm: * **The file you're editing**: paste the relevant `match` block or function. For example, the `match self.adapter_type()` block from `adapter_impl.rs` that needs a new arm. * **The equivalent reference file** from the breakdown below. For example, when writing your auth module, paste `crates/dbt-auth/src/exasol/mod.rs` as the pattern. * **The compiler error output.** For example: `error[E0004]: non-exhaustive patterns: AdapterType::MyWarehouse not covered` * **Your warehouse's specifics**: system catalog table names and the connection fields from your `profiles.yml`. Watch out for: * **Hallucinated file paths**: AI often invents dbt v2 paths. Use the file breakdown below as ground truth. * **Always verify with the type checker**: run `cargo build -p <crate>` after any AI-generated changes. * **SQL macro patterns from v1** may not apply cleanly in dbt v2. Compare against the reference `adapters.sql` at `crates/dbt-loader/src/dbt_macro_assets/dbt-exasol/macros/adapters.sql` in [dbt-labs/dbt](https://github.com/dbt-labs/dbt). #### Step 5: Build a new adapter This step walks you through each crate you need to touch. Work through them in order. Each builds on the last. After each sub-step, run the type checker to catch missed match arms and type errors: ```shell cargo build -p <crate-name> ``` Replace `<crate-name>` with the crate you just edited, for example `dbt-adapter-core`, `dbt-xdbc`, `dbt-schemas`, `dbt-auth`, `dbt-adapter`, or `dbt-loader`. ##### 5.1: Register the adapter type **Crate:** `crates/dbt-adapter-core/` `AdapterType` is the central enum that identifies your warehouse throughout the entire codebase. Add your variant here first. Rust's `match` expressions must handle every possible variant explicitly. So when you add `AdapterType::MyWarehouse`, every `match adapter_type()` block that doesn't handle your variant becomes a **compile error**. Run `cargo build -p <crate>` and the compiler hands you a complete list of exactly what still needs to be written. Nothing is hidden or implicit. ```rust // crates/dbt-adapter-core/src/lib.rs pub enum AdapterType { Postgres, Snowflake, Bigquery, Databricks, Redshift, Salesforce, Spark, DuckDB, Fabric, ClickHouse, Athena, Starburst, Trino, Datafusion, Dremio, Oracle, Exasol, // ← already added MyWarehouse, // ← add yours here } ``` Also add a `quote_char` arm in the same file. Use double quotes `'"'` for most warehouses; BigQuery and Databricks use a backtick instead: ```rust // crates/dbt-adapter-core/src/lib.rs fn quote_char(&self) -> char { match self { // ... existing adapters ... Exasol => '"', // double-quote (most warehouses) MyWarehouse => '"', // ← add yours here BigQuery | Databricks => '`', } } ``` ##### 5.2: Register the ADBC driver **Crate:** `crates/dbt-xdbc/` Your driver already exists (covered in Step 2). This step is where you register it so dbt v2 knows its library name and how to load it at runtime. | File | What to do | Required? | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | | `src/driver.rs` | Add a variant to the `Backend` enum; add the ADBC library name (e.g. `"adbc_driver_exasol"`) and FFI protocol; also define a `LoadStrategy` | **Yes** | | `src/install.rs` | Add CDN download URL and platform strings | No, only for drivers distributed via the dbt Labs CDN, which requires separate coordination with dbt Labs | The `Backend` enum maps to the ADBC shared library name (`lib<name>.so` / `<name>.dll` / `lib<name>.dylib`). You're registering its identity so dbt v2 knows what to load. You're not writing the driver here. ###### CDN-distributed vs. manual install For these 11 adapters, dbt v2 automatically downloads the driver on first use. All other adapters require manual installation. Users take one extra setup step. Make this clear in your documentation (Step 7). | On the CDN: auto-downloaded | | ----------------------------------------------------------------------------------------------------------------------- | | Snowflake, BigQuery, ClickHouse, Postgres, Databricks, Redshift, DuckDB, DuckDB Extended, Salesforce, Spark, SQL Server | For **custom Arrow type mappings** (for example, DuckDB needed this for `HUGEINT`, `UTINYINT`): only add warehouse-specific type handling if your driver returns types that Arrow's standard schema doesn't cover. Most warehouses don't need this. ##### 5.3: Add your connection profile **Crate:** `crates/dbt-schemas/` | File | What to do | | ------------------------- | ----------------------------------------------------------------------- | | `src/schemas/profiles.rs` | Add a `DbConfig` variant and a config struct with all connection fields | The config struct should include everything a user would put in their `profiles.yml` for your warehouse. Example from Exasol: ```rust #[derive(Debug, Clone, Serialize, Deserialize, Default, DbtSchema, Merge)] pub struct ExasolDbConfig { pub user: Option<String>, #[serde(skip_serializing_if = "Option::is_none", alias = "pass")] pub password: Option<String>, pub host: Option<String>, pub port: Option<StringOrInteger>, pub database: Option<String>, pub schema: Option<String>, pub encryption: Option<bool>, pub certificate_validation: Option<bool>, pub certificate_fingerprint: Option<String>, pub connection_timeout: Option<StringOrInteger>, pub threads: Option<StringOrInteger>, } ``` After adding the config struct, also add `DbConfig::MyWarehouse(Box<MyWarehouseDbConfig>)` as a new variant to the `DbConfig` enum. The compiler will then point you at every place that reads from `DbConfig` and needs a new case for your warehouse. Follow those errors to wire it in. ##### 5.3.1: Add `dbt init` profile generation (optional but strongly encouraged) This is what makes your adapter easy to use. When someone runs `dbt init` and selects your warehouse, they get a guided prompt that builds their `profiles.yml`, without documentation hunting or manual YAML editing. Add support in `crates/dbt-init/`: 1. Register your adapter in `get_available_adapters()` in `crates/dbt-init/src/profile_setup.rs`: ```rust pub fn get_available_adapters() -> &'static [AdapterType] { &[ // ... existing adapters ... AdapterType::MyWarehouse, // ← add this ] } ``` 2. Create a config file `crates/dbt-init/src/adapter_config/<warehouse>_config.rs` implementing `InteractiveSetup` for your `DbConfig` struct: define the fields `dbt init` should prompt for and a `set_field()` handler. See `postgres_config.rs` as a minimal reference. 3. Export it from `crates/dbt-init/src/adapter_config/mod.rs`: ```rust pub mod mywarehouse_config; pub use mywarehouse_config::setup_mywarehouse_profile; ``` 4. Add a match arm in `create_profile_for_adapter()` in `profile_setup.rs`: ```rust AdapterType::MyWarehouse => { let config = match existing_config { Some(DbConfig::MyWarehouse(c)) => Some(c), _ => None, }; DbConfig::MyWarehouse(setup_mywarehouse_profile(config.map(Box::as_ref))?) } ``` ##### 5.4: Add authentication **Crate:** `crates/dbt-auth/` | File | What to do | Required? | | ------------------------- | ------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | `src/<warehouse>/mod.rs` | Credential resolution, reads config fields, env vars, key files, tokens; builds the ADBC connection URI and credentials | **Yes** | | `src/<warehouse>/init.rs` | Init SQL generation, SQL that must run when the connection opens (e.g. `USE SCHEMA`, `SET` statements, extension loading) | Optional, only for warehouses that need SQL on connection open. Most don't. | | `src/lib.rs` | Register the new module with `mod <warehouse>;` and wire it into the auth dispatch match | **Yes** | The auth module turns a `DbConfig` into a live, authenticated ADBC connection. At minimum you need basic credential handling. More sophisticated auth (OAuth, SSO, key-pair) can be added incrementally. The pattern is: read config fields → construct URI → call `builder.with_parse_uri(uri)`, `builder.with_username(user)`, `builder.with_password(password)`. ##### 5.5: Build the adapter layer **Crate:** `crates/dbt-adapter/` This is the largest step. You're adding warehouse-specific behavior to the shared adapter layer via `match adapter_type()` arms. Simple vs. complex adapters Most adapters, including Exasol, Athena, Trino, Starburst, Dremio, Oracle, and ClickHouse, add match arms directly to the shared files in `src/relation/`. Only adapters with highly custom relation logic (Snowflake's multi-part names and case rules; BigQuery's project/dataset structure) have their own subdirectory under `src/relation/`. If your warehouse uses standard `schema.table` or `database.schema.table` naming with straightforward quoting, you're a simple adapter. Start simple and only add complexity if the compiler forces it. ###### Relation type and quoting **File:** `crates/dbt-adapter/src/relation/relation_impl.rs` Add a match arm that sets your quoting policy. The policy controls three things: whether the database is included in fully-qualified names, whether the schema is quoted, and whether the identifier is quoted. For example, Exasol uses a 2-part name (`schema.table`, no database prefix) and quotes both: ```rust // crates/dbt-adapter/src/relation/relation_impl.rs fn include_policy(adapter_type: AdapterType, path: &RelationPath) -> Policy { match adapter_type { // ... existing adapters ... AdapterType::MyWarehouse => Policy::new(false, true, true), // ↑ ↑ ↑ // database schema identifier // disabled quoted quoted _ => Policy::trues(), // default: all parts included and quoted } } ``` Decide up front whether your warehouse uses 2-part or 3-part names, and whether identifiers are case-sensitive. For example, Exasol uppercases unquoted identifiers by default, so all catalog lookup SQL uses `upper()` comparisons. **File:** `src/relation/factory.rs` Add your `AdapterType` to the `create_static_relation` match, wiring it to `RelationStatic` (the generic static relation used for Jinja's `api.Relation`): ```rust Databricks | Spark | Fabric | DuckDB | Exasol | Postgres | Redshift | Salesforce | Bigquery => { let relation_type = RelationStatic { adapter_type, quoting }; StaticBaseRelationObject::new(Arc::new(relation_type)) } ``` ###### Catalog introspection **File:** `src/metadata/get_relation.rs` Add a match arm and a function that queries your warehouse's system catalog to look up a single relation by name. Follow the reference implementations closely. This area of the codebase is expected to evolve: ```rust AdapterType::Exasol => exasol_get_relation( adapter, state, ctx, conn, database, schema, identifier, token, ), ``` ```rust fn exasol_get_relation( adapter: &AdapterImpl, state: &State, ctx: &QueryCtx, conn: &mut dyn Connection, database: &str, schema: &str, identifier: &str, token: CancellationToken, ) -> AdapterResult<Option<Box<dyn BaseRelation>>> { let q_schema = schema.to_uppercase(); let q_ident = identifier.to_uppercase(); let sql = format!( "select 'table' as \"type\" from sys.exa_all_tables \ where table_schema = '{q_schema}' and table_name = '{q_ident}' \ union all \ select 'view' from sys.exa_all_views \ where view_schema = '{q_schema}' and view_name = '{q_ident}'" ); // execute, read result, return relation } ``` Use `information_schema` if your warehouse supports standard SQL, or system catalog tables (`sys.*`, `information_schema.*`) as appropriate. ###### Adapter match arms **File:** `src/adapter/adapter_impl.rs` After adding `AdapterType::MyWarehouse`, the compiler will enumerate every `match self.adapter_type()` block that needs a new arm. Most arms simply delegate to an existing pattern. For example: ```rust // Column builder: delegate to postgres-like builder Exasol => Ok(Self::build_postgres_like(field, type_ops)), // Schema column name for listing relations Exasol => "name", // DATA_TYPE column name in information schema AdapterType::Exasol => "DATA_TYPE", // in src/sql_types.rs ``` For capabilities your adapter doesn't support yet (like `valid_incremental_strategies`), return `unimplemented!()`. That's fine for an initial community adapter contribution. ###### Column builder **File:** `src/column/column_builder.rs` Add a match arm for how your warehouse's Arrow record batches map to dbt column objects. Most adapters can delegate to `build_postgres_like`: ```rust Exasol => Ok(Self::build_postgres_like(field, type_ops)), ``` Only implement custom logic if your warehouse has unusual type handling. ##### 5.6: Write your SQL macros **Crate:** `crates/dbt-loader/` Create a new directory at `src/dbt_macro_assets/dbt-<adapter_type>/`. Registration is automatic The loader discovers adapter packages by scanning `src/dbt_macro_assets/`. You don't need to add any Rust code to register your directory. Creating it and the `dbt_project.yml` is sufficient. | File | What to do | | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `dbt_macro_assets/dbt-<warehouse>/dbt_project.yml` | Adapter plugin project definition: `name: dbt_<warehouse>`, `macro-paths: ["macros"]` | | `macros/adapters.sql` | dbt v1 adapter macros: `create_schema`, `drop_schema`, `drop_relation`, `rename_relation`, `truncate_relation`, `create_table_as`, `create_view_as`, `list_schemas`, `check_schema_exists`, `information_schema_name`, `current_timestamp`, `get_columns_in_relation`, `list_relations_without_caching` | v2 uses the same `adapter.dispatch()` pattern as v1. Your macros use the `<warehouse>__` prefix to override defaults: ```sql {% macro exasol__create_table_as(temporary, relation, sql) -%} create or replace table {{ relation }} as ( {{ sql }} ) {%- endmacro %} {% macro exasol__create_view_as(relation, sql) -%} create or replace view {{ relation }} as ( {{ sql }} ) {%- endmacro %} {% macro exasol__drop_relation(relation) -%} {% call statement('drop_relation', auto_begin=False) -%} drop {{ relation.type }} if exists {{ relation }} cascade {%- endcall %} {% endmacro %} {% macro exasol__rename_relation(from_relation, to_relation) -%} {% call statement('rename_relation') -%} rename {{ from_relation.type }} {{ from_relation }} to {{ to_relation.identifier }} {%- endcall %} {% endmacro %} ``` Note: `rename_relation` uses only `to_relation.identifier`, not the full relation. Exasol's `RENAME` syntax doesn't take a fully qualified target. For catalog introspection, use your warehouse's system catalog. For example, Exasol queries `sys.*` instead of `information_schema`: ```sql {% macro exasol__list_relations_without_caching(schema_relation) -%} {% call statement('list_relations_without_caching', fetch_result=True) -%} select '{{ schema_relation.database }}' as "database", table_name as "name", table_schema as "schema", 'table' as "type" from sys.exa_all_tables where upper(table_schema) = upper('{{ schema_relation.schema }}') union all select '{{ schema_relation.database }}' as "database", view_name as "name", view_schema as "schema", 'view' as "type" from sys.exa_all_views where upper(view_schema) = upper('{{ schema_relation.schema }}') {%- endcall %} {{ return(load_result('list_relations_without_caching').table) }} {%- endmacro %} {% macro exasol__information_schema_name(database) -%} sys {%- endmacro %} ``` If your warehouse is similar to an existing one (e.g. Postgres-compatible), start by delegating to that dialect's macros and only override where behavior differs: ```sql {% macro mywarehouse__create_table_as(temporary, relation, sql) -%} {{ return(postgres__create_table_as(temporary, relation, sql)) }} {%- endmacro %} ``` #### Step 6: Test your adapter ##### Type check after each crate Run the type checker after completing work in each crate to catch missed match arms and type errors immediately: ```shell cargo build -p dbt-adapter-core cargo build -p dbt-xdbc cargo build -p dbt-schemas cargo build -p dbt-auth cargo build -p dbt-adapter cargo build -p dbt-loader ``` ##### Smoke testing with a dbt project Run a real `dbt build` against your warehouse and ensure it builds error-free. At minimum, exercise table, view, incremental, and seed materializations. A clean `dbt build` on [jaffle-shop](https://github.com/dbt-labs/jaffle-shop) is one example of an acceptance bar for a community adapter. ```shell # Build the CLI cargo build --bin dbt # Create a test project (or clone jaffle-shop) ./target/debug/dbt init # Run against your warehouse ./target/debug/dbt build --project-dir <your-project> ``` ##### CI testing CI testing for community adapter PRs is coordinated with the dbt Labs adapters team: the test infrastructure is not publicly distributed. When your PR is ready, reach out in `#adapter-ecosystem` on the [dbt Community Slack](https://community.getdbt.com/), tag Hope Watson (`@Hope Watson (dbt Labs)`), and the adapters team will work with you on warehouse validation. Note that this may take some time and coordination. Known gap: CI requires coordination Community contributors cannot run CI independently. dbt Labs' CI pipeline requires certain checks. For certain adapters, that means warehouse credentials. dbt Labs is still defining this handoff process for dbt v2. Expect to coordinate closely with the adapters team, for the approval process to take some time, and for some back-and-forth. If you hit friction, flag it in `#adapter-ecosystem`. #### Step 7: Document your adapter Once your adapter is merged and available in a release, document it so users can find and configure it. ##### Write a setup guide Document the `profiles.yml` configuration for your warehouse: what fields are required, what's optional, and example values. Follow the format of existing [adapter setup guides](https://docs.getdbt.com/docs/local/connect-data-platform/about-dbt-connections.md?version=2.0). Driver installation is critical to document Unlike adapters distributed via the CDN, your users won't get the driver automatically: dbt v2 won't download it for them. Your setup guide must explain where to get the driver binary and how to install it so dbt v2 can find it at runtime. Without this, users will configure a valid profile and still get a connection error. Include the exact library name dbt v2 looks for (e.g. `libadbc_driver_<yourwarehouse>.dylib`) and where to put it. ##### General documentation guidelines * Assume the reader knows dbt fundamentals but is not an expert on your warehouse's inner workings. * Include a complete working `profiles.yml` example. * Document any warehouse-specific quirks (e.g. 2-part vs 3-part naming, identifier case sensitivity). * Link to the warehouse vendor's ADBC driver documentation. #### Step 8: Promote your adapter Your PR must be merged first dbt Labs reviews and merges community adapter PRs into `dbt-labs/dbt`. Wait until the PR is merged and the adapter ships in a published release before directing users to it. ##### Community channels Join the [dbt Community Slack](https://community.getdbt.com/) and find: * **`#adapter-ecosystem`**: the main channel for adapter developers * **`#db-<yourwarehouse>`**: if a channel exists for your warehouse, let users know v2 support is available. Note: v1 adapter users will still be on the Python-based adapter and will need to migrate. ##### Before you announce Align with the adapters team on: which materializations you're targeting in the initial implementation, any known gaps in your ADBC driver, and timeline. This prevents surprises during review and sets accurate expectations for users. #### Reference: File-by-file implementation guide A community-contributed dbt v2 adapter touches roughly 13 files, all in the public [dbt-labs/dbt](https://github.com/dbt-labs/dbt) repo. The "Exasol example" column shows what it looks like in practice. Substitute your warehouse name and system catalog throughout. | Generic path | What it does | Exasol example | | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | `.changes/unreleased/Features-*.yaml` | Changelog entry | Same for all adapters | | `crates/dbt-adapter-core/src/lib.rs` | Add `AdapterType::<Warehouse>` variant and `quote_char` arm | `AdapterType::Exasol` | | `crates/dbt-xdbc/src/driver.rs` | Add `Backend::<Warehouse>` variant: the ADBC library name and FFI protocol | `Backend::Exasol`, library name `"adbc_driver_exasol"` | | `crates/dbt-schemas/src/schemas/profiles.rs` | `<Warehouse>DbConfig` struct + `DbConfig::<Warehouse>` variant, wired into all config match arms | `ExasolDbConfig` struct + `DbConfig::Exasol` variant | | `crates/dbt-auth/src/<warehouse>/mod.rs` | Auth module, reads config fields, resolves credentials, constructs connection URI | `src/exasol/mod.rs`, builds URI from host, port, user/pass, TLS options | | `crates/dbt-adapter/src/adapter/adapter_impl.rs` | All exhaustive `match adapter_type()` arms, most delegate to existing patterns; use `unimplemented!()` for features not yet supported | Exasol adds ~10 arms, most are 1–2 lines | | `crates/dbt-adapter/src/column/column_builder.rs` | Column builder match arm, maps Arrow record batches to dbt column objects | `Exasol => Ok(Self::build_postgres_like(...))` | | `crates/dbt-adapter/src/metadata/get_relation.rs` | Catalog lookup function, queries system tables to find a relation by name | `exasol_get_relation()`, queries `sys.exa_all_tables` / `sys.exa_all_views` | | `crates/dbt-adapter/src/relation/relation_impl.rs` | Add your adapter to the `include_policy()` match, controls whether the database prefix is included and whether parts are quoted | `Policy::new(false, true, true)`, database disabled, schema + identifier quoted. Default `Policy::trues()` includes and quotes all three parts. | | `crates/dbt-adapter/src/relation/factory.rs` | Add warehouse to `RelationStatic` arm in `create_static_relation` | Exasol added alongside Postgres, Redshift, DuckDB, etc. | | `crates/dbt-adapter/src/sql_types.rs` | Metadata column name arms, `DATA_TYPE` and schema listing column names | Exasol uses standard `DATA_TYPE` column name | | `crates/dbt-loader/src/dbt_macro_assets/dbt-<warehouse>/dbt_project.yml` | Macro plugin project definition | `dbt-exasol/dbt_project.yml` | | `crates/dbt-loader/src/dbt_macro_assets/dbt-<warehouse>/macros/adapters.sql` | All required adapter macros | `dbt-exasol/macros/adapters.sql`, uses `sys.*` instead of `information_schema` | #### Reference: Useful commands ```shell # Type check a specific crate cargo build -p <crate-name> # Build the CLI binary cargo build --bin dbt # Run a full dbt build against your warehouse ./target/debug/dbt build --project-dir <your-project> # Free disk space cargo clean ``` --- ### Create Datadog events from dbt results [Back to guides](https://docs.getdbt.com/guides.md) Webhooks Advanced [Menu ]() #### Introduction This guide will teach you how to build and host a basic Python app which will add dbt job events to Datadog. To do this, when a dbt job completes it will create a log entry for each node that was run, containing all information about the node provided by the [Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-schema-job-models.md). In this example, we will use [fly.io](https://fly.io) for hosting/running the service. fly.io is a platform for running full stack apps without provisioning servers etc. This level of usage should comfortably fit inside of the Free tier. You can also use an alternative tool such as [AWS Lambda](https://ademoverflow.com/en/posts/tutorial-fastapi-aws-lambda-serverless/) or [Google Cloud Run](https://github.com/sekR4/FastAPI-on-Google-Cloud-Run). ##### Prerequisites This guide assumes some familiarity with: * [dbt Webhooks](https://docs.getdbt.com/docs/deploy/webhooks.md) * CLI apps * Deploying code to a serverless code runner like fly.io or AWS Lambda #### Clone the `dbt-cloud-webhooks-datadog` repo [This repository](https://github.com/dpguthrie/dbt-cloud-webhooks-datadog) contains the sample code for validating a webhook and creating logs in Datadog. #### Install `flyctl` and sign up for fly.io Follow the directions for your OS in the [fly.io docs](https://fly.io/docs/hands-on/install-flyctl/), then from your command line, run the following commands: Switch to the directory containing the repo you cloned in step 1: ```shell #example: replace with your actual path cd ~/Documents/GitHub/dbt-cloud-webhooks-datadog ``` Sign up for fly.io: ```shell flyctl auth signup ``` Your console should show `successfully logged in as YOUR_EMAIL` when you're done, but if it doesn't then sign in to fly.io from your command line: ```shell flyctl auth login ``` #### Launch your fly.io app Launching your app publishes it to the web and makes it ready to catch webhook events: ```shell flyctl launch ``` 1. You will see a message saying that an existing `fly.toml` file was found. Type `y` to copy its configuration to your new app. 2. Choose an app name of your choosing, such as `YOUR_COMPANY-dbt-cloud-webhook-datadog`, or leave blank and one will be generated for you. Note that your name can only contain numbers, lowercase letters and dashes. 3. Choose a deployment region, and take note of the hostname that is generated (normally `APP_NAME.fly.dev`). 4. When asked if you would like to set up Postgresql or Redis databases, type `n` for each. 5. Type `y` when asked if you would like to deploy now. Sample output from the setup wizard: \`\` `joel@Joel-Labes dbt-cloud-webhooks-datadog % flyctl launch An existing fly.toml file was found for app dbt-cloud-webhooks-datadog ? Would you like to copy its configuration to the new app? Yes Creating app in /Users/joel/Documents/GitHub/dbt-cloud-webhooks-datadog Scanning source code Detected a Dockerfile app ? Choose an app name (leave blank to generate one): demo-dbt-cloud-webhook-datadog automatically selected personal organization: Joel Labes Some regions require a paid plan (fra, maa). See https://fly.io/plans to set up a plan. ? Choose a region for deployment: [Use arrows to move, type to filter] ? Choose a region for deployment: Sydney, Australia (syd) Created app dbtlabs-dbt-cloud-webhook-datadog in organization personal Admin URL: https://fly.io/apps/demo-dbt-cloud-webhook-datadog Hostname: demo-dbt-cloud-webhook-datadog.fly.dev ? Would you like to set up a Postgresql database now? No ? Would you like to set up an Upstash Redis database now? No Wrote config file fly.toml ? Would you like to deploy now? Yes` ##### 4. Create a Datadog API Key [Create an API Key for your Datadog account](https://docs.datadoghq.com/account_management/api-app-keys/) and make note of it and your Datadog site (e.g. `datadoghq.com`) for later. #### Configure a new webhook in dbt 1. See [Create a webhook subscription](https://docs.getdbt.com/docs/deploy/webhooks.md#create-a-webhook-subscription) for full instructions. Your event should be **Run completed**. 2. Set the webhook URL to the host name you created earlier (`APP_NAME.fly.dev`). 3. Make note of the Webhook Secret Key for later. *Do not test the endpoint*; it won't work until you have stored the auth keys (next step) #### Store secrets The application requires four secrets to be set, using these names: * `DBT_CLOUD_SERVICE_TOKEN`: a dbt [personal access token](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) or [service account token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) with at least the `Metdata Only` permission. * `DBT_CLOUD_AUTH_TOKEN`: the Secret Key for the dbt webhook you created earlier. * `DD_API_KEY`: the API key you created earlier. * `DD_SITE`: The Datadog site for your organisation, e.g. `datadoghq.com`. Set these secrets as follows, replacing `abc123` etc with actual values: ```shell flyctl secrets set DBT_CLOUD_SERVICE_TOKEN=abc123 DBT_CLOUD_AUTH_TOKEN=def456 DD_API_KEY=ghi789 DD_SITE=datadoghq.com ``` #### Deploy your app After you set your secrets, fly.io will redeploy your application. When it has completed successfully, go back to the dbt webhook settings and click **Test Endpoint**. --- ### Create new materializations [Back to guides](https://docs.getdbt.com/guides.md) Advanced [Menu ]() #### Introduction The model materializations you're familiar with, `table`, `view`, and `incremental` are implemented as macros in a package that's distributed along with dbt. You can check out the [source code for these materializations](https://github.com/dbt-labs/dbt-adapters/tree/60005a0a2bd33b61cb65a591bc1604b1b3fd25d5/dbt/include/global_project/macros/materializations). If you need to create your own materializations, reading these files is a good place to start. Continue reading below for a deep-dive into dbt materializations. caution This is an advanced feature of dbt. Let us know if you need a hand! We're always happy to [chat](http://community.getdbt.com/). (Applies to dbt v2.0 and later) Custom materializations and static analysis dbt v2 automatically sets [`static_analysis: off`](https://docs.getdbt.com/reference/resource-configs/static-analysis.md#custom-materializations) for models built with a custom materialization, because the materialization can change the schema of the persisted model. That setting cascades downstream, so models downstream of a model with a custom materialization are also ineligible for static analysis. Refer to [Custom materializations](https://docs.getdbt.com/docs/build/about-static-analysis.md#custom-materializations) in the static analysis docs for details. #### Creating a materialization Learn by video! For video tutorials on Materializations, go to dbt Learn and check out the [Materializations fundamentals course](https://learn.getdbt.com/courses/materializations-fundamentals). Materialization blocks make it possible for dbt to load custom materializations from packages. The materialization blocks work very much like `macro` blocks, with a couple of key exceptions. Materializations are defined as follows: ```sql {% materialization [materialization name], ["specified adapter" | default] %} ... {% endmaterialization %} ``` Materializations can be given a name, and they can be tied to a specific adapter. dbt will pick the materialization tied to the currently-in-use adapter if one exists, or it will fall back to the `default` adapter. In practice, this looks like: macros/my\_materialization.sql ```sql {% materialization my_materialization_name, default %} -- cross-adapter materialization... assume Redshift is not supported {% endmaterialization %} {% materialization my_materialization_name, adapter='redshift' %} -- override the materialization for Redshift {% endmaterialization %} ``` info dbt's ability to dynamically pick the correct materialization based on the active database target is called [multiple dispatch](https://en.wikipedia.org/wiki/Multiple_dispatch). This feature unlocks a whole world of cross-database compatibility features -- if you're interested in this, please let us know on Slack! ##### Anatomy of a materialization Materializations are responsible for taking a dbt model SQL statement and turning it into a transformed dataset in a database. As such, materializations generally take the following shape: 1. Prepare the database for the new model 2. Run pre-hooks 3. Execute any SQL required to implement the desired materialization 4. Run post-model hooks 5. Clean up the database as required 6. Update the Relation cache Each of these tasks are explained in sections below. ##### Prepare the database Materializations are responsible for creating new tables or views in the database, or inserting/updating/deleting data from existing tables. As such, materializations need to know about the state of the database to determine exactly what SQL they should run. Here is some pseudocode for the "setup" phase of the **table** materialization: ```sql -- Refer to the table materialization (linked above) for an example of real syntax -- This code will not work and is only intended for demonstration purposes {% set existing = adapter.get_relation(this) %} {% if existing and existing.is_view %} {% do adapter.drop_relation(existing) %} {% endif %} ``` In this example, the `get_relation` method is used to fetch the state of the currently-executing model from the database. If the model exists as a view, then the view is dropped to make room for the table that will be built later in the materialization. This is a simplified example, and the setup phase for a materialization can become quite complicated indeed! When building a materialization, be sure to consider the state of the database and any supplied [flags](https://docs.getdbt.com/reference/dbt-jinja-functions/flags.md) (ie. `--full-refresh`) to ensure that the materialization code behaves correctly in different scenarios. ##### Run pre-hooks Pre- and post-hooks can be specified for any model -- be sure that your materialization plays nicely with these settings. Two variables, `pre_hooks` and `post_hooks` are automatically injected into the materialization context. Invoke these hooks at the appropriate time with: ```sql ... {{ run_hooks(pre_hooks) }} .... ``` ##### Executing SQL Construct your materialization DML to account for the different permutations of table existence, materialization flags, etc. There are a number of [adapter functions](https://docs.getdbt.com/reference/dbt-jinja-functions/adapter.md) and context variables that can help you here. Be sure to consult the Reference section of this site for a full list of variables and functions at your disposal. ##### Run post-hooks See the section above on pre-hooks for more information on running post-hooks. ##### Clean up The "cleanup" phase of the materialization typically renames or drops relations and commits the transaction opened in "preparation" step above. The `table` materialization, for instance, executes the following cleanup code: ```text {{ drop_relation_if_exists(backup_relation) }} ``` Be sure to `commit` the transaction in the `cleanup` phase of the materialization with `{{ adapter.commit() }}`. If you do not commit this transaction, it will be rolled back by dbt and the transformations applied in your materialization will be discarded. ##### Update the Relation cache Materializations should [return](https://docs.getdbt.com/reference/dbt-jinja-functions/return.md) the list of Relations that they have created at the end of execution. dbt will use this list of Relations to update the relation cache in order to reduce the number of queries executed against the database's `information_schema`. If a list of Relations is not returned, then dbt will raise a Deprecation Warning and infer the created relation from the model's configured database, schema, and alias. macros/my\_view\_materialization.sql ```sql {%- materialization my_view, default -%} {%- set target_relation = api.Relation.create( identifier=this.identifier, schema=this.schema, database=this.database, type='view') -%} -- ... setup database ... -- ... run pre-hooks... -- build model {% call statement('main') -%} {{ create_view_as(target_relation, sql) }} {%- endcall %} -- ... run post-hooks ... -- ... clean up the database... -- Return the relations created in this materialization {{ return({'relations': [target_relation]}) }} {%- endmaterialization -%} ``` If a materialization solely creates a single relation, then returning that relation at the end of the materialization is sufficient to synchronize the dbt Relation cache. If the materialization *renames* or *drops* Relations other than the relation returned by the materialization, then additional work is required to keep the cache in sync with the database. To explicitly remove a relation from the cache, use [adapter.drop\_relation](https://docs.getdbt.com/reference/dbt-jinja-functions/adapter.md). To explicitly rename a relation in the cache, use [adapter.rename\_relation](https://docs.getdbt.com/reference/dbt-jinja-functions/adapter.md). Calling these methods is preferable to executing the corresponding SQL directly, as they will mutate the cache as required. If you do need to execute the SQL to drop or rename relations directly, use the `adapter.cache_dropped` and `adapter.cache_renamed` methods to synchronize the cache. #### Materialization Configuration Materializations support custom configuration. You might be familiar with some of these configs from materializations like `unique_key` in [incremental models](https://docs.getdbt.com/docs/build/incremental-models.md) or `strategy` in [snapshots](https://docs.getdbt.com/docs/build/snapshots.md) . ##### Specifying configuration options Materialization configurations can either be "optional" or "required". If a user fails to provide required configurations, then dbt will raise a compilation error. You can define these configuration options with the `config.get` and `config.require` functions. ```text # optional config.get('optional_config_name', default="the default") # required config.require('required_config_name') ``` For more information on the `config` dbt Jinja function, see the [config](https://docs.getdbt.com/reference/dbt-jinja-functions/config.md) reference. #### Materialization precedence dbt will pick the materialization macro in the following order (lower takes priority): 1. global project - default 2. global project - plugin specific 3. imported package - default 4. imported package - plugin specific 5. local project - default 6. local project - plugin specific In each of the stated search spaces, a materialization can only be defined once. Two different imported packages may not supply the same materialization - an error will be raised. Specific materializations can be selected by using the dot-notation when selecting a materialization from the context. We recommend *not* overriding materialization names directly, and instead using a prefix or suffix to denote that the materialization changes the behavior of the default implementation (eg. my\_project\_incremental). --- ### Customize dbt models database, schema, and alias [Back to guides](https://docs.getdbt.com/guides.md) Advanced [Menu ]() #### Introduction This guide explains how to customize the [schema](https://docs.getdbt.com/docs/build/custom-schemas.md), [database](https://docs.getdbt.com/docs/build/custom-databases.md), and [alias](https://docs.getdbt.com/docs/build/custom-aliases.md) naming conventions in dbt to fit your data warehouse governance and design needs. When we develop dbt models and execute certain [commands](https://docs.getdbt.com/reference/dbt-commands.md) (such as `dbt run` or `dbt build`), objects (like tables and views) get created in the data warehouse based on these naming conventions. A word on naming Different warehouses have different names for *logical databases*. The information in this document covers "databases" on Snowflake, Redshift, and Postgres; "projects" on BigQuery; and "catalogs" on Databricks Unity Catalog. The following is dbt's out-of-the-box default behavior: * The database where the object is created is defined by the database configured at the [environment level in dbt](https://docs.getdbt.com/docs/dbt-platform-environments.md) or in the [`profiles.yml` file](https://docs.getdbt.com/docs/local/profiles.yml.md) in dbt v1. * The schema depends on whether you have defined a [custom schema](https://docs.getdbt.com/docs/build/custom-schemas.md) for the model: * If you haven't defined a custom schema, dbt creates the object in the default schema. In dbt, this is typically `dbt_username` for development and the default schema for deployment environments. In dbt v1, it uses the schema specified in the `profiles.yml` file. * If you define a custom schema, dbt concatenates the schema mentioned earlier with the custom one. * For example, if the configured schema is `dbt_myschema` and the custom one is `marketing`, the objects will be created under `dbt_myschema_marketing`. * Note that for automated CI jobs, the schema name derives from the job number and PR number: `dbt_cloud_pr_<job_id>_<pr_id>`. * The object name depends on whether an [alias](https://docs.getdbt.com/reference/resource-configs/alias.md) has been defined on the model: * If no alias is defined, the object will be created with the same name as the model, without the `.sql` or `.py` at the end. * For example, suppose that we have a model where the SQL file is titled `fct_orders_complete.sql`, the custom schema is `marketing`, and no custom alias is configured. The resulting model will be created in `dbt_myschema_marketing.fct_orders_complete` in the dev environment. * If an alias is defined, the object will be created with the configured alias. * For example, suppose that we have a model where the SQL file is titled `fct_orders_complete.sql`, the custom schema is `marketing`, and the alias is configured to be `fct_orders`. The resulting model will be created in `dbt_myschema_marketing.fct_orders` These default rules are a great starting point, and many organizations choose to stick with those without any customization required. The defaults allow developers to work in their isolated schemas (sandboxes) without overwriting each other's work — even if they're working on the same tables. #### How to customize this behavior While the default behavior will fit the needs of most organizations, there are occasions where this approach won't work. For example, dbt expects that it has permission to create schemas as needed (and we recommend that the users running dbt have this ability), but it might not be allowed at your company. Or, based on how you've designed your warehouse, you may wish to minimize the number of schemas in your dev environment (and avoid schema sprawl by not creating the combination of all developer schemas and custom schemas). Alternatively, you may even want your dev schemas to be named after feature branches instead of the developer name. For this reason, dbt offers three macros to customize what objects are created in the data warehouse: * [`generate_database_name()`](https://docs.getdbt.com/docs/build/custom-databases.md#generate_database_name) * [`generate_schema_name()`](https://docs.getdbt.com/docs/build/custom-schemas.md#how-does-dbt-generate-a-models-schema-name) * [`generate_alias_name()`](https://docs.getdbt.com/docs/build/custom-aliases.md#generate_alias_name) By overwriting one or multiple of those macros, we can tailor where dbt objects are created in the data warehouse and align with any existing requirement. Key concept Models run from two different contexts must result in unique objects in the data warehouse. For example, a developer named Suzie is working on enhancements to `fct_player_stats`, but Darren is developing against the exact same object. In order to prevent overwriting each other's work, both Suzie and Darren should each have their unique versions of `fct_player_stats` in the development environment. Further, the staging version of `fct_player_stats` should exist in a unique location apart from the development versions, and the production version. We often leverage the following when customizing these macros: * In dbt, we recommend utilizing [environment variables](https://docs.getdbt.com/docs/build/environment-variables.md) to define where the dbt invocation is occurring (dev/stg/prod). * They can be set at the environment level and all jobs will automatically inherit the default values. We'll add Jinja logic (`if/else/endif`) to identify whether the run happens in dev, prod, Ci, and more. * Or as an alternative to environment variables, you can use `target.name`. For more information, you can refer to [About target variables](https://docs.getdbt.com/reference/dbt-jinja-functions/target.md). [![Custom schema environmental variables target name.](</img/docs/dbt-platform/using-dbt-platform/Environment Variables/custom-schema-env-var.png?v=2> "Custom schema environmental variables target name.")](#)Custom schema environmental variables target name. To allow the database/schema/object name to depend on the current branch, you can use the out-of-the-box `DBT_CLOUD_GIT_BRANCH` environment variable in dbt [special environment variables](https://docs.getdbt.com/docs/build/environment-variables.md#special-environment-variables). #### Example use cases Here are some typical examples we've encountered with dbt users leveraging those 3 macros and different logic. note Note that the following examples are not comprehensive and do not cover all the available options. These examples are meant to be templates for you to develop your own behaviors. * [Use custom schema without concatenating target schema in production](https://docs.getdbt.com/guides/customize-schema-alias.md?step=3#1-custom-schemas-without-target-schema-concatenation-in-production) * [Add developer identities to tables](https://docs.getdbt.com/guides/customize-schema-alias.md?step=3#2-static-schemas-add-developer-identities-to-tables) * [Use branch name as schema prefix](https://docs.getdbt.com/guides/customize-schema-alias.md?step=3#3-use-branch-name-as-schema-prefix) * [Use a static schema for CI](https://docs.getdbt.com/guides/customize-schema-alias.md?step=3#4-use-a-static-schema-for-ci) ##### 1. Custom schemas without target schema concatenation in production The most common use case is using the custom schema without concatenating it with the default schema name when in production. To do so, you can create a new file called `generate_schema_name.sql` under your macros folder with the following code: macros/generate\_schema\_name.sql ```jinja {% macro generate_schema_name(custom_schema_name, node) -%} {%- set default_schema = target.schema -%} {%- if custom_schema_name is none -%} {{ default_schema }} {%- elif env_var('DBT_ENV_TYPE','DEV') == 'PROD' -%} {{ custom_schema_name | trim }} {%- else -%} {{ default_schema }}_{{ custom_schema_name | trim }} {%- endif -%} {%- endmacro %} ``` This will generate the following outputs for a model called `my_model` with a custom schema of `marketing`, preventing any overlap of objects between dbt runs from different contexts. | Context | Target database | Target schema | Resulting object | | ----------- | --------------- | ------------- | ------------------------------------ | | Developer 1 | dev | dbt\_dev1 | dev.dbt\_dev1\_marketing.my\_model | | Developer 2 | dev | dbt\_dev2 | dev.dbt\_dev2\_marketing.my\_model | | CI PR 123 | ci | dbt\_pr\_123 | ci.dbt\_pr\_123\_marketing.my\_model | | CI PR 234 | ci | dbt\_pr\_234 | ci.dbt\_pr\_234\_marketing.my\_model | | Production | prod | analytics | prod.marketing.my\_model | note We added logic to check if the current dbt run is happening in production or not. This is important, and we explain why in the [What not to do](https://docs.getdbt.com/guides/customize-schema-alias.md?step=3#what-not-to-do) section. ##### 2. Static schemas: Add developer identities to tables Occasionally, we run into instances where the security posture of the organization prevents developers from creating schemas and all developers have to develop in a single schema. In this case, we can: * Create a new file called generate\_schema\_name.sql under your macros folder with the following code: * Change `generate_schema_name()` to use a single schema for all developers, even if a custom schema is set. * Update `generate_alias_name()` to append the developer alias and the custom schema to the front of the table name in the dev environment. * This method is not ideal, as it can cause long table names, but it will let developers see in which schema the model will be created in production. macros/generate\_schema\_name.sql ```jinja {% macro generate_schema_name(custom_schema_name, node) -%} {%- set default_schema = target.schema -%} {%- if custom_schema_name is none -%} {{ default_schema }} {%- elif env_var('DBT_ENV_TYPE','DEV') != 'CI' -%} {{ custom_schema_name | trim }} {%- else -%} {{ default_schema }}_{{ custom_schema_name | trim }} {%- endif -%} {%- endmacro %} ``` macros/generate\_alias\_name.sql ```jinja {% macro generate_alias_name(custom_alias_name=none, node=none) -%} {%- if env_var('DBT_ENV_TYPE','DEV') == 'DEV' -%} {%- if custom_alias_name -%} {{ target.schema }}__{{ custom_alias_name | trim }} {%- elif node.version -%} {{ target.schema }}__{{ node.name ~ "_v" ~ (node.version | replace(".", "_")) }} {%- else -%} {{ target.schema }}__{{ node.name }} {%- endif -%} {%- else -%} {%- if custom_alias_name -%} {{ custom_alias_name | trim }} {%- elif node.version -%} {{ return(node.name ~ "_v" ~ (node.version | replace(".", "_"))) }} {%- else -%} {{ node.name }} {%- endif -%} {%- endif -%} {%- endmacro %} ``` This will generate the following outputs for a model called `my_model` with a custom schema of `marketing`, preventing any overlap of objects between dbt runs from different contexts. | Context | Target database | Target schema | Resulting object | | ----------- | --------------- | ------------- | ------------------------------------ | | Developer 1 | dev | dbt\_dev1 | dev.marketing.dbt\_dev1\_my\_model | | Developer 2 | dev | dbt\_dev2 | dev.marketing.dbt\_dev2\_my\_model | | CI PR 123 | ci | dbt\_pr\_123 | ci.dbt\_pr\_123\_marketing.my\_model | | CI PR 234 | ci | dbt\_pr\_234 | ci.dbt\_pr\_234\_marketing.my\_model | | Production | prod | analytics | prod.marketing.my\_model | ##### 3. Use branch name as schema prefix For teams who prefer to isolate work based on the feature branch, you may want to take advantage of the `DBT_CLOUD_GIT_BRANCH` special environment variable. Please note that developers will write to the exact same schema when they are on the same feature branch. note The `DBT_CLOUD_GIT_BRANCH` variable is only available within the Studio IDE and not the dbt platform CLI. We’ve also seen some organizations prefer to organize their dev databases by branch name. This requires implementing similar logic in `generate_database_name()` instead of the `generate_schema_name()` macro. By default, dbt will not automatically create the databases. Refer to the [Tips and tricks](https://docs.getdbt.com/guides/customize-schema-alias.md?step=5) section to learn more. macros/generate\_schema\_name.sql ```jinja {% macro generate_schema_name(custom_schema_name, node) -%} {%- set default_schema = target.schema -%} {%- if env_var('DBT_ENV_TYPE','DEV') == 'DEV' -%} {#- we replace characters not allowed in the schema names by "_" -#} {%- set re = modules.re -%} {%- set cleaned_branch = re.sub("\W", "_", env_var('DBT_CLOUD_GIT_BRANCH')) -%} {%- if custom_schema_name is none -%} {{ cleaned_branch }} {%- else -%} {{ cleaned_branch }}_{{ custom_schema_name | trim }} {%- endif -%} {%- else -%} {{ default_schema }}_{{ custom_schema_name | trim }} {%- endif -%} {%- endmacro %} ``` This will generate the following outputs for a model called `my_model` with a custom schema of `marketing`, preventing any overlap of objects between dbt runs from different contexts. | Context | Branch | Target database | Target schema | Resulting object | | ----------- | ------------ | --------------- | ------------- | ------------------------------------ | | Developer 1 | `featureABC` | dev | dbt\_dev1 | dev.featureABC\_marketing.my\_model | | Developer 2 | `featureABC` | dev | dbt\_dev2 | dev.featureABC\_marketing.my\_model | | Developer 1 | `feature123` | dev | dbt\_dev1 | dev.feature123\_marketing.my\_model | | CI PR 123 | | ci | dbt\_pr\_123 | ci.dbt\_pr\_123\_marketing.my\_model | | CI PR 234 | | ci | dbt\_pr\_234 | ci.dbt\_pr\_234\_marketing.my\_model | | Production | | prod | analytics | prod.marketing.my\_model | When developer 1 and developer 2 are checked out on the same branch, they will generate the same object in the data warehouse. This shouldn't be a problem as being on the same branch means the model's code will be the same for both developers. ##### 4. Use a static schema for CI Some organizations prefer to write their CI jobs to a single schema with the PR identifier prefixed to the front of the table name. It's important to note that this will result in long table names. To do so, you can create a new file called `generate_schema_name.sql` under your macros folder with the following code: macros/generate\_schema\_name.sql ```jinja {% macro generate_schema_name(custom_schema_name=none, node=none) -%} {%- set default_schema = target.schema -%} {# If the CI Job does not exist in its own environment, use the target.name variable inside the job instead #} {# {%- if target.name == 'CI' -%} #} {%- if env_var('DBT_ENV_TYPE','DEV') == 'CI' -%} ci_schema {%- elif custom_schema_name is none -%} {{ default_schema }} {%- else -%} {{ default_schema }}_{{ custom_schema_name | trim }} {%- endif -%} {%- endmacro %} ``` macros/generate\_alias\_name.sql ```jinja {% macro generate_alias_name(custom_alias_name=none, node=none) -%} {# If the CI Job does not exist in its own environment, use the target.name variable inside the job instead #} {# {%- if target.name == 'CI' -%} #} {%- if env_var('DBT_ENV_TYPE','DEV') == 'CI' -%} {%- if custom_alias_name -%} {{ target.schema }}__{{ node.config.schema }}__{{ custom_alias_name | trim }} {%- elif node.version -%} {{ target.schema }}__{{ node.config.schema }}__{{ node.name ~ "_v" ~ (node.version | replace(".", "_")) }} {%- else -%} {{ target.schema }}__{{ node.config.schema }}__{{ node.name }} {%- endif -%} {%- else -%} {%- if custom_alias_name -%} {{ custom_alias_name | trim }} {%- elif node.version -%} {{ return(node.name ~ "_v" ~ (node.version | replace(".", "_"))) }} {%- else -%} {{ node.name }} {%- endif -%} {%- endif -%} {%- endmacro %} ``` This will generate the following outputs for a model called `my_model` with a custom schema of `marketing`, preventing any overlap of objects between dbt runs from different contexts. | Context | Target database | Target schema | Resulting object | | ----------- | --------------- | ------------- | ------------------------------------------------ | | Developer 1 | dev | dbt\_dev1 | dev.dbt\_dev1\_marketing.my\_model | | Developer 2 | dev | dbt\_dev2 | dev.dbt\_dev2\_marketing.my\_model | | CI PR 123 | ci | dbt\_pr\_123 | ci.ci\_schema.dbt\_pr\_123\_marketing\_my\_model | | CI PR 234 | ci | dbt\_pr\_234 | ci.ci\_schema.dbt\_pr\_234\_marketing\_my\_model | | Production | prod | analytics | prod.marketing.my\_model | #### What not to do This section will provide an outline of what users should avoid doing when customizing their schema and alias due to the issues that may arise. ##### Update generate\_schema\_name() to always use the custom schema Some people prefer to only use the custom schema when it is set instead of concatenating the default schema with the custom one, as it happens in the out of the box behavior. ##### Problem When modifying the default macro for `generate_schema_name()`, this might result in creating this new version. macros/generate\_schema\_name.sql ```jinja {% macro generate_schema_name(custom_schema_name, node) -%} {%- set default_schema = target.schema -%} {%- if custom_schema_name is none -%} {{ default_schema }} {%- else -%} # The following is incorrect as it omits {{ default_schema }} before {{ custom_schema_name | trim }}. {{ custom_schema_name | trim }} {%- endif -%} {%- endmacro %} ``` While it may provide the expected output for production, where a dedicated database is used, it will generate conflicts anywhere people share a database. Let’s look at the example of a model called `my_model` with a custom schema of `marketing`. | Context | Target database | Target schema | Resulting object | | ----------- | --------------- | ------------- | ------------------------ | | Production | prod | analytics | prod.marketing.my\_model | | Developer 1 | dev | dbt\_dev1 | dev.marketing.my\_model | | Developer 2 | dev | dbt\_dev2 | dev.marketing.my\_model | | CI PR 123 | ci | dbt\_pr\_123 | ci.marketing.my\_model | | CI PR 234 | ci | dbt\_pr\_234 | ci.marketing.my\_model | We can see that both developer 1 and developer 2 get the same object for `my_model`. This means that if they both work on this model at the same time, it will be impossible to know if the version currently in the data warehouse is the one from developer 1 and developer 2. Similarly, different PRs will result in the exact same object in the data warehouse. If different PRs are open at the same time and modifying the same models, it is very likely that we will get issues, slowing down the whole development and code promotion. ##### Solution As described in the previous example, update the macro to check if dbt is running in production. Only in production should we remove the concatenation and use the custom schema alone. #### Tips and tricks This section will provide some useful tips on how to properly adjust your `generate_database_name()` and `generate_alias_name()` macros. ##### Creating non existing databases from dbt dbt will automatically try to create a schema if it doesn’t exist and if an object needs to be created in it, but it won’t automatically try to create a database that doesn’t exist. So, if your `generate_database_name()` configuration points to different databases, which might not exist, dbt will fail if you do a simple `dbt build`. It is still possible to get it working in dbt by creating some macros that will check if a database exists and if not, dbt will create it. You can then call those macros either in [a `dbt run-operation ...` step](https://docs.getdbt.com/reference/commands/run-operation.md) in your jobs or as a [`on-run-start` hook](https://docs.getdbt.com/reference/project-configs/on-run-start-on-run-end.md). ##### Assuming context using environment variables rather than `target.name` We prefer to use [environment variables](https://docs.getdbt.com/docs/build/environment-variables.md) over `target.name` For a further read, have a look at ([About target variables](https://docs.getdbt.com/reference/dbt-jinja-functions/target.md)) to decipher the context of the dbt invocation. * `target.name` cannot be set at the environment-level. Therefore, every job within the environment must explicitly specify the `target.name` override. If the job does not have the appropriate `target.name` value set, the database/schema/alias may not resolve properly. Alternatively, environment variable values are inherited by the jobs within their corresponding environment. The environment variable values can also be overwritten within the jobs if needed. [![Customize schema alias env var.](/img/docs/dbt-platform/using-dbt-platform/custom-schema-env-var-targetname.png?v=2 "Customize schema alias env var.")](#)Customize schema alias env var. * `target.name` requires every developer to input the same value (often ‘dev’) into the target name section of their project user credentials. If a developer doesn’t have the appropriate target name value set, their database/schema/alias may not resolve properly. [![User credentials.](/img/docs/dbt-platform/using-dbt-platform/development-credentials.png?v=2 "User credentials.")](#)User credentials. ##### Always enforce custom schemas Some users prefer to enforce custom schemas on all objects within their projects. This avoids writing to unintended “default” locations. You can add this logic to your `generate_schema_name()` macro to [raise a compilation error](https://docs.getdbt.com/reference/dbt-jinja-functions/exceptions.md) if a custom schema is not defined for an object. macros/generate\_schema\_name.sql ```jinja {% macro generate_schema_name(custom_schema_name, node) -%} {%- set default_schema = target.schema -%} {%- set node_custom_schema = node.config.get('schema') -%} {%- if custom_schema_name is none and node_custom_schema is none and node.resource_type == 'model' -%} {{ exceptions.raise_compiler_error("Error: No Custom Schema Defined for the model " ~ node.name ) }} {%- elif custom_schema_name is none -%} {{ default_schema }} {%- elif env_var('DBT_ENV_TYPE','DEV') == 'PROD' -%} {{ custom_schema_name | trim }} {%- else -%} {{ default_schema }}_{{ custom_schema_name | trim }} {%- endif -%} {%- endmacro %} ``` --- ### Customizing CI/CD with custom pipelines [Back to guides](https://docs.getdbt.com/guides.md) dbt platform Orchestration CI Intermediate [Menu ]() #### Introduction One of the core tenets of dbt is that analytic code should be version controlled. This provides a ton of benefit to your organization in terms of collaboration, code consistency, stability, and the ability to roll back to a prior version. There’s an additional benefit that is provided with your code hosting platform that is often overlooked or underutilized. Some of you may have experience using dbt’s [webhook functionality](https://docs.getdbt.com/docs/deploy/continuous-integration.md) to run a job when a PR is created. This is a fantastic capability, and meets most use cases for testing your code before merging to production. However, there are circumstances when an organization needs additional functionality, like running workflows on every commit (linting), or running workflows after a merge is complete. In this article, we will show you how to setup custom pipelines to lint your project and trigger a dbt job via the API. A note on parlance in this article since each code hosting platform uses different terms for similar concepts. The terms `pull request` (PR) and `merge request` (MR) are used interchangeably to mean the process of merging one branch into another branch. ##### What are pipelines? Pipelines (which are known by many names, such as workflows, actions, or build steps) are a series of pre-defined jobs that are triggered by specific events in your repository (PR created, commit pushed, branch merged, etc). Those jobs can do pretty much anything your heart desires assuming you have the proper security access and coding chops. Jobs are executed on [runners](https://docs.github.com/en/actions/learn-github-actions/understanding-github-actions#runners), which are virtual servers. The runners come pre-configured with Ubuntu Linux, macOS, or Windows. That means the commands you execute are determined by the operating system of your runner. You’ll see how this comes into play later in the setup, but for now just remember that your code is executed on virtual servers that are, typically, hosted by the code hosting platform. ![Diagram of how pipelines work](/assets/images/pipeline-diagram-25fbe103dc697bf0237ff92be4a993db.png) Please note, runners hosted by your code hosting platform provide a certain amount of free time. After that, billing charges may apply depending on how your account is setup. You also have the ability to host your own runners. That is beyond the scope of this article, but checkout the links below for more information if you’re interested in setting that up: * Repo-hosted runner billing information: * [GitHub](https://docs.github.com/en/billing/managing-billing-for-github-actions/about-billing-for-github-actions) * [GitLab](https://docs.gitlab.com/ee/ci/pipelines/cicd_minutes.html) * [Bitbucket](https://bitbucket.org/product/features/pipelines#) * Self-hosted runner information: * [GitHub](https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners) * [GitLab](https://docs.gitlab.com/runner/) * [Bitbucket](https://support.atlassian.com/bitbucket-cloud/docs/runners/) Additionally, if you’re using the free tier of GitLab you can still follow this guide, but it may ask you to provide a credit card to verify your account. You’ll see something like this the first time you try to run a pipeline: ![Warning from GitLab showing payment information is required](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAvQAAABzCAAAAADSncmDAAAdl0lEQVR4Aeyb608Ud/v/f3/S+/Hn4TyZJ5Nsssk+2JCQJSGGYCBEA4aUEqLU0AjRtUELtljxRPGA3GKkB4QSIWqLJVsLEQ9EULEgsoiICzv7/t0zwzXM3Ktb+d7a+/R59bBbrs9c1zXjK2S1vP8f/8fQaLT0Gi29RqOl1/zPoaVfXyfvD9mB/9Jo/oukH+y6ToeerjvcZBlY21D4iS4vAG295r9K+nYYNslZ4FZQeu4ypn3pM/TpinfyL2QtHl+mRvO+NP+NLn9rLiT9NPA7yTMwsiHpmePbpE/iAP9CXgGL1Gjel7/t+Ju8FJCeMRwlWYJmbnQUIdq06klfGZ8iz+9Qu1JAxi9VKqj4MHPflquyszk63I9XkEzGB8iLCRX5bJl++Y94YjSh0nQYKFXm7hmSwzWGVfcs+Ga01og0TZMH4sPkbHyHc11qB2LXeT0GRCuo0WzDenG+gPSdiJCLQIqNQBQo96Q3kWIPABhAxi+VADAH2A7EgKN0+A0myRqcZxdQZaKIfnkGUMALkuwHIoB6xSFAAcbzrTej3ps5VqCPvAe1eR3UxqAJGAluB4223nG+sPRPgEe8DDNnf7pnmneBFZHeBhrXnseBjF/yPt7MAFOcVpgLS1+CK1zdV7/kl2eAuucbdDi+Z5BrBkapcMR+ncCZrTcGkvabBOpD0rfbD4Fb1B9vto+23nG+sPRM4CSr0EoyN5MaAR6J9LPAa/ImkPFLnvTfA8PDw8C1sPQ1iHRN2oHyDPCCwvydG8ClObfbQmrGf/OHM403YIWkf0lWoUNL/3HQ0ncjsQrcJa8qAAHpR12dXTu9kkh/BB5fh6V/HAVg9m6VZwBuMhGFw6UbMOkSeGOQjudrQelJNuGrjyK9Rn+84QLQjYhrd/vYVED6B0CWTAEZv+RJ34PogwcPph4sedIrm6zAeUftU0XAPb8ckN5E3chkBJceuT2Zy4Xf8FcYrECX0+5jS6/Rv5ElywF8RQ7BzPF2QPoN4Axz1UDGL7EVleQkjAWy58spkpwD7nBB4bx94psF0sCwX96SPg08ZVrhkg1cJuvQvvVGoZf8FLu5D5U2kyHpXwMT/MBo9B9Zkr0Apl17yxoQkJ5fAFETQMYv8Weg+JZdDXNvKdQiSeYMoEwB51mOyNFPgFW/HPpOH200gUs8CewsBma23pwFyovg/WGRlUBIepZANfBDodH/c0pYBuIk2aeALoUZV3oLKWabFMwfgHW/RLse6GNmnwEkpujymwl8uhsX+LwaQHySfnl2S/pJC2ioQi95IgJExhl4cyoCFN8g7QbA+BZq87omfE3ejMCkRvPRfuAs9zzLMPaL3D+Wchvuv/94TZ+FFXpk59ZIvxwmvcZNll75b6S2eTrzzKYgZG1qNPpHizWa/xTpNRotvUajpddotPQajZZeo9HSazRaeo1GS6/RaOk1WnqNRkuv0WjpNRotvUajpddotPQajZZeo9HSazRaeo1GS6/RaOk1Gi29RqOl12i09BqNll6jpbc3PHJ8PyY6R1kAe2rBfcmS/KNrg+TS1DpJ/n2Ezbfxoj3Nf56Vrj/rMjbIj4K9QfZO8cZ1kh9kBafbj5sn33TN819H7pspOsjNvT9Xf5dbKMxsF4XshgcfXsgXQ3b5QNKfg8cI34spxK/wnaztVUDkdzIZJ/mtSY4WAWje4EsASLQ+Zx7DuEFBmLjKbTKLKRamuYwfhVNYo9HN+mqS77nC+UcsgNMtXkyXFxin8NMv/Kjkb5VGKxfOrVJu7v2Jdji38OcrX0OOm9TAY/WyCovh7/LhpH+eSnXjWir1ku9Ft8rynWR2mn1PJxtw15O+poE30Pjrs35zD1/iROpGZ9x6xDzmmMfX5n+O9PsquE3pbVz9M+lXXuVLX7aPH4+3b7WQ5a+Y+79Jv/JKVn4/6R+lUo1GKpWyRXpfDNnlw0lPevfF3lJVNUaXPZ0kY0NcqTfM/avk+C61c5QOTQrWaUbPlGAt2xq1mtfI+b1G0Vm63PTMq211pbdVLxO7SfKXoqcvcZ1kJrqHDvetvphqndmhEtN8ZM0xemGnio8zExkjp635gwrWRb/xYJEqukGX+QYzmlznfeu7IlUxTwf7q4iqmOUsLiXcL40WqXgv2Vt23GzzFz8ZUwf2e8a5hXFrjewr9RvJq8x0DwUHO6v3kaxskx1kVxafE+m99t0RoylLf3RfifHJTGiFxxZUJPi8J6xH5FJk9NVnprH7hdvtQJKcrlHxaxjnTIWyDmaZAKxJvy3ZXWSTNyMvpNEZZ4Gm1s3dSdono6rkDpltjRnV83QWj7RkmHfe23hzKzJlLXM50k82HGV84AcDZgPrdx8yIt/SQdZxGd+t4hcY9mGtyTR7oh3OLXgr+wf9a+9bPbFSXi9TlWeRy/ted1n1RY3Psq4Y0+Uqcto7Eh8QHz+c9N/jQH8lHtAhcZgk+thgDV9RSc6qqv5az+ffPjUGHxJo7su2qLPnVR0zxdHLbeilw0lFD1f6h5ix0U0PT3q2mnSYgNV3CKqt3yjjFGYJHLsQKeEbjJB38WSiVg0+ksazOHSzDmmSXItHLp1UezkBo+eYaqNDH3qGi4o4C9V9QlVwDrXfN2CCpxDrnpDFe3Hg6g540ruFm1glu0y/kbzKTO9qGexRU07O4YbsILuy6mlQ+lMovroftyijB3GoryiyGlxh9Rqah4LPO6s6yT6s7lc9580qt1t1PV9HIpc6FMbtSFH/1zjNkeiOwWVpS/IexsnGhN/oaBHJXQ3e7iT5Pdqulqgsm3DiYiRirxVZPR2qMf+8t7G3lbMeRjmEWtrqCtXlp1/hwjjrsftqFe6QpKzj8EiVX/0c/SEfWI/OixY6nFtwVw4clGsnoI5fv4+d/Y14m/SI97Wg0xEjZ+68+bVnDtVl8fHDSV9cR2YiyZD00QZy9gFbI+u04wfp0GmRRDP5GlfJISzfwFOyJUaH+kRQ+osGZ3AzLP13WPakv05aNeQZ5Un/Ofk9FkUk9+6l8SAecz31iiRHMe3YsTyBAXJ/lA7N1gbTqdwseshWlUunslxX3/IUFukvHv+EXDVE+kVSpJdG8ioz3UP+YI8hpNltZGUHf1cyLP0CaTX7o4v3kYv4KbSCjavh591STO6q5d0n5Gkl0g87g85gPJtaJquqvc8K0tYhlmRWdUujgMSLdHk8RU5gahXnyYe1T0bxgBxoyDsvG/sfb0q/5IEiZd/HvCOa9/HGyPIVukhS1nFoNdfI9s6QDy/RQ0560jsrBw/KtRO4RiatNXLXW6WfJot3O2IsoY+c/EOk93z8cNJnnfZs3BGS/hiKv54mLTQ2NiIakP4KmUJ5Y2Mlxk64RWRcASNB6es+4RIGwtKfw4Yn/RxZdITshyd9HzmDyZD00vi5ofYOZOjwjUnyGcYmMEv2gg63YCVv5ziLCXII63zU2VSHEzxl0l98w723PZ5xbkGkl0byKjPdQ/5gj4y6wh0t/g7vkt4gWb9LRmfhvh6TFUT68PO+g/lVDDP7Y3IvINJ3mCQfYJzpiy31iHoGhX4pTN7Got/Il9jkJrfbPitGKoVZOhw36JB3Xjb2pf+6lJERNdkT5Zb0VSRjbXSQdUiWN9Ah5MM4HpNUIenloFw7gaebXzv/VulzZEuJK8YO7Dqfpkjv+fjhpH+D7x1riz3pk570uZuNBrpp4MiRI7VdAen7Hdkajxxp/eRhm1OsT66RjjaeIzlXeuMCabR7XxDp9xXRk/5ZWPp+8okj/XVyclN6v/HiqTJE3Dtvs0i+wKh7eR/o8uCLKKq938gOY30c0QPHHOktUhb37q1+U3rLlf4VedakNJJXmekeksFC4+40fvN3kF2D0kv7hl0yeg2W83pdVhDpw887Z10YVOusRHVrVKT3Bs1ifNk0GtpQ7BkU/KWYw9TBCr/RlsQWPU6hNNmE1C94Rod2iw5vP98QkD6FGaztOVnf4kvv3VzclV7WcShtpEPIh1uYJ2kEpZeDcq33uEv2e4LnS69IJj3p31ytAe6I9J6PH056Rg+SueImOpRXkUvoYzrL9eoImxMkXy6HpX+BG6S9kL2GVfLNAh3u4gTJbKTBkX4GD8hdRpqO2rOe9L8iWUj6LE6Tg470BimN1145578nnVKaHMVCUPpXa+RJzIv0zabNjEgvi0daSDsSkH4cE2RznvQy0zskg4Vf0G3l/B1k13dIL6OjR0guroVWsNEbft5sL69v4gv0kcd96X/EEvkdxp1PWSzfSZbV02/rqdRuXt1q1KGyZNSX3n3PFFJpDJDp7sUhLJBT5/PP+9L30iWr9u1k707z+qb0MyHpZR2HlqhNjgyHfFjAEPkHNqWvZ/CgXOs+7qZ4jvyssPTZ5RzTZpNI7/n4AaX/BpfvHcYvdEji++ka9GWNytnZ4irexhfTI+pwWHqWW7em92FpyaicmCguosshdE5P7MKEI/1lZZOPVen1x31qF1+iY+xasyp9U0h6xmNjd6J4wh8wkJbGF3FtqReTJJk2dt65Ze1gUPpac2Jxv9oQ6Y/jxnS9SC+Lt+PyVBMC0i+jZmoI+dLLTPeQDD46ShfbQNvWDrLrO6SX0R3om+nGSGgFRiqmQs+b08CvzKBu5iflS7+odv4+amA8hc7ZU9hJNpnjb6StSw+wSr/RMI5NH0dA+vLo1O8xpFhpjE5WqsyyueP3n61deedlY3crl904yzlg1RVtHl/MBaWXdbz3B+5fxdmQDyw1b6ZKPendlQMH5Vr3cd/CoakLQI7J3e+UfhptCyl1ZlN68fGDSD/uSJ87bCDSR5d0MZDEFY5ZwI4n5NUIsM/T9aQv/ctdgHWTvFsMlD6hS/aAAVhjdKRvcC0YTwBoXONLAChre8mA9Edd6e9tSv8Ud3nHcAY/5WoCx6Xxm0ZAnaXLVDFQtexdfgV0mNsBWD9zFvfI61hPlwGfRzp52iL9xe1GYGd9OR28wgUFa6/pN5JXmekcksFbvy9JYnprB9nVl75mq/3eXf7oXFJBnWFoBfYqhJ43GTdt8oqCedCTvqaevB2F6sK43QyU1lSQExZS0tYlDWemNLL3AmVle50NPCYtoAO/cbUaiNwh7yeAnQv552VjZyuX87hPRkpc0cgmlHs3F28nSVnH5YcIcGA97MPzMuBwtMO5BXflwEG51n3c7DYQaUOOuyI2PTqsLelLXTG+Va49nvTi44eQXsgt0+eVN4grb+iyvMF81jcveP060OTRIj3Mrs0W0xt8f9JZumTsrcbZNH1er1MQ3qwwyOoaffzF118xjL2UYz7Bm5HBKchzqSuWHfxdCyCjc0t23gr2ujzvgkstewMymwdzGWnrMY1RMtAo85Jh0ptHNzZHv84UOu9tlc9Glj6yjs+yne/DqieNrCwH867Npb0X+32EEB//rX/gbA53+V9B36bqk3swyn8njlmJHDX/VtJ32fyvYPwGXe5++Qv/rThx5hU1/1bSazRaeo1GS6/RaOk1Gi29RqOl1wSxt3/Ff/Z92X+N9D8ObiPqKUUJN35oZru2Udh+jLYAryZfB59GbiMv1PrWJ7bW9/VrfngWvdRxo5rkW8kdPbLMPLwrnJrU/5wnRzr5l3DryCCFPvRwu+vLth9E+njxNqKeUpRw44fmGnLbLGwjUViA8RIAO587T8O76WGsMJzve/sTq0XtCvPxcqfcXsWnOwqgZpmMY4DPunrJOSDNADbwhPm4Vzi1d9b5Y9c0g9yGyb+EduyjkMQB2l1dr7a3vmz7AaRfebWdqKcU5/jfJP1t1N+au1ZqvnaehnPTBaUPPjHVxXz8fOb2KsIJIFFtwMo40RnegulKv/Q+0jtXFJa+BH3/eukzYxlmgMfbXn8b0ocCjtN7VNEN63c/JXsgGUybFo56SjEUbrxcqnb8Sma/iJn1c36w1BsqmdCHNSo+bE1u5UnpR10/O7zfsC6SwUBlaCEpSGdJhErdYXPOLK6WqKqJQhFg2VPmuuyoJMnVcudpeDct0vuh1mO1KtFPlwNJSZ1OWzAiGXdbiat6E73c6WfJOnVzM6gqIVKvIvu5J/yH4rEInCEXDfSyNvbz9yYQ+86X/mmtqXaM0waulRp10yT5MhZbZjJ2ko9iCeeKoDWdsW/+fpe9JPt2qkSHzRLASHidGixj14yjUV/U+DzL3MUiFTmY4S+xPcfNL9lXajR1xr4mL5epcu/Gj8cayLFYws62FavKAfJorIesj91YicX6Y6V0+Hm3Edm/+o/NrpQYTYcD0p+OnZiJApGDf7I+U3uMaNMy/z5guFhVPXel3xW74HjYXFj6UMDRyWUeVxjzs1PV9eG0aaGopxSD4cYf0HSlAo/ZhM7eaOy1xI28oZuZ0JdWtPcbhXE/T0r6UddK1F0pxyqDgcrQQlKQzpIIlbrD5pxZ4EhfkbFYIAIse/pzSdro4ibV9d5Ni/R+qBV7+vdgbPOMpE5XBtEyaLvbSlzVm+jlTitR1fvHZpJDQqReRfZzT/gPxWMUKkcyPbfMOIb6DcC6ItLbFiwLeGHDJfKaJCP4mQYS7EeVc0XQmha4vGEvYAEtLAJUnCSXLCgF9fI2XL7lVaAISHIIAJqH4dLkXJgAXOtvOI2SqGM9YALX2IAOMoHv0wAQcyUFDKCM4WY34BD6eNMyYwFm05+sP6Wcdom1NFw6XOmPYAcZx6XC0ocCjm4usyssfTBtWjDqKcVguLF4D7nWMLKKHnIWw6KmO1QyoQPOPfRg3M+TkpSoa2XU5jwGg4HK8EJSkM6SCJU6t+bMIkkuoKdABFj29OeSXs737pUrV35znkbo440fai22mS2qE+kldUp8720rcVWZ6H6IqYzZpEgvaVOnIvt5J/yH4tGBUnLkq6++GnAcCH+8WT7Vnc1Z+MkGajO/wcuqNaPzKYC1FnTlS2+tzAMjNHHakXbF/3jTC2s1m8DV28CvmQrU8vqpO+xHzPG0L7NRiar1OQP7aWKENxD3oie4ySiGngK/8wiKgtI3rWyQZD1quagwF25WjdLVaRWWXj7eFFx/L3blFhSup4FuO4kiV/opYPUFsFRY+lDA0c1lPgpLH0ybFox6SjEQbvS+RKbcW7COiZruUMmEfmmRfIJxyZOSpERdKxtIGmeDgcrQQn5BOksiVOrcmjOLWySLDxSIAMue/lyS8448xw0D+/Ok90OtbSRboyK9pE496c2tuKpM9KTfS196CZGGpXdOBB+Kw3GUkE0AqvOl5/zphl1Avw2kyAp0enn22n40YawEU/nSN5E7cGEReE4q/OJL3wQn6GXbt6HIbpSTI1/UAhad5A1pYNg5s38RKC4pBta8aw7PA28GECGngTcB6Z/SJYIhp2su3MzCAPnZ26UvuH7UubAW7Wlggc6mjvSMYOQHlLOw9OHAq0VyzpE+KdKH06aFop5SDIQbM+inw8+Yd7ZslWCpO1QyoZ0GySmMS56UJCXqWtlI0jwbDFSGFvIL0lkSoVKnP8fJm5Lcsa9ABFj2lLnebZ6gg3k0T3o/1NpB8pgp0ksWyZPe2oqrykRP+saA9JI2dSuyn3Mi+FAcBqBWuXC/FgfypZ9TsKo86e+S1ThGksswmzCDNig7X/pDZCUuPAWWSQMjvvR7vVinp1EvypkEdijX07h4uhdNT4G6ujqzbtk7G+tHHa84J2aBVw1oJ4tc6ZfpYsK7jXAzE8Nky1ulL7C+XNiAw+6ACZH+KxxsxKX3kV4Cjm4u8weMeSlZX3o/m1kw6inFYLgx9jmZuzS5hJ+c/X+QYKk7VDKhtzBENmLcz5OSlKiryBcMVIYWkoJ0lkSo1Lk1Z9Y5sqa6CkSAZc+g9KwxnJUm8YMnfa9IHwi1VpGs2P1O6SWuKhOd3Kk3QoKqkjZ1KrKfd8J/KB7PgQM25wxc2pReZX3pe5Bg1nKlb+WCwiAdimBU0TTwCd8lPRUG+BBYYAnO0KEb8SyTpSO+9Ao32ed7ugfxh6PAfipMcWNkNLeZpizDT06fZ7yICJtQyRUVlL7GsbmydD7crBL1XLXypZ/kn6xfi8+5ZuLHsPTTiBhYeh/pJeC4qMrvjBoY81KyvvR+2rRg1FOKwXDjKfQ8+AKT3GnenKg0FiVY6g6VTGhuD8piEYz7eVKSEnUV+fxAJRleSArSWRKhUufWnFng2t1azBSKAMueQemnUTz2+AcV23CehnvTwxgeGxub2Aq1ouNeB/rfKb3EVWWikzv1RkhQVdKmTkX2807IQ5EkaQdglQGRN670L4Ci63OAaVlWz3Vgfwlc6RFTUC/o0Aqc5l7g0rulPwxVY6CCPARVS5KPFYorgHlf+iIUNUE8dcQGDOznYRgtCZRKQBp4QzsBsxro5EUgbiIo/SCwK45ouBn74GiKfZwuq/WlZxxm55+sfx2ojMFcCkvPOLCT7yW9BBxvR4EOjPkp2Zr6cNq0YNRTisFwY6sB1Uuu7AaKJ/1gqTdUMqGZwWTXDMYlT+ogUdcqV76urUAlGV5ICn5nSYRKfWvOLDoNWEMFI8Cyp8x1mYwCqJqj8zTcmx6GQ3wr1HooDtVOF3lie13pvW0lrupPbEK5N0KCqpI2dSqyn3dCHookSXM9CkDlE9cBsl3h3BxcTmYbgYpiV/pOhdhtutwEJngFmM2z5iCSjvQXaR8yoGpfkzNFUHRIxQHzpqfRZZQzZUI1w+JPrqec/rrx+8Noci9E1QJd7gCfklyuAox2MlMJ1BThh2VAYok9FpB48o/NjigUf4ZGXkPDlvQ/maj5k/XZFwFKHtEZEJC+E+gtLH1+wDGdTWNMUrIF06Z5UU8pSqNgEJIbr4NfDmZCnx2cJLuwJHnSvKir9MlbSArSWRKhUs/LntrpghFg2TOP9L3V8E0HG0sEtCDpDZkYzJ1KUFU6SEX2k5BtKEmam5ta5RbBwRm/YL/ktliwN5vLDa2mKeTd4LfVHZnHFvqc8/NvZPZFIEWH7AJdVvwWPktr0iyAvUKHTvy43R+0SedHZlcaoFbfT/ogjvR/JXZCVcbRzP8UNL/BoXiZQQygnv8MjVjmP80MgF5uX/pM1zz/UrLfHz1+i/85aJ6dafryxw2GOJj8Mcd/hpNf8p9ncW9bSv9osUajpddo6TUaLb1Go6XXaLT0HzbOafOvRaOll8ThNug5MuVdOnC42X6vnOU91UCXx0dO0sdPRP6VaLT0ki3bDqW4SofTgCogfSDEdR1RuvwCM5TI9BKR/ylotPRVaMvxvaTPpV6EpZdEppeI/IvRaOklULlxJG7U3KGD5Bu5vM9SpTe2atmjUat7U/oiwExQIpF/j0zWGaMk22JJsi/2KW+XKbNm3pX+dqyaTO02KnthUgpuItNJREpG0wlNdltGc5aaj4uWXgKV9VDFUA9J+lEvViFWBUz7taNw8KSPAioukUgvcDlIcghGjpX45gkQV4hvfryJcV7BwaQUvERmEi1+RnMILuPUfFS09BKofAq1wFYcCkkP/MyR8w/9msLh7MCm9CzDZfqRyBaoVDZL8g0wtQ7MTp0a4DMgLdJ3Q82+SsCkFLyPN470ktEcAqZeJ3CYmo+Kll4ClQNASQmQCElfCRQlpyi1eWCRtALS+5HIFuylRwPOjCNOTrbVVQDPRPr9OEB+B9Mv+NL7Gc0hp9lXqKXmo6Kll0DlFSfwWFqSJEnJN3KlLQrgltQeAytkLCC9H4l0YjgeI6hox0kOA0XVwJxI34Ak3SywFHzp/YymG6U5paX/2GjpJVD5EMYan4/8RtLPN67d/tVeKMEhqdlAL+8hIL0fidySfl0hjqf8BEc5G/hOfwqRNdbC9AtOItOV3s9obkmfukfNR0NLL4FKO4GiQybaSfr5xoxC9TcGfvRrDUAcQeklEhmQno1AsWOz0WwFvtNPA0YEMP2Cl8h0pJeMpi/9BLBIzUdDSy+BSifciGSW3Mo3MhUFVHvWr73eDTSUoJ8O5ejzI5Fe4NJPZXaRz0uA1sB3ev5owWiH5Re8RKYjvWQ0fxLpH0OtUvOR0NIHA5Ub81lS8o3yZiEXqmXWGEYikfmsZhhmORcuMJcLZzR9Vt9Q0Gj0T1lqNFr6/98uHQgAAAAACPK3HuRiCKQH6ZEepAfpQXqQHqQH6UF6fqQH6UF6kB6kB+lBepAeAijCS9ryYvlHAAAAAElFTkSuQmCC) ##### How to setup pipelines This guide provides details for multiple code hosting platforms. Where steps are unique, they are presented without a selection option. If code is specific to a platform (i.e. GitHub, GitLab, Bitbucket) you will see a selection option for each. Pipelines can be triggered by various events. The [dbt webhook](https://docs.getdbt.com/docs/deploy/continuous-integration.md) process already triggers a run if you want to run your jobs on a merge request, so this guide focuses on running pipelines for every push and when PRs are merged. Since pushes happen frequently in a project, we’ll keep this job super simple and fast by linting with SQLFluff. The pipeline that runs on merge requests will run less frequently, and can be used to call the dbt API to trigger a specific job. This can be helpful if you have specific requirements that need to happen when code is updated in production, like running a `--full-refresh` on all impacted incremental models. Here’s a quick look at what this pipeline will accomplish: ![Diagram showing the pipelines to be created and the programs involved](/assets/images/pipeline-programs-diagram-c05dd62e86c2d0dfbea0241644c8afa2.png) #### Run a dbt job on merge This job will take a bit more to setup, but is a good example of how to call the dbt API from a CI/CD pipeline. The concepts presented here can be generalized and used in whatever way best suits your use case. Run on merge If your Git provider has a native integration with dbt, you can take advantage of setting up [Merge jobs](https://docs.getdbt.com/docs/deploy/merge-jobs.md) in the UI. The setup below shows how to call the dbt API to run a job every time there's a push to your [default branch](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-branches-in-your-repository/changing-the-default-branch). ##### 1. Get your dbt API key When running a CI/CD pipeline you’ll want to use a service token instead of any individual’s API key. There are [detailed docs](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) available on this, but below is a quick rundown (this must be performed by an Account Admin): 1. Log in to your dbt account. 2. Click your account name at the bottom left-hand menu and go to **Account settings**. 3. Click [**Service tokens**](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md) on the left. 4. Click **+ Create service token** to create a new token specifically for CI/CD API calls. 5. Name your token something like “CICD Token”. 6. Click the **+Add permission** button under **Access**, and grant this token the **Job Admin** permission. 7. Click **Save** and you’ll see a grey box appear with your token. Copy that and save it somewhere safe (this is a password, and should be treated as such). [![View of the dbt page where service tokens are created](/img/guides/orchestration/custom-cicd-pipelines/dbt-service-token-page.png?v=2 "View of the dbt page where service tokens are created")](#)View of the dbt page where service tokens are created [![Creating a new service token](/img/guides/orchestration/custom-cicd-pipelines/dbt-new-service-token-page.png?v=2 "Creating a new service token")](#)Creating a new service token ##### 2. Put your dbt API key into your repo Adding you dbt API key to your repo happens in your code hosting platform. You need to save your API key from the previous section as a repository secret so the job we create can access it. Never save passwords or API keys in your code. The following steps ensure that your key stays secure, but is still usable for your pipelines. ##### GitHub * Open up your repository where you want to run the pipeline (the same one that houses your dbt project). * Click *Settings* to open up the repository options. * On the left click the *Secrets and variables* dropdown in the *Security* section. * From that list, click on *Actions*. * Towards the middle of the screen, click the *New repository secret* button. * It will ask you for a name, so let’s call ours `DBT_API_KEY`. * **It’s very important that you copy/paste this name exactly because it’s used in the scripts below.** * In the *Secret* section, paste in the key you copied from dbt. * Click *Add secret* and you’re all set! \*\* A quick note on security: while using a repository secret is the most straightforward way to setup this secret, there are other options available to you in GitHub. They’re beyond the scope of this guide, but could be helpful if you need to create a more secure environment for running actions. Checkout GitHub’s documentation on secrets [here](https://docs.github.com/en/actions/security-guides/encrypted-secrets).\* Here’s a video showing these steps: ##### GitLab * Open up your repository where you want to run the pipeline (the same one that houses your dbt project). * Click *Settings* > *CI/CD*. * Under the *Variables* section, click *Expand,* then click *Add variable*. * It will ask you for a name, so let’s call ours `DBT_API_KEY`. * **It’s very important that you copy/paste this name exactly because it’s used in the scripts below.** * In the *Value* section, paste in the key you copied from dbt. * Make sure the check box next to *Protect variable* is unchecked, and the box next to *Mask variable* is selected (see below). * “Protected” means that the variable is only available in pipelines that run on protected branches or protected tags - that won’t work for us because we want to run this pipeline on multiple branches. “Masked” means that it will be available to your pipeline runner, but will be masked in the logs. [![\[View of the GitLab window for entering DBT\_API\_KEY](/img/guides/orchestration/custom-cicd-pipelines/dbt-api-key-gitlab.png?v=2 "\[View of the GitLab window for entering DBT_API_KEY")](#)\[View of the GitLab window for entering DBT\_API\_KEY Here’s a video showing these steps: ##### Azure DevOps In Azure: * Open up your Azure DevOps project where you want to run the pipeline (the same one that houses your dbt project). * Click on *Pipelines* and then *Create Pipeline*. * Select where your git code is located. It should be *Azure Repos Git*. * Select your git repository from the list. * Select *Starter pipeline* (this will be updated later in Step 4). * Click on *Variables* and then *New variable*. * In the *Name* field, enter the `DBT_API_KEY`. * **It’s very important that you copy/paste this name exactly because it’s used in the scripts below.** * In the *Value* section, paste in the key you copied from dbt. * Make sure the check box next to *Keep this value secret* is checked. This will mask the value in logs, and you won't be able to see the value for the variable in the UI. * Click *OK* and then *Save* to save the variable. * Save your new Azure pipeline. [![View of the Azure pipelines window for entering DBT\_API\_KEY](/img/guides/orchestration/custom-cicd-pipelines/dbt-api-key-azure.png?v=2 "View of the Azure pipelines window for entering DBT_API_KEY")](#)View of the Azure pipelines window for entering DBT\_API\_KEY ##### Bitbucket In Bitbucket: * Open up your repository where you want to run the pipeline (the same one that houses your dbt project). * In the left menu, click **Repository Settings**. * Scroll to the bottom of the left menu, and select **Repository variables**. * In the **Name** field, input `DBT_API_KEY` * **It’s very important that you copy/paste this name exactly because it’s used in the scripts below.** * In the Value section, paste in the key you copied from dbt * Make sure the check box next to Secured is checked. This will mask the value in logs, and you won't be able to see the value for the variable in the UI. * Click **Add** to save the variable ![View of the Bitbucket window for entering DBT\_API\_KEY](/assets/images/dbt-api-key-bitbucket-8b71a7b1de5da9986737d3cf494ff1d8.png) Here’s a video showing these steps: ##### CodeCommit In AWS, store your API key in Systems Manager (SSM) Parameter Store instead of a repository secret: 1. Open the AWS console and go to **Systems Manager** → **Parameter Store**. 2. Click **Create parameter**. 3. Enter a name for the parameter (for example, `DBT_API_KEY`). 4. Set the type to **SecureString** and paste in your dbt service token as the value. 5. Click **Create parameter**. You’ll reference this parameter in the CodeBuild project environment in step 4. ##### 3. Create script to trigger dbt job via an API call In your project, create a new folder at the root level named `python`. In that folder, create a file named `run_and_monitor_dbt_job.py`. You'll copy/paste the contents from this [gist](https://gist.github.com/b-per/f4942acb8584638e3be363cb87769b48) into that file. ```yaml my_awesome_project ├── python │ └── run_and_monitor_dbt_job.py ``` The following Python file has everything you need to call the dbt API, but requires a few inputs as outlined below. These inputs will be fed to this script through environment variables that will be defined in the next step. ```python #------------------------------------------------------------------------------ # get environment variables #------------------------------------------------------------------------------ api_base = os.getenv('DBT_URL', 'https://cloud.getdbt.com/') # default to multitenant url job_cause = os.getenv('DBT_JOB_CAUSE', 'API-triggered job') # default to generic message git_branch = os.getenv('DBT_JOB_BRANCH', None) # default to None schema_override = os.getenv('DBT_JOB_SCHEMA_OVERRIDE', None) # default to None api_key = os.environ['DBT_API_KEY'] # no default here, just throw an error here if key not provided account_id = os.environ['DBT_ACCOUNT_ID'] # no default here, just throw an error here if id not provided project_id = os.environ['DBT_PROJECT_ID'] # no default here, just throw an error here if id not provided job_id = os.environ['DBT_PR_JOB_ID'] # no default here, just throw an error here if id not provided ``` **Required input:** In order to call the dbt API, there are a few pieces of info the script needs. The easiest way to get these values is to open up the job you want to run in dbt. The URL when you’re inside the job has all the values you need: * `DBT_ACCOUNT_ID` - this is the number just after `accounts/` in the URL * `DBT_PROJECT_ID` - this is the number just after `projects/` in the URL * `DBT_PR_JOB_ID` - this is the number just after `jobs/` in the URL ![Image of a dbt job URL with the pieces for account, project, and job highlighted](/assets/images/dbt-cloud-job-url-30ca274dcf77589fb60b72371b59597c.png) ##### 4. Update your project to include the new API call ##### GitHub For this new job, we'll add a file for the dbt API call named `dbt_run_on_merge.yml`. ```yaml my_awesome_project ├── python │ └── run_and_monitor_dbt_job.py ├── .github │ ├── workflows │ │ └── dbt_run_on_merge.yml │ │ └── lint_on_push.yml ``` The YAML file will look pretty similar to our earlier job, but there is a new section called `env` that we’ll use to pass in the required variables. Update the variables below to match your setup based on the comments in the file. It’s worth noting that we changed the `on:` section to now run **only** when there are pushes to a branch named `main` (for example, a pull request is merged). Have a look through [GitHub documentation](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows) on these filters for additional use cases. For information about `github` context property names and their use cases, refer to the [GitHub documentation](https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/accessing-contextual-information-about-workflow-runs). ```yaml name: run dbt job on push # This filter says only run this job when there is a push to the default branch # This works off the assumption that you've restricted this branch to only all PRs to push to the default branch # Update the name to match the name of your default branch on: push: branches: - 'main' jobs: # the job calls the dbt API to run a job run_dbt_cloud_job: name: Run dbt Job runs-on: ubuntu-latest # Set the environment variables needed for the run env: DBT_ACCOUNT_ID: 00000 # enter your account id DBT_PROJECT_ID: 00000 # enter your project id DBT_PR_JOB_ID: 00000 # enter your job id DBT_API_KEY: ${{ secrets.DBT_API_KEY }} DBT_URL: https://cloud.getdbt.com # enter a URL that matches your job DBT_JOB_CAUSE: 'GitHub Pipeline CI Job' DBT_JOB_BRANCH: ${{ github.head_ref }} # Resolves to the head_ref or source branch of the pull request in a workflow run. steps: - uses: "actions/checkout@v4" - uses: "actions/setup-python@v5" with: python-version: "3.9" - name: Run dbt job run: "python python/run_and_monitor_dbt_job.py" ``` ##### GitLab For this job, we'll set it up using the `gitlab-ci.yml` file as in the prior step (see Step 1 of the linting setup for more info). The YAML file will look pretty similar to our earlier job, but there is a new section called `variables` that we’ll use to pass in the required variables to the Python script. Update this section to match your setup based on the comments in the file. Please note that the `rules:` section now says to run **only** when there are pushes to a branch named `main`, such as a PR being merged. Have a look through [GitLab’s docs](https://docs.gitlab.com/ee/ci/yaml/#rules) on these filters for additional use cases. ##### Only dbt job ```yaml image: python:3.9 variables: DBT_ACCOUNT_ID: 00000 # enter your account id DBT_PROJECT_ID: 00000 # enter your project id DBT_PR_JOB_ID: 00000 # enter your job id DBT_API_KEY: $DBT_API_KEY # secret variable in gitlab account DBT_URL: https://cloud.getdbt.com DBT_JOB_CAUSE: 'GitLab Pipeline CI Job' DBT_JOB_BRANCH: $CI_COMMIT_BRANCH stages: - build # this job calls the dbt API to run a job run-dbt-cloud-job: stage: build rules: - if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == 'main' script: - python python/run_and_monitor_dbt_job.py ``` ##### Lint and dbt job ```yaml image: python:3.9 variables: DBT_ACCOUNT_ID: 00000 # enter your account id DBT_PROJECT_ID: 00000 # enter your project id DBT_PR_JOB_ID: 00000 # enter your job id DBT_API_KEY: $DBT_API_KEY # secret variable in gitlab account DBT_URL: https://cloud.getdbt.com DBT_JOB_CAUSE: 'GitLab Pipeline CI Job' DBT_JOB_BRANCH: $CI_COMMIT_BRANCH stages: - pre-build - build # this job runs SQLFluff with a specific set of rules # note the dialect is set to Snowflake, so make that specific to your setup # details on linter rules: https://docs.sqlfluff.com/en/stable/rules.html lint-project: stage: pre-build rules: - if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH != 'main' script: - python -m pip install sqlfluff==0.13.1 - sqlfluff lint models --dialect snowflake --rules L019,L020,L021,L022 # this job calls the dbt API to run a job run-dbt-cloud-job: stage: build rules: - if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == 'main' script: - python python/run_and_monitor_dbt_job.py ``` ##### Azure DevOps For this new job, open the existing Azure pipeline you created above and select the *Edit* button. We'll want to edit the corresponding Azure pipeline YAML file with the appropriate configuration, instead of the starter code, along with including a `variables` section to pass in the required variables. Copy the below YAML file into your Azure pipeline and update the variables below to match your setup based on the comments in the file. It's worth noting that we changed the `trigger` section so that it will run *only* when there are pushes (for example, when a PR is merged) to `main`, but you should use the name of your [default branch](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-branches-in-your-repository/changing-the-default-branch). Read through [Azure's docs](https://learn.microsoft.com/en-us/azure/devops/pipelines/build/triggers?view=azure-devops) on these filters for additional use cases. ```yaml name: Run dbt Job trigger: [ main ] # runs on pushes to main variables: DBT_URL: https://cloud.getdbt.com # no trailing backslash, adjust this accordingly for single-tenant deployments DBT_JOB_CAUSE: 'Azure Pipeline CI Job' # provide a descriptive job cause here for easier debugging down the road DBT_ACCOUNT_ID: 00000 # enter your account id DBT_PROJECT_ID: 00000 # enter your project id DBT_PR_JOB_ID: 00000 # enter your job id steps: - task: UsePythonVersion@0 inputs: versionSpec: '3.7' displayName: 'Use Python 3.7' - script: | python -m pip install requests displayName: 'Install python dependencies' - script: | python -u ./python/run_and_monitor_dbt_job.py displayName: 'Run dbt job ' env: DBT_API_KEY: $(DBT_API_KEY) # Set these values as secrets in the Azure pipelines Web UI ``` ##### Bitbucket For this job, we'll set it up using the `bitbucket-pipelines.yml` file as in the prior step (see Step 1 of the linting setup for more info). The YAML file will look pretty similar to our earlier job, but we’ll pass in the required variables to the Python script using `export` statements. Update this section to match your setup based on the comments in the file. ##### Only job ```yaml image: python:3.11.1 pipelines: branches: 'main': # override if your default branch doesn't run on a branch named "main" - step: name: 'Run dbt Job' script: - export DBT_URL="https://cloud.getdbt.com" # if you have a single-tenant deployment, adjust this accordingly - export DBT_JOB_CAUSE="Bitbucket Pipeline CI Job" - export DBT_ACCOUNT_ID=00000 # enter your account id here - export DBT_PROJECT_ID=00000 # enter your project id here - export DBT_PR_JOB_ID=00000 # enter your job id here - python python/run_and_monitor_dbt_job.py ``` ##### Lint and dbt job ```yaml image: python:3.11.1 pipelines: branches: '**': # this sets a wildcard to run on every branch unless specified by name below - step: name: Lint dbt project script: - python -m pip install sqlfluff==0.13.1 - sqlfluff lint models --dialect snowflake --rules L019,L020,L021,L022 'main': # override if your default branch doesn't run on a branch named "main" - step: name: 'Run dbt Job' script: - export DBT_URL="https://cloud.getdbt.com" # if you have a single-tenant deployment, adjust this accordingly - export DBT_JOB_CAUSE="Bitbucket Pipeline CI Job" - export DBT_ACCOUNT_ID=00000 # enter your account id here - export DBT_PROJECT_ID=00000 # enter your project id here - export DBT_PR_JOB_ID=00000 # enter your job id here - python python/run_and_monitor_dbt_job.py ``` ##### CodeCommit For CodeCommit, you'll configure a CodeBuild project and an EventBridge rule to trigger a dbt job every time code is pushed to your default branch (for example, when a pull request is merged to `main`). Add the following files to your dbt project: ```text my_awesome_project ├── ci-configuration │ └── buildspec-merge.yml ├── python │ └── run_and_monitor_dbt_cloud_job.py ``` **Create `python/run_and_monitor_dbt_cloud_job.py` with the following content:** ```python import requests import os import time api_base = os.getenv('DBT_URL', 'https://cloud.getdbt.com') job_cause = os.getenv('DBT_JOB_CAUSE', 'API-triggered job') git_branch = os.getenv('DBT_JOB_BRANCH', None) schema_override = os.getenv('DBT_JOB_SCHEMA_OVERRIDE', None) api_key = os.environ['DBT_API_KEY'] account_id = os.environ['DBT_ACCOUNT_ID'] project_id = os.environ['DBT_PROJECT_ID'] job_id = os.environ['DBT_PR_JOB_ID'] print(f""" Configuration: api_base: {api_base} job_cause: {job_cause} git_branch: {git_branch} schema_override: {schema_override} account_id: {account_id} project_id: {project_id} job_id: {job_id} """) req_auth_header = {'Authorization': f'Token {api_key}'} req_job_url = f'{api_base}/api/v2/accounts/{account_id}/jobs/{job_id}/run/' run_status_map = { 1: 'Queued', 2: 'Starting', 3: 'Running', 10: 'Success', 20: 'Error', 30: 'Cancelled', } def run_job(url, headers, cause, branch=None, schema_override=None) -> int: req_payload = {'cause': cause} if branch: req_payload['git_branch'] = branch.replace('refs/heads/', '') if schema_override: req_payload['schema_override'] = schema_override.replace('-', '_') print(f'Triggering job:\n\turl: {url}\n\tpayload: {req_payload}') run_job_resp = requests.post(url, headers=headers, data=req_payload).json() return run_job_resp['data']['id'] def get_run_status(url, headers) -> str: req_status_resp = requests.get(url, headers=headers).json() run_status_code = req_status_resp['data']['status'] return run_status_map[run_status_code] def main(): print('Beginning request for job run...') try: run_id = run_job(req_job_url, req_auth_header, job_cause, git_branch, schema_override) except Exception as e: print(f'ERROR! - Could not trigger job:\n {e}') raise req_status_url = f'{api_base}/api/v2/accounts/{account_id}/runs/{run_id}/' run_status_link = f'{api_base}/#/accounts/{account_id}/projects/{project_id}/runs/{run_id}/' print(f'Job running! See job status at {run_status_link}') time.sleep(30) while True: status = get_run_status(req_status_url, req_auth_header) print(f'Run status -> {status}') if status in ['Error', 'Cancelled']: raise Exception(f'Run failed or canceled. See why at {run_status_link}') if status == 'Success': print(f'Job completed successfully! See details at {run_status_link}') return time.sleep(10) if __name__ == "__main__": main() ``` **Create `ci-configuration/buildspec-merge.yml` with the following content:** Replace the placeholder values with your actual dbt account details: * `YOUR_DBT_ACCOUNT_ID`: The number after `accounts/` in your dbt job URL * `YOUR_DBT_PROJECT_ID`: The number after `projects/` in your dbt job URL * `YOUR_DBT_MERGE_JOB_ID`: The number after `jobs/` in the URL of the dbt job you want to run on merge * `YOUR_SSM_PARAMETER_NAME`: The name of the SSM parameter you created in step 2 (for example, `DBT_API_KEY`) ```yaml version: 0.2 env: variables: DBT_ACCOUNT_ID: YOUR_DBT_ACCOUNT_ID DBT_PROJECT_ID: YOUR_DBT_PROJECT_ID DBT_PR_JOB_ID: YOUR_DBT_MERGE_JOB_ID DBT_URL: https://cloud.getdbt.com DBT_JOB_CAUSE: 'CodeCommit Merge CI Job' parameter-store: DBT_API_KEY: "YOUR_SSM_PARAMETER_NAME" phases: install: commands: - apt-get update -y - python -m pip install --upgrade pip - pip install requests build: commands: - python -u ./python/run_and_monitor_dbt_cloud_job.py ``` **Create a CodeBuild project** Follow the [AWS CodeBuild documentation](https://docs.aws.amazon.com/codebuild/latest/userguide/create-project-console.html) to create a project. Configure the following settings: * **Source**: Select **AWS CodeCommit**, set the reference type to **Branch**, and select your main branch * **Environment**: Select **Ubuntu** as the operating system * **Buildspec**: Select **Use a buildspec file** and enter `ci-configuration/buildspec-merge.yml` as the path Note the service role name shown in the **Environment** section. In **IAM**, find that role and add an inline policy granting permission to read and decrypt the SSM parameter: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": ["ssm:GetParameter", "ssm:GetParameters"], "Resource": "arn:aws:ssm:YOUR_AWS_REGION:YOUR_AWS_ACCOUNT_ID:parameter/YOUR_SSM_PARAMETER_NAME" }, { "Effect": "Allow", "Action": ["kms:Decrypt"], "Resource": "arn:aws:kms:YOUR_AWS_REGION:YOUR_AWS_ACCOUNT_ID:key/alias/aws/ssm" } ] } ``` **Create an EventBridge rule** 1. In the AWS console, go to **Amazon EventBridge** → **Rules** → **Create rule** 2. Give the rule a name (for example, `dbt-run-on-merge`) 3. Under **Event pattern**, select **Custom pattern** and paste the following JSON. Substitute `YOUR_AWS_ACCOUNT_ID`, `YOUR_AWS_REGION`, and `YOUR_CODECOMMIT_REPO_NAME` with your values: ```json { "source": ["aws.codecommit"], "account": ["YOUR_AWS_ACCOUNT_ID"], "region": ["YOUR_AWS_REGION"], "detail-type": ["CodeCommit Repository State Change"], "resources": ["arn:aws:codecommit:YOUR_AWS_REGION:YOUR_AWS_ACCOUNT_ID:YOUR_CODECOMMIT_REPO_NAME"], "detail": { "event": ["referenceUpdated"], "referenceType": ["branch"], "referenceName": ["main"] } } ``` 4. Under **Target**, select **AWS service** → **CodeBuild** and choose the project you created above 5. When prompted, allow EventBridge to create a new IAM role for the target automatically 6. Save the rule ##### 5. Test your new action Now that you have a shiny new action, it’s time to test it out! Since this change is setup to only run on merges to your default branch, you’ll need to create and merge this change into your main branch. Once you do that, you’ll see a new pipeline job has been triggered to run the dbt job you assigned in the variables section. Additionally, you’ll see the job in the run history of dbt. It should be fairly easy to spot because it will say it was triggered by the API, and the *INFO* section will have the branch you used for this guide. ##### GitHub [![dbt run on merge job in GitHub](/img/guides/orchestration/custom-cicd-pipelines/dbt-run-on-merge-github.png?v=2 "dbt run on merge job in GitHub")](#)dbt run on merge job in GitHub [![dbt job showing it was triggered by GitHub](/img/guides/orchestration/custom-cicd-pipelines/dbt-cloud-job-github-triggered.png?v=2 "dbt job showing it was triggered by GitHub")](#)dbt job showing it was triggered by GitHub ##### GitLab [![dbt run on merge job in GitLab](/img/guides/orchestration/custom-cicd-pipelines/dbt-run-on-merge-gitlab.png?v=2 "dbt run on merge job in GitLab")](#)dbt run on merge job in GitLab [![dbt job showing it was triggered by GitLab](/img/guides/orchestration/custom-cicd-pipelines/dbt-cloud-job-gitlab-triggered.png?v=2 "dbt job showing it was triggered by GitLab")](#)dbt job showing it was triggered by GitLab ##### Azure DevOps [![dbt run on merge job in ADO](/img/guides/orchestration/custom-cicd-pipelines/dbt-run-on-merge-azure.png?v=2 "dbt run on merge job in ADO")](#)dbt run on merge job in ADO [![ADO-triggered job in dbt](/img/guides/orchestration/custom-cicd-pipelines/dbt-cloud-job-azure-triggered.png?v=2 "ADO-triggered job in dbt")](#)ADO-triggered job in dbt ##### Bitbucket [![dbt run on merge job in Bitbucket](/img/guides/orchestration/custom-cicd-pipelines/dbt-run-on-merge-bitbucket.png?v=2 "dbt run on merge job in Bitbucket")](#)dbt run on merge job in Bitbucket [![dbt job showing it was triggered by Bitbucket](/img/guides/orchestration/custom-cicd-pipelines/dbt-cloud-job-bitbucket-triggered.png?v=2 "dbt job showing it was triggered by Bitbucket")](#)dbt job showing it was triggered by Bitbucket ##### CodeCommit To test the pipeline, merge a pull request (or push a commit directly) to your main branch. You should see a new build start in your CodeBuild project within a few seconds. You can monitor the run in the CodeBuild console under **Build history**. [![Find new build under build history of CodeBuild console](/img/guides/orchestration/custom-cicd-pipelines/codecommit-run-on-pr.png?v=2 "Find new build under build history of CodeBuild console")](#)Find new build under build history of CodeBuild console In dbt, the job will appear in your run history with a cause of "CodeCommit Merge CI Job". [![Check dbt run history for CodeCommit build triggered](/img/guides/orchestration/custom-cicd-pipelines/codecommit-dbt-cloud-pr.png?v=2 "Check dbt run history for CodeCommit build triggered")](#)Check dbt run history for CodeCommit build triggered #### Run a dbt job on pull request If your git provider is not one with a native integration with dbt, but you still want to take advantage of CI builds, you've come to the right spot! With just a bit of work it's possible to setup a job that will run a dbt job when a pull request (PR) is created. Run on PR If your git provider has a native integration with dbt, you can take advantage of the [setup instructions](https://docs.getdbt.com/docs/deploy/ci-jobs.md). This section is only for those projects that connect to their git repository using an SSH key. The setup for this pipeline will use the same steps as the prior page. Before moving on, make sure you follow [steps 1-5](https://docs.getdbt.com/guides/custom-cicd-pipelines.md?step=2) from the previous page. ##### 1. Create a pipeline job that runs when PRs are created ##### Bitbucket For this job, we’ll set it up using the `bitbucket-pipelines.yml` file as in the prior step. The YAML file will look pretty similar to our earlier job, but we’ll pass in the required variables to the Python script using `export` statements. Update this section to match your setup based on the comments in the file. **What is this pipeline going to do?**<br />The setup below will trigger a dbt job to run every time a PR is opened in this repository. It will also run a fresh version of the pipeline for every commit that is made on the PR until it is merged. For example, if you open a PR, it will run the pipeline. If you then decide additional changes are needed, and commit/push to the PR branch, a new pipeline will run with the updated code. The following variables control this job: * `DBT_JOB_BRANCH`: Tells the dbt job to run the code in the branch that created this PR * `DBT_JOB_SCHEMA_OVERRIDE`: Tells the dbt job to run this into a custom target schema * The format of this will look like: `DBT_CLOUD_PR_{REPO_KEY}_{PR_NUMBER}` ```yaml image: python:3.11.1 pipelines: # This job will run when pull requests are created in the repository pull-requests: '**': - step: name: 'Run dbt PR Job' script: # Check to only build if PR destination is master (or other branch). # Comment or remove line below if you want to run on all PR’s regardless of destination branch. - if [ "${BITBUCKET_PR_DESTINATION_BRANCH}" != "main" ]; then printf ‘PR Destination is not master, exiting.’; exit; fi - export DBT_URL="https://cloud.getdbt.com" - export DBT_JOB_CAUSE="Bitbucket Pipeline CI Job" - export DBT_JOB_BRANCH=$BITBUCKET_BRANCH - export DBT_JOB_SCHEMA_OVERRIDE="DBT_CLOUD_PR_"$BITBUCKET_PROJECT_KEY"_"$BITBUCKET_PR_ID - export DBT_ACCOUNT_ID=00000 # enter your account id here - export DBT_PROJECT_ID=00000 # enter your project id here - export DBT_PR_JOB_ID=00000 # enter your job id here - python python/run_and_monitor_dbt_job.py ``` ##### CodeCommit The setup for CodeCommit follows the same steps as the prior page. Before moving on, follow steps 1–5 from the [prior page](https://docs.getdbt.com/guides/custom-cicd-pipelines.md?step=2) to store your API key, add the Python script to your repository, and configure the merge pipeline. Prerequisites Before proceeding, make sure you have also: * Connected your dbt project to CodeCommit by following the [CodeCommit integration guide](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md#aws-codecommit) * Configured the [pull request URL template for CodeCommit](https://docs.getdbt.com/docs/platform/git/pr-template.md#aws-codecommit) in your dbt project settings * Created a [CI job](https://docs.getdbt.com/docs/deploy/ci-jobs.md) in dbt — do not configure it to trigger on pull requests, since this pipeline will trigger it through the API The pull request pipeline uses a separate CodeBuild project from the merge pipeline, because it runs a different dbt job (your CI job) and requires branch and schema values that are dynamically passed at build time from the pull request event. **1. Add `ci-configuration/buildspec.yml` to your project** This is separate from the `buildspec-merge.yml` you created in the prior step. It references your CI job and leaves `DBT_JOB_BRANCH` and `DBT_JOB_SCHEMA_OVERRIDE` empty — a Lambda function will pass those values at build time. Replace the placeholder values with your actual dbt account details: * `YOUR_DBT_ACCOUNT_ID`: The number after `accounts/` in your dbt job URL * `YOUR_DBT_PROJECT_ID`: The number after `projects/` in your dbt job URL * `YOUR_DBT_PR_JOB_ID`: The number after `jobs/` in the URL of your CI job * `YOUR_SSM_PARAMETER_NAME`: The name of the SSM parameter you created in step 2 of the prior page (for example, `DBT_API_KEY`) ```yaml version: 0.2 env: variables: DBT_ACCOUNT_ID: YOUR_DBT_ACCOUNT_ID DBT_PROJECT_ID: YOUR_DBT_PROJECT_ID DBT_PR_JOB_ID: YOUR_DBT_PR_JOB_ID DBT_URL: https://cloud.getdbt.com DBT_JOB_CAUSE: 'CodeCommit Pipeline CI Job' DBT_JOB_BRANCH: '' DBT_JOB_SCHEMA_OVERRIDE: '' parameter-store: DBT_API_KEY: "YOUR_SSM_PARAMETER_NAME" phases: install: commands: - apt-get update -y - python -m pip install --upgrade pip - pip install requests build: commands: - python -u ./python/run_and_monitor_dbt_cloud_job.py ``` **2. Create a CodeBuild project for pull requests** Follow the [AWS CodeBuild documentation](https://docs.aws.amazon.com/codebuild/latest/userguide/create-project-console.html) to create a second project (separate from the one you created for merge). Configure the following settings: * **Source**: Select **AWS CodeCommit**, set the reference type to **Branch**, and select your main branch * **Environment**: Select **Ubuntu** as the operating system * **Buildspec**: Select **Use a buildspec file** and enter `ci-configuration/buildspec.yml` as the path Note the service role name shown in the **Environment** section. In **IAM**, find that role and add an inline policy granting permission to read and decrypt the SSM parameter: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": ["ssm:GetParameter", "ssm:GetParameters"], "Resource": "arn:aws:ssm:YOUR_AWS_REGION:YOUR_AWS_ACCOUNT_ID:parameter/YOUR_SSM_PARAMETER_NAME" }, { "Effect": "Allow", "Action": ["kms:Decrypt"], "Resource": "arn:aws:kms:YOUR_AWS_REGION:YOUR_AWS_ACCOUNT_ID:key/alias/aws/ssm" } ] } ``` **3. Create a Lambda trigger function** The Lambda function receives the pull request event from EventBridge, extracts the PR branch and PR ID, then starts CodeBuild with those values as environment variable overrides. This is what ensures the dbt CI job runs against your PR branch with a PR-specific schema. 1. In the AWS console, go to **Lambda** → **Create function** 2. Select **Author from scratch**, name the function (for example, `dbt-ci-trigger`), and choose **Python 3.12** as the runtime 3. Click **Create function**, then replace the default code with the following: ```python import boto3 def lambda_handler(event, context): cb = boto3.client('codebuild') source_ref = event['detail']['sourceReference'] pr_id = event['detail']['pullRequestId'] repo_name = event['detail']['repositoryNames'][0] branch = source_ref.replace('refs/heads/', '') schema_override = f'DBT_CLOUD_PR_{repo_name}_{pr_id}'.replace('-', '_').upper() print(f'Triggering CodeBuild for branch: {branch}, schema_override: {schema_override}') cb.start_build( projectName='YOUR_CODEBUILD_PROJECT_NAME', environmentVariablesOverride=[ {'name': 'DBT_JOB_BRANCH', 'value': branch, 'type': 'PLAINTEXT'}, {'name': 'DBT_JOB_SCHEMA_OVERRIDE', 'value': schema_override, 'type': 'PLAINTEXT'} ] ) ``` Replace `YOUR_CODEBUILD_PROJECT_NAME` with the name of the CodeBuild project you created above, then click **Deploy**. 4. Go to **Configuration** → **Permissions** → click the execution role name to open IAM 5. Add an inline policy granting permission to start the CodeBuild project: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "codebuild:StartBuild", "Resource": "arn:aws:codebuild:YOUR_AWS_REGION:YOUR_AWS_ACCOUNT_ID:project/YOUR_CODEBUILD_PROJECT_NAME" } ] } ``` **4. Create an EventBridge rule** 1. In the AWS console, go to **Amazon EventBridge** → **Rules** → **Create rule** 2. Give the rule a name (for example, `dbt-ci-on-pr-created`) 3. Under **Event pattern**, select **Custom pattern** and paste the following JSON. Substitute `YOUR_AWS_ACCOUNT_ID`, `YOUR_AWS_REGION`, and `YOUR_CODECOMMIT_REPO_NAME` with your values: ```json { "source": ["aws.codecommit"], "account": ["YOUR_AWS_ACCOUNT_ID"], "region": ["YOUR_AWS_REGION"], "detail-type": ["CodeCommit Pull Request State Change"], "resources": ["arn:aws:codecommit:YOUR_AWS_REGION:YOUR_AWS_ACCOUNT_ID:YOUR_CODECOMMIT_REPO_NAME"], "detail": { "event": ["pullRequestCreated"], "repositoryNames": ["YOUR_CODECOMMIT_REPO_NAME"] } } ``` 4. Under **Target**, select **AWS service** → **Lambda function**, and choose the Lambda function you created above 5. Save the rule ##### 2. Confirm the pipeline runs Now that you have a new pipeline, it's time to run it and make sure it works. Since this only triggers when a PR is created, you'll need to create a new PR on a branch that contains the code above. Once you do that, you should see a pipeline that looks like this: ##### Bitbucket Bitbucket pipeline: ![dbt run on PR job in Bitbucket](/assets/images/bitbucket-run-on-pr-1887d932eaa80e51157249beef6114a3.png) dbt job: ![ job showing it was triggered by Bitbucket](/assets/images/bitbucket-dbt-cloud-pr-1453e2c293a941eec4561ab9ef045a05.png) ##### CodeCommit To test the pipeline, create a new pull request in your CodeCommit repository on a branch that contains the files you added above. Once you open the pull request, the EventBridge rule fires, Lambda extracts the branch and PR ID, and CodeBuild starts automatically. CodeBuild run: [![CodeBuild run triggered by a CodeCommit pull request](/img/guides/orchestration/custom-cicd-pipelines/codecommit-run-on-pr.png?v=2 "CodeBuild run triggered by a CodeCommit pull request")](#)CodeBuild run triggered by a CodeCommit pull request In dbt, the job run appears in your run history. It will show the PR branch and a schema override in the format `DBT_CLOUD_PR_{REPO_NAME}_{PR_NUMBER}`: [![dbt job showing it was triggered by a CodeCommit pull request](/img/guides/orchestration/custom-cicd-pipelines/codecommit-dbt-cloud-pr.png?v=2 "dbt job showing it was triggered by a CodeCommit pull request")](#)dbt job showing it was triggered by a CodeCommit pull request ##### 3. Handle those extra schemas in your database As noted above, when the PR job runs it will create a new schema based on the PR. To avoid having your database overwhelmed with PR schemas, consider adding a "cleanup" job to your dbt account. This job can run on a scheduled basis to cleanup any PR schemas that haven't been updated/used recently. Add this as a macro to your project. It takes 2 arguments that lets you control which schema get dropped: * `age_in_days`: The number of days since the schema was last altered before it should be dropped (default 10 days) * `database_to_clean`: The name of the database to remove schemas from ```sql {# This macro finds PR schemas older than a set date and drops them The macro defaults to 10 days old, but can be configured with the input argument age_in_days Sample usage with different date: dbt run-operation pr_schema_cleanup --args "{'database_to_clean': 'analytics','age_in_days':'15'}" #} {% macro pr_schema_cleanup(database_to_clean, age_in_days=10) %} {% set find_old_schemas %} select 'drop schema {{ database_to_clean }}.'||schema_name||';' from {{ database_to_clean }}.information_schema.schemata where catalog_name = '{{ database_to_clean | upper }}' and schema_name ilike 'DBT_CLOUD_PR%' and last_altered <= (current_date() - interval '{{ age_in_days }} days') {% endset %} {% if execute %} {{ log('Schema drop statements:' ,True) }} {% set schema_drop_list = run_query(find_old_schemas).columns[0].values() %} {% for schema_to_drop in schema_drop_list %} {% do run_query(schema_to_drop) %} {{ log(schema_to_drop ,True) }} {% endfor %} {% endif %} {% endmacro %} ``` This macro goes into a dbt job that is run on a schedule. The command will look like this (text below for copy/paste): ![ job showing the run operation command for the cleanup macro](/assets/images/dbt-macro-cleanup-pr-c053bfe70d3bc2d4aefa3211713238ce.png) `dbt run-operation pr_schema_cleanup --args "{ 'database_to_clean': 'development','age_in_days':15}"` #### Consider risk of conflicts when using multiple orchestration tools Running dbt jobs through a CI/CD pipeline is a form of job orchestration. If you also run jobs using dbt’s built in scheduler, you now have 2 orchestration tools running jobs. The risk with this is that you could run into conflicts - you can imagine a case where you are triggering a pipeline on certain actions and running scheduled jobs in dbt, you would probably run into job clashes. The more tools you have, the more you have to make sure everything talks to each other. That being said, if **the only reason you want to use pipelines is for adding a lint check or run on merge**, you might decide the pros outweigh the cons, and as such you want to go with a hybrid approach. Just keep in mind that if two processes try and run the same job at the same time, dbt will queue the jobs and run one after the other. It’s a balancing act but can be accomplished with diligence to ensure you’re orchestrating jobs in a manner that does not conflict. --- ### dbt v2 package upgrade guide Learn how to upgrade your packages to be compatible with dbt v2. [Back to guides](https://docs.getdbt.com/guides.md) dbt v2 Advanced [Menu ]() #### Introduction Thank you for being part of the [dbt's package hub community](https://hub.getdbt.com/) and maintaining [packages](https://docs.getdbt.com/docs/build/packages.md)! Your work makes dbt’s ecosystem possible and helps thousands of teams reuse trusted models and macros to build faster, more reliable analytics. This guide helps you upgrade your dbt packages to be [dbt v2](https://docs.getdbt.com/docs/introduction.md)-compatible. A dbt v2-compatible package: * Supports [dbt v2](https://docs.getdbt.com/docs/introduction.md) version `2.0.0` * Uses the [`require-dbt-version` config](https://docs.getdbt.com/reference/project-configs/require-dbt-version.md) to signal compatibility in the dbt package hub * Aligns with the latest JSON schema introduced in dbt v1.10.0 In this guide, we'll go over: * Updating your package to be compatible with dbt v2 * Testing your package with dbt v2 * Updating the `require-dbt-version` config to include `2.0.0` * Updating your README to note that the package is compatible with dbt v2 ##### Who is this for? This guide is for any dbt package maintainer, like [`dbt-utils`](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/), that's looking to upgrade their package to be compatible with dbt v2. Updating your package ensures users have the latest version of your package, your package stays trusted on dbt package hub, and users benefit from the latest features and bug fixes. A user stores their package in a `packages.yml` or `dependencies.yml` file. If a package excludes `2.0.0`, dbt v2 warns today and errors in a future release, matching dbt v1 behavior. This guide assumes you're using the command line and Git to make changes in your package repository. If you're interested in creating a new package from scratch, we recommend using the [dbt package guide](https://docs.getdbt.com/guides/building-packages.md) to get started. #### Prerequisites Before you begin, make sure you meet the following: * dbt package maintainer — You maintain a package on [dbt's package hub](https://hub.getdbt.com/) or are interested in [creating one](https://docs.getdbt.com/guides/building-packages.md?step=1). * `dbt-autofix` installed — [Install `dbt-autofix`](https://github.com/dbt-labs/dbt-autofix?tab=readme-ov-file#installation) to automatically update the package's YAML files to align with the latest dbt updates and best practices. We recommend [using/installing uv/uvx](https://docs.astral.sh/uv/getting-started/installation/) to run the tool. * Run the command `uvx dbt-autofix` for the latest version of the tool. For more installation options, see the [official `dbt-autofix` doc](https://github.com/dbt-labs/dbt-autofix?tab=readme-ov-file#installation). * Repository access — You’ll need permission to create a branch and release updates/a new version of your package. You’ll need to tag a new version of your package once it’s dbt v2-compatible. * A dbt v2 installation or test environment — You can use dbt v2 locally (using the `dbtf` binary) or in your CI pipeline to validate compatibility. * CLI and Git usage — You’re comfortable using the command line and Git to update the repository. #### Upgrade the package This section covers how to upgrade your package to be compatible with dbt v2 by: * [Using `dbt-autofix` to automatically update your YAML files](https://docs.getdbt.com/guides/dbt-package-compat.md?step=) * [Testing your package with dbt v2](https://docs.getdbt.com/guides/dbt-package-compat.md?step=5) * [Updating your `require-dbt-version` config](https://docs.getdbt.com/guides/dbt-package-compat.md?step=6) * [Publishing a new release of your package](https://docs.getdbt.com/guides/dbt-package-compat.md?step=7) If you're ready to get started, let's begin! #### Run dbt-autofix 1. Before you begin, make sure you have `dbt-autofix` installed. If you don't have it installed, run the command `uvx dbt-autofix`. For more installation options, see the [official `dbt-autofix` doc](https://github.com/dbt-labs/dbt-autofix?tab=readme-ov-file#installation). 2. In your dbt package repository, create a branch to work in. For example: ```bash git checkout -b fusion-compat ``` 3. Run `dbt-autofix deprecations` in your package directory so it automatically updates your package code and rewrites YAML to conform to the latest JSON schema: ```bash dbt-autofix deprecations ``` #### Test package with dbt v2 Now that you've run `dbt-autofix`, let's test your package with dbt v2 to ensure it's compatible before [updating](https://docs.getdbt.com/guides/dbt-package-compat?step=6) your `require-dbt-version` config. Refer to the [dbt v2 limitations documentation](https://docs.getdbt.com/docs/dbt/supported-features.md#limitations) for more information on what to look out for. You can test your package two ways: * [Running your integration tests with dbt v2](#running-your-integration-tests-with-fusion) — Use if your package has [integration tests](https://docs.getdbt.com/guides/building-packages?step=4) using an `integration_tests/` folder. * [Manually validating your package](#manually-validating-your-package) — Use if your package doesn't have [integration tests](https://docs.getdbt.com/guides/building-packages?step=4). Consider creating one to help validate your package. ###### Running your integration tests with dbt v2 If your package includes an `integration_tests/` folder ([like `dbt-utils`](https://github.com/dbt-labs/dbt-utils/tree/main/integration_tests)), follow these steps: 1. Navigate to the folder (`cd integration_tests`) to run your tests. If you don't have an `integration_tests/` folder, you can either [create one](https://docs.getdbt.com/guides/building-packages?step=4) or navigate to the folder that contains your tests. 2. Then, run your tests with dbt v2 by running the following `dbtf build` command (or whatever dbt v2 executable is available in your environment). 3. If there are no errors, your package likely supports dbt v2 and you're ready to [update your `require-dbt-version`](https://docs.getdbt.com//guides/dbt-package-compat?step=5#update-your-require-dbt-version). If there are errors, you'll need to fix them first before updating your `require-dbt-version`. ###### Manually validating your package If your package doesn't have integration tests, follow these steps: 1. Create a small, dbt v2-compatible dbt project that installs your package and has a `packages.yml` or `dependencies.yml` file. 2. Run it with dbt v2 using the `dbtf run` command. 3. Confirm that models build successfully and that there are no warnings. If there are errors/warnings, you'll need to fix them first. If you still have issues, reach out to the [#package-ecosystem channel](https://getdbt.slack.com/archives/CU4MRJ7QB) on Slack for help. #### Update `require-dbt-version` Only update the [`require-dbt-version` config](https://docs.getdbt.com/reference/project-configs/require-dbt-version.md) after testing and confirming that your package works with dbt v2. 1. Update the `require-dbt-version` in your `dbt_project.yml` to include `2.0.0`. We recommend using a range to ensure stability across releases: ```yaml require-dbt-version: [">=1.10.0,<3.0.0"] ``` This signals that your package supports both dbt v1 and dbt v2. dbt Labs uses this release metadata to mark your package with a dbt v2-compatible badge in the [dbt package hub](https://hub.getdbt.com/). Packages without this metadata don't display the dbt v2-compatible badge. 2. Commit and push your changes to your repository. #### Publish a new release 1. After committing and pushing your changes, publish a new release of your package by merging your branch into main (or whatever branch you're using for your package). 2. Update your `README` to note that the package is dbt v2-compatible. 3. (Optional) Announce it in [#package-ecosystem on dbt Slack](https://getdbt.slack.com/archives/CU4MRJ7QB) if you’d like. CI dbt v2 testing When possible, add a step to your CI pipeline that runs `dbtf build` or equivalent to ensure ongoing dbt v2 compatibility. Your package is now dbt v2-compatible and the dbt package hub reflects these changes. To summarize, you've now: * Created a v2 compatible branch * Run `dbt-autofix` deprecations * Reviewed, committed, and tested changes * Updated `require-dbt-version: [">=1.10.0,<3.0.0"]` to include `2.0.0` * Published a new release * Announced the update (optional) * Celebrate your new dbt v2-compatible badge 🎉 #### Final thoughts Now that you've upgraded your package to be dbt v2-compatible, users can use your package with dbt v2! 🎉 By upgrading now, you’re ensuring a smoother experience for users, paving the way for the next generation of dbt projects, and helping dbt v2 reach full stability. If you have questions or run into issues: * Join the conversation in the [#package-ecosystem channel](https://getdbt.slack.com/archives/CU4MRJ7QB) on Slack. * Open an issue in the [dbt-autofix repository](https://github.com/dbt-labs/dbt-autofix/issues) on GitHub. Lastly, thank you for your help in making the dbt ecosystem stronger — one package at a time 💜. #### Frequently asked questions The following are some frequently asked questions about upgrading your package to be dbt v2-compatible.  Why do we need to update our package? dbt v2 and dbt v1.10+ use the same new authoring layer. Ensuring your package supports `2.0.0` in your `require-dbt-version` config ensures your package is compatible with both. Updating your package ensures users have the latest version of your package, your package stays trusted on dbt package hub, and users benefit from the latest features and bug fixes. dbt v2-compatible packages display a badge in the dbt package hub. If a package excludes `2.0.0`, dbt v2 will warn today and error in a future release, matching dbt v1 behavior.  How do I test v2 in CI? Add a separate job that installs dbt v2 (`dbtf`) and runs `dbtf build`. See this [PR](https://github.com/godatadriven/dbt-date/pull/31) for a working example. You want to do this to ensure any changes to your package remain compatible with dbt v2.  How will users know my package is v2-compatible? Users can identify your package as dbt v2-compatible by checking for 2.0.0 or higher in the `require-dbt-version` range config. dbt v2-compatible packages also display a badge in the dbt package hub. This is automatically determined based on your package’s metadata and version requirements. --- ### Debug errors [Back to guides](https://docs.getdbt.com/guides.md) Troubleshooting dbt platform Beginner [Menu ]() #### General process of debugging Learning how to debug is a skill, and one that will make you great at your role! 1. Read the error message — when writing the code behind dbt, we try our best to make error messages as useful as we can. The error message dbt produces will normally contain the type of error (more on these error types below), and the file where the error occurred. 2. Inspect the file that was known to cause the issue, and see if there's an immediate fix. 3. Isolate the problem — for example, by running one model a time, or by undoing the code that broke things. 4. Get comfortable with compiled files and the logs. * The `target/compiled` directory contains `select` statements that you can run in any query editor. * The `target/run` directory contains the SQL dbt executes to build your models. * The `logs/dbt.log` file contains all the queries that dbt runs, and additional logging. Recent errors will be at the bottom of the file. * **dbt users**: Use the above, or the `Details` tab in the command output. * **dbt v1 users**: Note that your code editor *may* be hiding these files from the tree view [VSCode help](https://stackoverflow.com/questions/42891463/how-can-i-show-ignored-files-in-visual-studio-code)). 5. If you are really stuck, try [asking for help](https://docs.getdbt.com/community/resources/getting-help.md). Before doing so, take the time to write your question well so that others can diagnose the problem quickly. #### Types of errors Below, we've listed some of common errors. It's useful to understand what dbt is doing behind the scenes when you execute a command like `dbt run`. | Step | Description | Error type | | ---------------- | --------------------------------------------------------------------------------- | ------------------- | | Initialize | Check that this a dbt project, and that dbt can connect to the warehouse | `Runtime Error` | | Parsing | Check that the Jinja snippets in `.sql` files valid, and that `.yml` files valid. | `Compilation Error` | | Graph validation | Compile the dependencies into a graph. Check that it's acyclic. | `Dependency Error` | | SQL execution | Run the models | `Database Error` | Let's dive into some of these errors and how to debug 👇. Note: not all errors are covered here! #### Runtime Errors *Note: If you're using the Studio IDE to work on your project, you're unlikely to encounter these errors.* ##### Not a dbt project ```text Running with dbt=1.7.1 Encountered an error: Runtime Error fatal: Not a dbt project (or any of the parent directories). Missing dbt_project.yml file ``` Debugging * Use `pwd` to check that you're in the right directory. If not, `cd` your way there! * Check that you have a file named `dbt_project.yml` in the root directory of your project. You can use `ls` to list files in the directory, or also open the directory in a code editor and see files in the "tree view". ##### Could not find profile ```text Running with dbt=1.7.1 Encountered an error: Runtime Error Could not run dbt Could not find profile named 'jaffle_shops' ``` Debugging * Check the `profile:` key in your `dbt_project.yml`. For example, this project uses the `jaffle_shops` (note plural) profile: dbt\_project.yml ```yml profile: jaffle_shops # note the plural ``` * Check the profiles you have in your `profiles.yml` file. For example, this profile is named `jaffle_shop` (note singular). profiles.yml ```yaml jaffle_shop: # this does not match the profile: key target: dev outputs: dev: type: postgres schema: dbt_alice ... # other connection details ``` * Update these so that they match. * If you can't find your `profiles.yml` file, run `dbt debug --config-dir` for help: ```text $ dbt debug --config-dir Running with dbt=1.7.1 To view your profiles.yml file, run: open /Users/alice/.dbt ``` * Then execute `open /Users/alice/.dbt` (adjusting accordingly), and check that you have a `profiles.yml` file. If you do not have one, set one up using [these docs](https://docs.getdbt.com/docs/local/profiles.yml.md) ##### Failed to connect ```text Encountered an error: Runtime Error Database error while listing schemas in database "analytics" Database Error 250001 (08001): Failed to connect to DB: your_db.snowflakecomputing.com:443. Incorrect username or password was specified. ``` Debugging * Open your `profiles.yml` file (if you're unsure where this is, run `dbt debug --config-dir`) * Confirm that your credentials are correct — you may need to work with a DBA to confirm this. * After updating the credentials, run `dbt debug` to check you can connect ```text $ dbt debug Running with dbt=1.7.1 Using profiles.yml file at /Users/alice/.dbt/profiles.yml Using dbt_project.yml file at /Users/alice/jaffle-shop-dbt/dbt_project.yml Configuration: profiles.yml file [OK found and valid] dbt_project.yml file [OK found and valid] Required dependencies: - git [OK found] Connection: ... Connection test: OK connection ok ``` ##### Invalid `dbt_project.yml` file ```text Encountered an error while reading the project: ERROR: Runtime Error at path []: Additional properties are not allowed ('hello' was unexpected) Error encountered in /Users/alice/jaffle-shop-dbt/dbt_project.yml Encountered an error: Runtime Error Could not run dbt ``` Debugging * Open your `dbt_project.yml` file. * Find the offending key (e.g. `hello`, as per "'hello' was unexpected") dbt\_project.yml ```yml name: jaffle_shop hello: world # this is not allowed ``` * Use the reference section for [`dbt_project.yml` files](https://docs.getdbt.com/reference/dbt_project.yml.md) to correct this issue. * If you're using a key that is valid according to the documentation, check that you're using the latest version of dbt with `dbt --version`. #### Compilation Errors *Note: if you're using the Studio IDE to work on your dbt project, this error often shows as a red bar in your command prompt as you work on your dbt project. For dbt v1 users, these won't get picked up until you run `dbt run` or `dbt compile`.* ##### Invalid `ref` function ```text $ dbt run -s customers Running with dbt=1.1.0 Encountered an error: Compilation Error in model customers (models/customers.sql) Model 'model.jaffle_shop.customers' (models/customers.sql) depends on a node named 'stg_customer' which was not found ``` Debugging * Open the `models/customers.sql` file. * `cmd + f` (or equivalent) for `stg_customer`. There must be a file named `stg_customer.sql` for this to work. * Replace this reference with a reference to another model (i.e. the filename for another model), in this case `stg_customers`. OR rename your model to `stg_customer` ##### Invalid Jinja ```text $ dbt run Running with dbt=1.7.1 Compilation Error in macro (macros/cents_to_dollars.sql) Reached EOF without finding a close tag for macro (searched from line 1) ``` Debugging Here, we rely on the Jinja library to pass back an error, and then just pass it on to you. This particular example is for a forgotten `{% endmacro %}` tag, but you can also get errors like this for: * Forgetting a closing `}` * Closing a `for` loop before closing an `if` statement To fix this: * Navigate to the offending file (e.g. `macros/cents_to_dollars.sql`) as listed in the error message * Use the error message to find your mistake To prevent this: * *(dbt v1 only)* Use snippets to auto-complete pieces of Jinja ([atom-dbt package](https://github.com/dbt-labs/atom-dbt)) ##### Invalid YAML dbt wasn't able to turn your YAML into a valid dictionary. ```text $ dbt run Running with dbt=1.7.1 Encountered an error: Compilation Error Error reading jaffle_shop: schema.yml - Runtime Error Syntax error near line 5 ------------------------------ 2 | 3 | models: 4 | - name: customers 5 | columns: 6 | - name: customer_id 7 | data_tests: 8 | - unique Raw Error: ------------------------------ mapping values are not allowed in this context in "<unicode string>", line 5, column 12 ``` Debugging Usually, it's to do with indentation — here's the offending YAML that caused this error: ```yaml models: - name: customers columns: # this is indented too far! - name: customer_id data_tests: - unique - not_null ``` To fix this: * Open the offending file (e.g. `schema.yml`) * Check the line in the error message (e.g. `line 5`) * Find the mistake and fix it To prevent this: * (dbt v1 users) Turn on indentation guides in your code editor to help you inspect your files * Use a YAML validator ([example](http://www.yamllint.com/)) to debug any issues ##### Incorrect YAML spec Slightly different error — the YAML structure is right (i.e. the YAML parser can turn this into a python dictionary), *but* there's a key that dbt doesn't recognize. ```text $ dbt run Running with dbt=1.7.1 Encountered an error: Compilation Error Invalid models config given in models/schema.yml @ models: {'name': 'customers', 'hello': 'world', 'columns': [{'name': 'customer_id', 'tests': ['unique', 'not_null']}], 'original_file_path': 'models/schema.yml', 'yaml_key': 'models', 'package_name': 'jaffle_shop'} - at path []: Additional properties are not allowed ('hello' was unexpected) ``` Debugging * Open the file (e.g. `models/schema.yml`) as per the error message * Search for the offending key (e.g. `hello`, as per "**'hello'** was unexpected") * Fix it. Use the [model properties](https://docs.getdbt.com/reference/model-properties.md) docs to find valid keys * If you are using a valid key, check that you're using the latest version of dbt with `dbt --version` #### Dependency Errors ```text $ dbt run Running with dbt=1.7.1-rc Encountered an error: Found a cycle: model.jaffle_shop.customers --> model.jaffle_shop.stg_customers --> model.jaffle_shop.customers ``` Your dbt DAG is not acyclic, and needs to be fixed! * Update the `ref` functions to break the cycle. * If you need to reference the current model, use the [`{{ this }}` variable](https://docs.getdbt.com/reference/dbt-jinja-functions/this.md) instead. #### Database Errors The thorniest errors of all! These errors come from your data warehouse, and dbt passes the message on. You may need to use your warehouse docs (i.e. the Snowflake docs, or BigQuery docs) to debug these. ```text $ dbt run ... Completed with 1 error and 0 warnings: Database Error in model customers (models/customers.sql) 001003 (42000): SQL compilation error: syntax error line 14 at position 4 unexpected 'from'. compiled SQL at target/run/jaffle_shop/models/customers.sql ``` 90% of the time, there's a mistake in the SQL of your model. To fix this: 1. Open the offending file: * **dbt:** Open the model (in this case `models/customers.sql` as per the error message) * **dbt v1:** Open the model as above. Also open the compiled SQL (in this case `target/run/jaffle_shop/models/customers.sql` as per the error message) — it can be useful to show these side-by-side in your code editor. 2. Try to re-execute the SQL to isolate the error: * **dbt:** Use the `Preview` button from the model file * **dbt v1:** Copy and paste the compiled query into a query runner (e.g. the Snowflake UI, or a desktop app like DataGrip / TablePlus) and execute it 3. Fix the mistake. 4. Rerun the failed model. In some cases, these errors might occur as a result of queries that dbt runs "behind-the-scenes". These include: * Introspective queries to list objects in your database * Queries to `create` schemas * `pre-hooks`s, `post-hooks`, `on-run-end` hooks and `on-run-start` hooks * For incremental models, and snapshots: merge, update and insert statements In these cases, you should check out the logs — this contains *all* the queries dbt has run. * **dbt**: Use the `Details` in the command output to see logs, or check the `logs/dbt.log` file * **dbt v1**: Open the `logs/dbt.log` file. Isolating errors in the logs If you're hitting a strange `Database Error`, it can be a good idea to clean out your logs by opening the file, and deleting the contents. Then, re-execute `dbt run` for *just* the problematic model. The logs will *just* have the output you're looking for. #### Common pitfalls ##### `Preview` vs. `dbt run` *(Studio IDE users only)* There's two interfaces that look similar: * The `Preview` button executes whatever SQL statement is in the active tab. It is the equivalent of grabbing the compiled `select` statement from the `target/compiled` directory and running it in a query editor to see the results. * The `dbt run` command builds relations in your database Using the `Preview` button is useful when developing models and you want to visually inspect the results of a query. However, you'll need to make sure you have executed `dbt run` for any upstream models — otherwise dbt will try to select `from` tables and views that haven't been built. ##### Forgetting to save files before running We've all been there. dbt uses the last-saved version of a file when you execute a command. In most code editors, and in the Studio IDE, a dot next to a filename indicates that a file has unsaved changes. Make sure you hit `cmd + s` (or equivalent) before running any dbt commands — over time it becomes muscle memory. ##### Editing compiled files *(More likely for dbt v1 users)* If you just opened a SQL file in the `target/` directory to help debug an issue, it's not uncommon to accidentally edit that file! To avoid this, try changing your code editor settings to grey out any files in the `target/` directory — the visual cue will help avoid the issue. #### FAQs Here are some useful FAQs to help you debug your dbt project: * How to generate HAR files HTTP Archive (HAR) files are used to gather data from users’ browser, which dbt Support uses to troubleshoot network or resource issues. This information includes detailed timing information about the requests made between the browser and the server. The following sections describe how to generate HAR files using common browsers such as [Google Chrome](#google-chrome), [Mozilla Firefox](#mozilla-firefox), [Apple Safari](#apple-safari), and [Microsoft Edge](#microsoft-edge). info Remove or hide any confidential or personally identifying information before you send the HAR file to dbt Labs. You can edit the file using a text editor. ##### Google Chrome 1. Open Google Chrome. 2. Click on **View** --> **Developer Tools**. 3. Select the **Network** tab. 4. Ensure that Google Chrome is recording. A red button (🔴) indicates that a recording is already in progress. Otherwise, click **Record network log**. 5. Select **Preserve Log**. 6. Clear any existing logs by clicking **Clear network log** (🚫). 7. Go to the page where the issue occurred and reproduce the issue. 8. Click **Export HAR** (the down arrow icon) to export the file as HAR. The icon is located on the same row as the **Clear network log** button. 9. Save the HAR file. 10. Upload the HAR file to the dbt Support ticket thread. ##### Mozilla Firefox 1. Open Firefox. 2. Click the application menu and then **More tools** --> **Web Developer Tools**. 3. In the developer tools docked tab, select **Network**. 4. Go to the page where the issue occurred and reproduce the issue. The page automatically starts recording as you navigate. 5. When you're finished, click **Pause/Resume recording network log**. 6. Right-click anywhere in the **File** column and select **Save All as HAR**. 7. Save the HAR file. 8. Upload the HAR file to the dbt Support ticket thread. ##### Apple Safari 1. Open Safari. 2. In case the **Develop** menu doesn't appear in the menu bar, go to **Safari** and then **Settings**. 3. Click **Advanced**. 4. Select the **Show features for web developers** checkbox. 5. From the **Develop** menu, select **Show Web Inspector**. 6. Click the **Network tab**. 7. Go to the page where the issue occurred and reproduce the issue. 8. When you're finished, click **Export**. 9. Save the file. 10. Upload the HAR file to the dbt Support ticket thread. ##### Microsoft Edge 1. Open Microsoft Edge. 2. Click the **Settings and more** menu (...) to the right of the toolbar and then select **More tools** --> **Developer tools**. 3. Click **Network**. 4. Ensure that Microsoft Edge is recording. A red button (🔴) indicates that a recording is already in progress. Otherwise, click **Record network log**. 5. Go to the page where the issue occurred and reproduce the issue. 6. When you're finished, click **Stop recording network log**. 7. Click **Export HAR** (the down arrow icon) or press **Ctrl + S** to export the file as HAR. 8. Save the HAR file. 9. Upload the HAR file to the dbt Support ticket thread. ##### Additional resources Check out the [How to generate a HAR file in Chrome](https://www.loom.com/share/cabdb7be338243f188eb619b4d1d79ca) video for a visual guide on how to generate HAR files in Chrome. * Reconnecting to Snowflake OAuth after authentication expires When you connect Snowflake to dbt platform using [OAuth](https://docs.getdbt.com/docs/platform/manage-access/set-up-snowflake-oauth.md), dbt stores a refresh token. This allows your user credentials to be reused in tools like the Studio IDE and the dbt Semantic Layer without needing to re-authenticate each time. If you see an `authentication has expired` error when you try to run queries, you must renew your connection between Snowflake and the dbt platform. To resolve the issue, complete the following steps: 1. Go to **Your profile**, accessible from the navigation menu. 2. Navigate to **Credentials** and then choose the project where you're experiencing the issue. 3. Under **User credentials**, click the **Reconnect Snowflake Account** button. This will guide you through re-authenticating using your SSO workflow. Your Snowflake administrator can [configure the refresh token validity period](https://docs.getdbt.com/docs/platform/manage-access/set-up-snowflake-oauth.md#create-a-security-integration), up to the maximum 90 days. If you've tried these step and are still getting this error, please contact the Support team at <support@getdbt.com> for further assistance. * Receiving a 'Could not parse dbt\_project.yml' error in dbt job The error message `Could not parse dbt_project.yml: while scanning for...` in your dbt job run or development usually occurs for several reasons: * There's a parsing failure in a YAML file (such as a tab indentation or Unicode characters). * Your `dbt_project.yml` file has missing fields or incorrect formatting. * Your `dbt_project.yml` file doesn't exist in your dbt project repository. To resolve this issue, consider the following: * Use an online YAML parser or validator to check for any parsing errors in your YAML file. Some known parsing errors include missing fields, incorrect formatting, or tab indentation. * Or ensure your `dbt_project.yml` file exists. Once you've identified the issue, you can fix the error and rerun your dbt job. * How can I fix my .gitignore file? A gitignore file specifies which files Git should intentionally ignore. You can identify these files in your project by their italics formatting. If you can't revert changes, check out a branch, or click commit — this is usually do to your project missing a [.gitignore](https://github.com/dbt-labs/dbt-starter-project/blob/main/.gitignore) file OR your gitignore file doesn't contain the necessary content inside the folder. To fix this, complete the following steps: 1. In the Studio IDE, add the following [.gitignore contents](https://github.com/dbt-labs/dbt-starter-project/blob/main/.gitignore) in your dbt project `.gitignore` file: ```bash target/ dbt_packages/ logs/ # legacy -- renamed to dbt_packages in dbt v1 dbt_modules/ ``` 2. Save your changes but *don't commit* 3. Restart the Studio IDE by clicking on the three dots next to the **Studio IDE Status button** on the lower right of the Studio IDE. [![Restart the IDE by clicking the three dots on the lower right or click on the Status bar](/img/docs/dbt-platform/platform-ide/restart-ide.png?v=2 "Restart the IDE by clicking the three dots on the lower right or click on the Status bar")](#)Restart the IDE by clicking the three dots on the lower right or click on the Status bar 4. Select **Restart Studio IDE**. 5. Go back to the **File explorer** in the IDE and delete the following files or folders if you have them: * `target`, `dbt_modules`, `dbt_packages`, `logs` 6. **Save** and then **Commit and sync** your changes. 7. Restart the Studio IDE again. 8. Create a pull request (PR) under the **Version Control** menu to integrate your new changes. 9. Merge the PR on your git provider page. 10. Switch to your main branch and click on **Pull from remote** to pull in all the changes you made to your main branch. You can verify the changes by making sure the files/folders in the .gitignore file are in italics. [![A dbt project on the main branch that has properly configured gitignore folders (highlighted in italics).](/img/docs/dbt-platform/platform-ide/gitignore-italics.png?v=2 "A dbt project on the main branch that has properly configured gitignore folders (highlighted in italics).")](#)A dbt project on the main branch that has properly configured gitignore folders (highlighted in italics). For more info, refer to this [detailed video](https://www.loom.com/share/9b3b8e2b617f41a8bad76ec7e42dd014) for additional guidance. * I'm receiving a 'This run exceeded your account's run memory limits' error in my failed job If you're receiving a `This run exceeded your account's run memory limits` error in your failed job, it means that the job exceeded the [memory limits](https://docs.getdbt.com/docs/deploy/job-scheduler.md#job-memory) set for your account. All dbt accounts have a pod memory of 600Mib and memory limits are on a per run basis. They're typically influenced by the amount of result data that dbt has to ingest and process, which is small but can become bloated unexpectedly by project design choices. ##### Common reasons Some common reasons for higher memory usage are: * dbt run/build: Macros that capture large result sets from run query may not all be necessary and may be memory inefficient. * dbt docs generate: Source or model schemas with large numbers of tables (even if those tables aren't all used by dbt) cause the ingest of very large results for catalog queries. ##### Resolution There are various reasons why you could be experiencing this error but they are mostly the outcome of retrieving too much data back into dbt. For example, using the `run_query()` operations or similar macros, or even using database/schemas that have a lot of other non-dbt related tables/views. Try to reduce the amount of data / number of rows retrieved back into dbt by refactoring the SQL in your `run_query()` operation using `group`, `where`, or `limit` clauses. Additionally, you can also use a database/schema with fewer non-dbt related tables/views. Video example As an additional resource, check out [this example video](https://www.youtube.com/watch?v=sTqzNaFXiZ8), which demonstrates how to refactor the sample code by reducing the number of rows returned. If you've tried the earlier suggestions and are still experiencing failed job runs with this error about hitting the memory limits of your account, please [reach out to support](mailto:support@getdbt.com). We're happy to help! ##### Additional resources * [Blog post on how we shaved 90 mins off](https://docs.getdbt.com/blog/how-we-shaved-90-minutes-off-model) * Why am I receiving a Runtime Error in my packages? If you're receiving the runtime error below in your packages.yml folder, it may be due to an old version of your dbt\_utils package that isn't compatible with your current dbt version. ```shell Running with dbt=xxx Runtime Error Failed to read package: Runtime Error Invalid config version: 1, expected 2 Error encountered in dbt_utils/dbt_project.yml ``` Try updating the old version of the dbt\_utils package in your packages.yml to the latest version found in the [dbt hub](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/): ```shell packages: - package: dbt-labs/dbt_utils version: xxx ``` If you've tried the workaround above and are still experiencing this behavior - reach out to the Support team at <support@getdbt.com> and we'll be happy to help! * \[Error] Could not find my\_project package If a package name is included in the `search_order` of a project-level `dispatch` config, dbt expects that package to contain macros which are viable candidates for dispatching. If an included package does not contain *any* macros, dbt will raise an error like: ```shell Compilation Error In dispatch: Could not find package 'my_project' ``` This does not mean the package or root project is missing—it means that any macros from it are missing, and so it is missing from the search spaces available to `dispatch`. If you've tried the step above and are still experiencing this behavior - reach out to the Support team at <support@getdbt.com> and we'll be happy to help! * What happens if the SQL in my query is bad or I get a database error? If there's a mistake in your SQL, dbt will return the error that your database returns. ```shell $ dbt run --select customers Running with dbt=1.9.0 Found 3 models, 9 tests, 0 snapshots, 0 analyses, 133 macros, 0 operations, 0 seed files, 0 sources 14:04:12 | Concurrency: 1 threads (target='dev') 14:04:12 | 14:04:12 | 1 of 1 START view model dbt_alice.customers.......................... [RUN] 14:04:13 | 1 of 1 ERROR creating view model dbt_alice.customers................. [ERROR in 0.81s] 14:04:13 | 14:04:13 | Finished running 1 view model in 1.68s. Completed with 1 error and 0 warnings: Database Error in model customers (models/customers.sql) Syntax error: Expected ")" but got identifier `your-info-12345` at [13:15] compiled SQL at target/run/jaffle_shop/customers.sql Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1 ``` Any models downstream of this model will also be skipped. Use the error message and the [compiled SQL](https://docs.getdbt.com/faqs/Runs/checking-logs.md) to debug any errors. --- ### Debug schema names [Back to guides](https://docs.getdbt.com/guides.md) Troubleshooting Advanced [Menu ]() #### Introduction If a model uses the [`schema` config](https://docs.getdbt.com/reference/resource-properties/schema.md) but builds under an unexpected schema, here are some steps for debugging the issue. The full explanation of custom schemas can be found [here](https://docs.getdbt.com/docs/build/custom-schemas.md). You can also follow along via this video: #### Search for a macro named `generate_schema_name` Do a file search to check if you have a macro named `generate_schema_name` in the `macros` directory of your project. ##### You do not have a macro named `generate_schema_name` in your project This means that you are using dbt's default implementation of the macro, as defined [here](https://github.com/dbt-labs/dbt-adapters/blob/60005a0a2bd33b61cb65a591bc1604b1b3fd25d5/dbt/include/global_project/macros/get_custom_name/get_custom_schema.sql) ```sql {% macro generate_schema_name(custom_schema_name, node) -%} {%- set default_schema = target.schema -%} {%- if custom_schema_name is none -%} {{ default_schema }} {%- else -%} {{ default_schema }}_{{ custom_schema_name | trim }} {%- endif -%} {%- endmacro %} ``` Note that this logic is designed so that two dbt users won't accidentally overwrite each other's work by writing to the same schema. ##### You have a `generate_schema_name` macro in a project that calls another macro If your `generate_schema_name` macro looks like so: ```sql {% macro generate_schema_name(custom_schema_name, node) -%} {{ generate_schema_name_for_env(custom_schema_name, node) }} {%- endmacro %} ``` Your project is switching out the `generate_schema_name` macro for another macro, `generate_schema_name_for_env`. Similar to the above example, this is a macro which is defined in dbt's global project, [here](https://github.com/dbt-labs/dbt/blob/main/crates/dbt-loader/src/dbt_macro_assets/dbt-adapters/macros/get_custom_name/get_custom_schema.sql). ```sql {% macro generate_schema_name_for_env(custom_schema_name, node) -%} {%- set default_schema = target.schema -%} {%- if target.name == 'prod' and custom_schema_name is not none -%} {{ custom_schema_name | trim }} {%- else -%} {{ default_schema }} {%- endif -%} {%- endmacro %} ``` ##### You have a `generate_schema_name` macro with custom logic If this is the case — it might be a great idea to reach out to the person who added this macro to your project, as they will have context here — you can use [GitHub's blame feature](https://docs.github.com/en/free-pro-team@latest/github/managing-files-in-a-repository/tracking-changes-in-a-file) to do this. In all cases take a moment to read through the Jinja to see if you can follow the logic. #### Confirm your `schema` config Check if you are using the [`schema` config](https://docs.getdbt.com/reference/resource-properties/schema.md) in your model, either via a `{{ config() }}` block, or from `dbt_project.yml`. In both cases, dbt passes this value as the `custom_schema_name` parameter of the `generate_schema_name` macro. #### Confirm your target values Most `generate_schema_name` macros incorporate logic from the [`target` variable](https://docs.getdbt.com/reference/dbt-jinja-functions/target.md), in particular `target.schema` and `target.name`. Use the docs [here](https://docs.getdbt.com/reference/dbt-jinja-functions/target.md) to help you find the values of each key in this dictionary. #### Put the two together Now, re-read through the logic of your `generate_schema_name` macro, and mentally plug in your `customer_schema_name` and `target` values. You should find that the schema dbt is constructing for your model matches the output of your `generate_schema_name` macro. Be careful. Snapshots do not follow this behavior if target\_schema is set. To have environment-aware snapshots in v1.9+ or dbt, remove the [target\_schema config](https://docs.getdbt.com/reference/resource-configs/target_schema.md) from your snapshots. If you still want a custom schema for your snapshots, use the [`schema`](https://docs.getdbt.com/reference/resource-configs/schema.md) config instead. #### Prefixed schema names By default, dbt combines `target.schema` and `custom_schema_name` using the following pattern: `{target.schema}_{custom_schema_name}`. For example, when `target.schema` is `public` and a model sets `+schema: silver`, dbt builds the model in `public_silver`, not `silver`. This behavior is intentional. Including `target.schema` helps prevent developers and continuous integration (CI) jobs from building into the same schema and overwriting one another’s relations. If you want to use dedicated schema names such as `silver` and `gold` in production, use the environment-aware [`generate_schema_name_for_env` pattern](#you-have-a-generate_schema_name-macro-in-a-project-that-calls-another-macro) shown earlier. This pattern uses the custom schema name when `target.name` is `prod`, while retaining the target schema in development and CI environments. The macro looks like this: ```sql {% macro generate_schema_name_for_env(custom_schema_name, node) -%} {%- set default_schema = target.schema -%} {%- if target.name == 'prod' and custom_schema_name is not none -%} {{ custom_schema_name | trim }} {%- else -%} {{ default_schema }} {%- endif -%} {%- endmacro %} ``` Verify actual relation locations with `dbt ls --output json` or by querying your warehouse catalog (`pg_views`, `information_schema.tables`, or equivalent). #### Adjust as necessary Now that you understand how a model's schema is being generated, you can adjust as necessary: * You can adjust the logic in your `generate_schema_name` macro (or add this macro to your project if you don't yet have one and adjust from there) * You can also adjust your `target` details (for example, changing the name of a target) If you change the logic in `generate_schema_name`, it's important that you consider whether two users will end up writing to the same schema when developing dbt models. This consideration is the reason why the default implementation of the macro concatenates your target schema and custom schema together — we promise we were trying to be helpful by implementing this behavior, but acknowledge that the resulting schema name is unintuitive. --- ### Get hands-on with Snowflake Horizon and Apache Iceberg [Back to guides](https://docs.getdbt.com/guides.md) 60 minutes dbt v2 Snowflake Iceberg DuckDB Advanced [Menu ]() This guide shows you how to run dbt transformations across two engines (Snowflake and DuckDB) against a single, shared set of Apache Iceberg tables. You'll build your raw and staging layers as Snowflake-managed Iceberg tables, then use Snowflake Horizon's open Iceberg REST catalog to let DuckDB read and write those exact same tables directly. The result is a multi-engine workflow where each transformation runs on the engine that fits it best, your data is never copied or synced between systems, and your governance and lineage stay intact throughout. #### Access the same Iceberg tables from Snowflake and DuckDB By the end of this guide, your `fusion-jaffle-shop` project will transform data in two different engines against one shared set of tables: * **Snowflake:** Builds your raw and staging layers as Snowflake-managed Apache Iceberg tables, registered in Snowflake's built-in Horizon catalog. * **DuckDB:** Reads those exact same Iceberg tables through Horizon's open Iceberg REST catalog, builds a downstream model (`orders`), and writes the result *back* as an Iceberg table in the same catalog. The DuckDB step is the interesting part. Because Iceberg is an open table format and Horizon speaks the open Iceberg REST protocol, an external engine like DuckDB can operate directly on your governed Snowflake tables. Both solutions are reading and writing the same files, through the same catalog, without a separate copy of the data. Concretely, the transformation runs in a local DuckDB process (embedded in dbt v2); Horizon serves catalog metadata and vends short-lived storage credentials so DuckDB can access the underlying files, which live in your own S3 bucket the whole time. Snowflake warehouses are highly performant. Even a single-node XSMALL handles most production transformations well, and for the majority of your workloads, running them in Snowflake is still the right call. But there are times when it's useful to reach the same governed tables from somewhere else: a quick exploratory query from a laptop, a step that fits naturally into a pipeline already running on another engine, or a workload where you'd rather not spin up a warehouse at all. Horizon's Iceberg REST catalog is what makes that possible without duplicating data or losing governance. Why that's a big deal: * **One governed copy, many engines**: There's a single physical copy of the data, in open Parquet + Iceberg metadata, registered in Horizon. Snowflake, DuckDB, Spark, Trino, and others can all read and write it directly. No syncing, no ETL between systems. * **Governance stays put**: The tables are still first-class Snowflake objects. They show up in the catalog, honor grants, and can carry masking/row policies and lineage — even the ones DuckDB wrote. * **Flexibility to match engine to job**: Because Horizon exposes the same tables everywhere, you can route each transformation to whichever engine suits it best — including running a step locally in DuckDB — without duplicating data or re-architecting your pipeline. This is the foundation of cross-platform [dbt Mesh](https://docs.getdbt.com/docs/mesh/cross-platform-mesh): one project, one catalog, many engines. #### Notes about this guide Before you begin, there are a few considerations: * This is an advanced guide and assumes that you have fundamental knowledge of dbt, Snowflake, DuckDB, related tools, and how to install them. * While dbt handles SQL transformations with grace, some of the tools used in this guide are very specific about the syntax they accept. We highly recommend you remove code comments from examples in this guide before using them in a live environment. * The demo project used is a feature rich example of the existing Jaffle Shop project. This is to demonstrate some of the considerations you'll need to make in your own projects as you implement these workflows. To ensure a smooth outcome, we recommend you use the project link in this guide and not any other existing Jaffle Shop projects. * If you are using a Snowflake account that has not been configured for Python (for example, a brand new trial account), you may run into errors with the Python models in the dbt project. Deleting them for the duration of this guide will remove those errors (though there will be some non-blocking warnings). #### Prerequisites There are a number of prerequisites and personas you'll need to complete this guide. ##### Accounts and access To complete the required setup steps, you'll need: * A **Snowflake account** where you can act as `ACCOUNTADMIN`. This guide builds everything from scratch, so a brand-new account (a trial works) is exactly the assumed starting point. * An **AWS account** where you can create an S3 bucket and an IAM role + policy. * A local clone of the [**`fusion-jaffle-shop`** project](https://github.com/matthewshaver/fusion-jaffle-shop), with the seed CSVs present in `seeds/`. * Comfort running SQL in Snowsight and basic commands in a terminal. ##### Tools to install and verify Install and verify these before you start: | Requirement | Why | Check | | ------------------------------------ | --------------------------------------------------------- | ----------------------------- | | **dbt v2** (v2, `preview.194`+) | `catalogs.yml` v2 + DuckDB attach support | `dbt --version` | | **DuckDB 1.5.4+** | Iceberg write-compat (`ATTACH`, credential vending) | `duckdb --version` | | **An AWS account** | Hosts the S3 bucket that stores the Iceberg data/metadata | — | | **A new Snowflake account** | Where Horizon + the managed Iceberg tables live | — | | **The `fusion-jaffle-shop` project** | Cloned locally, with the seed CSVs present in `seeds/` | `ls seeds/` shows `raw_*.csv` | ##### Placeholders used in this guide Throughout this guide, replace these placeholders with your own values: * `<ACCOUNT_IDENTIFIER>`: Your Snowflake account identifier in `ORG-ACCOUNT` form (for example, `ABCDEFG-HI12345`). Find it in Snowsight under your account menu → **Account** → **View account details**, or in the account URL `https://<ORG>-<ACCOUNT>.snowflakecomputing.com`. * `<AWS_ACCOUNT_ID>`: Your 12-digit AWS account number. * `<YOUR_USER>`: Your Snowflake login username. * `<REGION>`: The AWS region for your bucket. This guide uses `us-east-2`; use whatever is closest to your Snowflake account's region. This guide is for v2 only. The `catalogs.yml` mechanism it relies on is not available in the legacy Python `dbt-duckdb` adapter. #### The project This guide will use a heavily augmented copy of the traditional Jaffle Shop project. The project itself contains more information than required to simply setup an Iceberg workflow, so you can use it to run more trials and observe results. Clone the [`fusion-jaffle-shop` project](https://github.com/matthewshaver/fusion-jaffle-shop) from GitHub: ```bash git clone https://github.com/matthewshaver/fusion-jaffle-shop.git ``` Then navigate into the project directory: ```bash cd fusion-jaffle-shop ``` #### Part 1: Set up the Snowflake account Log into Snowsight (the Snowflake web UI) with your admin user. Open a new SQL worksheet and run each block below. ##### 1.1 Create the warehouse, database, and schema ```sql USE ROLE ACCOUNTADMIN; CREATE WAREHOUSE IF NOT EXISTS COMPUTE_WH WAREHOUSE_SIZE = 'XSMALL' AUTO_SUSPEND = 60 AUTO_RESUME = TRUE; CREATE DATABASE IF NOT EXISTS DBT_ICEBERG; CREATE SCHEMA IF NOT EXISTS DBT_ICEBERG.RAW; ``` ##### 1.2 Create the `TRANSFORMER` role and grant it everything it needs You'll use one role, `TRANSFORMER`, for everything. Both the Snowflake-side build and the DuckDB offload. Because `TRANSFORMER` will *own* every table it creates, it automatically has read/write on them and you won't need extra `SELECT` grants later. ```sql USE ROLE ACCOUNTADMIN; CREATE ROLE IF NOT EXISTS TRANSFORMER; -- Let TRANSFORMER use compute and the database objects GRANT USAGE ON WAREHOUSE COMPUTE_WH TO ROLE TRANSFORMER; GRANT USAGE ON DATABASE DBT_ICEBERG TO ROLE TRANSFORMER; GRANT USAGE ON SCHEMA DBT_ICEBERG.RAW TO ROLE TRANSFORMER; -- Let TRANSFORMER create the tables dbt will build (seeds + Iceberg models) GRANT CREATE TABLE ON SCHEMA DBT_ICEBERG.RAW TO ROLE TRANSFORMER; GRANT CREATE ICEBERG TABLE ON SCHEMA DBT_ICEBERG.RAW TO ROLE TRANSFORMER; GRANT CREATE VIEW ON SCHEMA DBT_ICEBERG.RAW TO ROLE TRANSFORMER; -- Give the role to yourself so you can use it, and make it your default GRANT ROLE TRANSFORMER TO USER <YOUR_USER>; ALTER USER <YOUR_USER> SET DEFAULT_ROLE = TRANSFORMER; ``` We'll grant `TRANSFORMER` access to the external volume in Part 3, after it exists. #### Part 2: Create the S3 bucket and IAM role in AWS Snowflake-managed Iceberg tables store their data and metadata files in object storage that you own. Snowflake reaches that bucket by assuming an IAM role you create. This is a two-way handshake, so the order matters. ##### 2.1 Create the S3 bucket In your AWS console 1. Navigate to **S3 → Create bucket**. 2. **Bucket name**: pick a globally unique name. A good convention is `iceberg-<AWS_ACCOUNT_ID>-<REGION>`, like `iceberg-1234567890-us-east-2`. 3. **Region**: choose `<REGION>` (for example, `us-east-2`). 4. Leave the rest as defaults and click **Create bucket**. 5. (Optional but tidy) open the bucket and create a folder/prefix named `jaffle-iceberg/` for all Iceberg files to live under. ##### 2.2 Create the IAM permission policy In your AWS console: 1. Navigate to **IAM → Policies → Create policy** and switch to the **JSON** tab. 2. Paste the following, replacing the bucket name in both `Resource` blocks: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": ["s3:PutObject", "s3:GetObject", "s3:GetObjectVersion", "s3:DeleteObject", "s3:DeleteObjectVersion"], "Resource": "arn:aws:s3:::iceberg-<AWS_ACCOUNT_ID>-<REGION>/jaffle-iceberg/*" }, { "Effect": "Allow", "Action": ["s3:ListBucket", "s3:GetBucketLocation"], "Resource": "arn:aws:s3:::iceberg-<AWS_ACCOUNT_ID>-<REGION>", "Condition": { "StringLike": { "s3:prefix": ["jaffle-iceberg/*"] } } } ] } ``` 3. Click **Next**, name it `snowflake-iceberg-policy`, and **Create policy**. ##### 2.3 Create the IAM role (with a placeholder trust) You can't fill in the real trust policy yet as it needs values Snowflake generates in Part 3. So create the role with a temporary trust, then you'll fix it later. In the AWS console: 1. Navigate to **IAM → Roles → Create role**. 2. **Trusted entity type**: choose **AWS account**. 3. Select **This account (`<AWS_ACCOUNT_ID>`)**. Leave both **"Require external ID"** and **"Require MFA"** **unchecked**. (This is a throwaway trust; you'll replace it in Part 3.) 4. Click **Next**. On the permissions page, search for and check **`snowflake-iceberg-policy`**. 5. Click **Next**, name the role **`snowflake-iceberg-role`**, and **Create role**. 6. Open the new role and copy its **ARN** (`arn:aws:iam::<AWS_ACCOUNT_ID>:role/snowflake-iceberg-role`) — you'll need it next. #### Part 3: Connect Snowflake to S3 with an external volume An external volume is Snowflake's secure link to the S3 bucket where your Iceberg data and metadata live. In this part, you'll create the volume, complete the trust handshake between Snowflake and your IAM role, and set the volume as your database default so every Iceberg table knows where to store its files. ##### 3.1 Create the external volume Back in Snowsight (as `ACCOUNTADMIN`), copy and paste the following (replace `<PLACEHOLDERS>` with your info): ```sql USE ROLE ACCOUNTADMIN; CREATE OR REPLACE EXTERNAL VOLUME ICEBERG_EXT_VOL STORAGE_LOCATIONS = (( NAME = 'iceberg-s3-<REGION>' STORAGE_PROVIDER = 'S3' STORAGE_BASE_URL = 's3://iceberg-<AWS_ACCOUNT_ID>-<REGION>/jaffle-iceberg/' STORAGE_AWS_ROLE_ARN = 'arn:aws:iam::<AWS_ACCOUNT_ID>:role/snowflake-iceberg-role' )) ALLOW_WRITES = TRUE; ``` ##### 3.2 Get Snowflake's identity and finish the IAM trust ```sql DESCRIBE EXTERNAL VOLUME ICEBERG_EXT_VOL; ``` In the output, find the `STORAGE_LOCATION_1` row and read two values out of its JSON: * `STORAGE_AWS_IAM_USER_ARN` (looks like `arn:aws:iam::123456789012:user/abc1-s`) * `STORAGE_AWS_EXTERNAL_ID` (looks like `ABC12345_SFCRole=...=`) Now, go back to the AWS console **IAM → Roles → `snowflake-iceberg-role` → Trust relationships → Edit trust policy**, delete what's there, and paste this (with your two values): ```json { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": { "AWS": "<STORAGE_AWS_IAM_USER_ARN>" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "<STORAGE_AWS_EXTERNAL_ID>" } } }] } ``` Click **Update policy**. Copy the external ID *exactly*, including any trailing `=` and internal `/`. It's case-sensitive. ##### 3.3 Grant the volume and set it as the database default ```sql USE ROLE ACCOUNTADMIN; -- TRANSFORMER needs USAGE on the volume (required for credential vending) GRANT USAGE ON EXTERNAL VOLUME ICEBERG_EXT_VOL TO ROLE TRANSFORMER; -- Make this volume the default for Iceberg tables in the database. -- REQUIRED: when DuckDB creates a table through the REST catalog it can't -- name a volume, so Snowflake falls back to this default. ALTER DATABASE DBT_ICEBERG SET EXTERNAL_VOLUME = ICEBERG_EXT_VOL; ALTER DATABASE DBT_ICEBERG SET CATALOG = 'SNOWFLAKE'; ``` #### Part 4: Create a Programmatic Access Token for DuckDB DuckDB authenticates to Horizon's REST catalog over OAuth2, using a Programmatic Access Token (PAT) as its credential. To add the PAT in Snowsight: ```sql USE ROLE ACCOUNTADMIN; ALTER USER <YOUR_USER> ADD PROGRAMMATIC ACCESS TOKEN duckdb_pat ROLE_RESTRICTION = 'TRANSFORMER' DAYS_TO_EXPIRY = 90; ``` The result grid has two columns. Copy the long `token_secret` value (not `token_name`). You only see it once; save it somewhere safe for Part 6. **Verify the token works:** Before wiring it into dbt, in a terminal, run the OAuth2 exchange (this is exactly what DuckDB does under the hood). Note the **empty username** before the colon in `-u ":..."` — this is essential (see FAQ): ```bash curl -s "https://<ACCOUNT_IDENTIFIER>.snowflakecomputing.com/polaris/api/catalog/v1/oauth/tokens" \ -u ":<TOKEN_SECRET>" \ --data-urlencode 'grant_type=client_credentials' \ --data-urlencode 'scope=session:role:TRANSFORMER' ``` A JSON response containing `"access_token"` means you're good. If you see `Programmatic access token is invalid`, re-copy the token. **If you see an error mentioning a network policy, see the [FAQ](https://docs.getdbt.com/guides/iceberg.md?step=12#faq)**. Some Snowflake accounts require a network policy before a PAT can be created or used. #### Part 5: Load the seed data In this part, you'll load the dbt project's raw CSV seed data into Snowflake as regular tables. You'll run the seed against the prod target so you get the full historical dataset, rather than dev, which filters orders to the last year. ##### 5.1 Seed commands From the project root in your dbt CLI, install packages and seed the raw tables into Snowflake. (Make sure the seed CSVs are present in `seeds/` first.) ```bash dbt deps ``` ```bash dbt seed --target prod ``` ##### 5.2 First seed The project's `on-run-start` hook (`insert_freshness_heartbeat()`) queries a raw source table that doesn't exist until the seeds load, so if the *first* `dbt seed` fails with `Object 'DBT_ICEBERG.RAW.RAW_STORES' does not exist`. Before your first seed, comment out the `on-run-start` block in `dbt_project.yml`: ```yaml # on-run-start: # - "{{ insert_freshness_heartbeat() }}" ``` Uncomment it once the seeds exist. This loads the following into `DBT_ICEBERG.RAW` as regular Snowflake tables: * `raw_customers` * `raw_orders` * `raw_items` * `raw_products` * `raw_stores` * `raw_supplies` * `raw_tweets` We use the `prod` target — you'll define it in the next section. Why `prod` and not `dev`? This project's `stg_orders` model applies a `limit_in_dev` macro that filters orders to the last year **whenever the target is named `dev`**. The seed data is historical, so building on `dev` yields an **empty** `stg_orders` (and therefore empty `orders`). Building on any non-`dev` target skips that filter and gives you the full dataset. See the [FAQ](https://docs.getdbt.com/guides/iceberg.md?step=12#faq) for more information. #### Part 6: Configure the project ##### 6.1 `profiles.yml` Create/edit `~/.dbt/profiles.yml`. Define three targets: `dev` and `prod` (both Snowflake) and `duckdb` (the offload). Paste your PAT into `client_secret`. *You can, optionally, run `dbt init` to generate the `profiles.yml` in the proper location and configure the Snowflake profile.* If you already have a `~/.dbt/profiles.yml`, add the `jaffle_shop` profile below **before** running any commands. On an existing config, `dbt init` may not surface a `prod` target and later steps will fail to find it. ```yaml jaffle_shop: target: prod outputs: # Everyday Snowflake target. NOTE: stg_orders is limited to the last year here. dev: type: snowflake account: "<ACCOUNT_IDENTIFIER>" user: "<YOUR_USER>" authenticator: externalbrowser # or your preferred auth role: TRANSFORMER database: DBT_ICEBERG schema: RAW warehouse: COMPUTE_WH threads: 8 # Full-data Snowflake target used to build the Iceberg parents. prod: type: snowflake account: "<ACCOUNT_IDENTIFIER>" user: "<YOUR_USER>" authenticator: externalbrowser role: TRANSFORMER database: DBT_ICEBERG schema: RAW warehouse: COMPUTE_WH threads: 8 # DuckDB target reads/writes Iceberg via Horizon with ZERO Snowflake compute. duckdb: type: duckdb path: ':memory:' schema: RAW secrets: - type: iceberg name: horizon_secret client_id: "" # MUST be empty (see FAQ) client_secret: "<TOKEN_SECRET>" # your PAT from Part 4 oauth2_server_uri: "https://<ACCOUNT_IDENTIFIER>.snowflakecomputing.com/polaris/api/catalog/v1/oauth/tokens" oauth2_scope: "session:role:TRANSFORMER" ``` The v2 project's `dbt_project.yml` sets `profile: default`. Either rename the profile key above to `default`, or set `profile: jaffle_shop`. Keep it consistent. If `externalbrowser` fails For example `390190 (08004) ... SAML Identity Provider account parameter`, swap it for password auth on the `dev` and `prod` targets and keep the secret out of the file with an environment variable: ```yaml # authenticator: externalbrowser password: "{{ env_var('SNOWFLAKE_PASSWORD') }}" ``` ##### 6.2 `catalogs.yml` Create `catalogs.yml` in the project root. This one entry describes the *same* Horizon catalog for both engines. ```yaml catalogs: - name: horizon_catalog type: horizon table_format: iceberg config: snowflake: external_volume: ICEBERG_EXT_VOL duckdb: warehouse: DBT_ICEBERG # Horizon REST "warehouse" = the Snowflake database name endpoint: "https://<ACCOUNT_IDENTIFIER>.snowflakecomputing.com/polaris/api/catalog" secret: horizon_secret # references the profiles.yml secret above default_schema: RAW access_delegation_mode: VENDED_CREDENTIALS # Horizon vends temp S3 creds; no AWS keys needed ``` ##### 6.3 `dbt_project.yml` Add the behavior flag that enables `catalogs.yml`, and remove the project-level `marts` grants (they don't work on DuckDB — see 6.4 and the FAQ). Add near the top: ```yaml flags: use_catalogs_v2: true ``` Then, under `models: jaffle_shop: marts:`, **delete** the `+grants` block: ```yaml # DELETE these two lines from the marts config: +grants: select: ["ACCOUNTADMIN"] ``` ##### 6.4 Model changes These edits make the three models materialize as Iceberg and behave on both engines. * **`models/staging/stg_orders.sql`**: Add the config header and cast the timestamp to microsecond precision (Iceberg rejects Snowflake's default nanosecond scale): ```sql {{ config(materialized='table', catalog_name='horizon_catalog', alias='STG_ORDERS') }} ``` ...and change the timestamp line to: ```sql cast({{ dbt.date_trunc('day','ordered_at') }} as timestamp_ntz(6)) as ordered_at ``` * **`models/marts/order_items.sql`**: add at the very top: ```sql {{ config(materialized='table', catalog_name='horizon_catalog', alias='ORDER_ITEMS') }} ``` * **`models/marts/orders.sql`**: replace its config header with: ```sql {{ config( materialized='table', catalog_name='horizon_catalog', alias='ORDERS', grants={} if target.name == 'duckdb' else {'select': ['ACCOUNTADMIN']}, persist_docs={'relation': false, 'columns': false} if target.name == 'duckdb' else {'relation': true, 'columns': true} ) }} ``` * **`models/marts/orders.yml`**: Turn off the enforced contract (a single contract can't hold both Snowflake and DuckDB type names): ```yaml contract: enforced: false ``` * **`macros/insert_freshness_heartbeat.sql`**: Exclude the `duckdb` target (the heartbeat writes to the raw Snowflake source, which isn't attached in a DuckDB session): ```jinja {% if target.name not in ('ci', 'dev', 'duckdb') %} ``` * **Why uppercase `alias`?** Snowflake stores unquoted identifiers in UPPERCASE, and Iceberg catalogs are case-sensitive. DuckDB quotes model names in lowercase, so without the uppercase alias it would look for `"stg_orders"` and miss `STG_ORDERS`. #### Part 7: Build it This is where it all comes together! Snowflake builds the raw and staging Iceberg parents, then DuckDB reads those same tables through Horizon to build and write back the final orders model. All of this against the one shared catalog. ##### 7.1 Snowflake pass: Build the Iceberg parents ```bash dbt run --target prod -s +orders --exclude orders ``` This builds `stg_orders` and `order_items` as Snowflake-managed Iceberg tables in `DBT_ICEBERG.RAW`, registered in Horizon. Because you run as `TRANSFORMER`, it owns those tables. ##### 7.2 DuckDB pass: Read and write through Horizon ```bash dbt run --target duckdb -s orders ``` DuckDB attaches Horizon, reads `STG_ORDERS` + `ORDER_ITEMS` (fetching temp S3 credentials from Horizon), builds `orders` locally, and commits it back as an Iceberg table through Horizon's REST catalog without involving a Snowflake warehouse. ##### 7.3 Verify ```bash dbt show --target duckdb --inline "select count(*) from {{ ref('orders') }}" ``` You should get a real row count (tens of thousands). You'll also see `ORDERS` listed as an **Iceberg table** in Snowsight under `DBT_ICEBERG.RAW` and written entirely by DuckDB. **Confirm the cross-engine write:** in Snowsight, open **Admin → Cost Management** (or query `snowflake.account_usage.warehouse_metering_history`). You'll see credits for the `prod` build in 7.1, and none for the DuckDB run in 7.2 — confirming that DuckDB read and wrote the Iceberg tables directly through Horizon's REST catalog, rather than through a Snowflake warehouse. #### Congratulations! You've just built a single set of Apache Iceberg tables and transformed them with two different engines! Snowflake for the raw and staging layers, DuckDB for the final orders model, and all against one shared Horizon catalog. Because Horizon speaks the open Iceberg REST protocol, DuckDB was able to read and write those governed Snowflake tables directly. No data copies, no lock-in, and governance and lineage that stay intact no matter which engine does the work. This is the foundation of cross-platform dbt Mesh: one project, one catalog, and the freedom to route every job to whichever engine fits it best. #### FAQ  OAuth token exchange returns \`invalid\_scope\` for every role, even \`PUBLIC\`. The PAT must be sent with an **empty `client_id`** — as the HTTP Basic *password* with no username (`curl -u ":<PAT>"`), which is what `client_id: ""` produces in the DuckDB secret. If you pass a real username as `client_id`, Snowflake tries to resolve it as a registered OAuth client and rejects the scope. Empty client\_id is the fix.  The exchange returns \`Programmatic access token is invalid\`. The token value is wrong — you likely copied `token_name` instead of the long `token_secret`, or truncated it. Re-create the PAT and copy the `token_secret` carefully.  PAT creation or use fails complaining about a network policy. Some Snowflake accounts require a network policy to be in effect before a PAT can be created or used. If you hit this, create a network policy scoped to your egress IP range and attach it to your user, then retry. (Intentionally out of scope for this guide's main flow.)  DuckDB gets \`401 Unauthorized\` from the catalog. Auth problem. Check: the PAT is valid (test with the curl in Part 4); `client_id` is `""`; `oauth2_scope` is `session:role:TRANSFORMER` and your user actually holds `TRANSFORMER`; and the account in `oauth2_server_uri`/`endpoint` matches the account the PAT was minted in.  DuckDB gets \`404 Not Found\` on \`/v1/config\`. The `warehouse` value in `catalogs.yml` is wrong. For Horizon it's the **Snowflake database name** (`DBT_ICEBERG`), not the `catalogs.yml` entry name.  The catalog returns the namespace but lists no tables, or DuckDB can't find a table it should see. Two common causes: (1) a role lacks visibility — but if you built as `TRANSFORMER` (this guide) it owns the tables and sees them. If you built as a *different* role, note that `GRANT SELECT ON ALL TABLES` does **not** cover Iceberg tables; you must `GRANT SELECT ON ALL ICEBERG TABLES IN SCHEMA DBT_ICEBERG.RAW TO ROLE TRANSFORMER` (and `FUTURE ICEBERG TABLES`). (2) Case mismatch — see the uppercase-alias note below.  \`Catalog Error: Table with name raw.stg\_orders does not exist\`. Case sensitivity. Snowflake stores `STG_ORDERS`; DuckDB quotes `"stg_orders"`. Give the Iceberg models an uppercase `alias` (`alias='STG_ORDERS'`). For a whole project you'd standardize identifier casing rather than aliasing each model.  \`SQL compilation error: Invalid time type scale ... TIMESTAMP\_NTZ(9)\`. Iceberg only supports microsecond timestamps. Cast to `timestamp_ntz(6)`.  \`403 Forbidden\` reading a \`.metadata.json\` from S3, with \`region ''\` / \`No credentials provided\`. DuckDB needs storage credentials to read the data files. Set `access_delegation_mode: VENDED_CREDENTIALS` in the `duckdb` catalog block so Horizon vends temporary, region-scoped S3 credentials — no AWS keys required. If this occurs intermittently right after rebuilding a table, just re-run; a fresh process re-fetches current metadata and credentials.  \`403 Authorization failed\` on \`POST .../namespaces/RAW/tables\` (create). The database has no default external volume, so a REST `createTable` can't resolve where to write. Run `ALTER DATABASE DBT_ICEBERG SET EXTERNAL_VOLUME = ICEBERG_EXT_VOL;` (and `SET CATALOG = 'SNOWFLAKE';`). Also confirm `TRANSFORMER` has `CREATE ICEBERG TABLE` on the schema and `USAGE` on the volume.  \`500 ... Object 'ORDERS' already exists as TABLE\`. A regular (non-Iceberg) table of the same name already occupies that namespace (e.g. from an earlier full `dbt build`). Drop it: `DROP TABLE IF EXISTS DBT_ICEBERG.RAW.ORDERS;` and re-run so DuckDB can create the Iceberg version.  \`unknown method: map has no method named warn\_once\` during grants. The dbt-duckdb `apply_grants` macro hits a v2 incompatibility. Grants are meaningless on DuckDB anyway — make `grant_config` empty on the DuckDB target (`grants={} if target.name == 'duckdb' else {...}`) and remove the project-level `marts` `+grants` so nothing gets merged back in.  \`Not implemented Error: Only ALTER TABLE is supported for Iceberg\`. `persist_docs` emits `COMMENT` statements DuckDB's Iceberg writer doesn't support. Disable it on the DuckDB target (`persist_docs={'relation': false, 'columns': false} if target.name == 'duckdb' else {...}`).  \`Type with name number does not exist\` (DuckDB suggests \`numeric\`). An **enforced model contract** is generating a typed `CREATE TABLE` with Snowflake type names (`number`, `timestamp_ntz`) that DuckDB doesn't recognize. A single contract can't satisfy two engines, so set `contract: enforced: false` on models you build on DuckDB.  \`orders\` builds successfully but has 0 rows. `stg_orders` came out empty because you built it on the `dev` target, where `limit_in_dev` restricts orders to the last year and the seed data is older. Build the parents on `prod` (or any non-`dev` target).  \`TRANSFORMER\` can't run \`count(\*)\` / \`must specify a warehouse\`. Metadata commands (`show`, `describe`) don't need a warehouse, but queries do. Ensure `GRANT USAGE ON WAREHOUSE COMPUTE_WH TO ROLE TRANSFORMER`. To read row counts without any warehouse, read `summary.total-records` from the Iceberg table metadata via the REST `loadTable` endpoint.  How do I extend this beyond \`orders\`? Move the DuckDB-specific overrides (empty `grants`, disabled `persist_docs`) to a **target-aware `marts` config** in `dbt_project.yml` instead of per-model, standardize identifier casing project-wide, and add `catalog_name`/`alias` to each model you want materialized as Iceberg. Everything Snowflake writes, DuckDB can read, and vice-versa. --- ### Get started with Continuous Integration tests [Back to guides](https://docs.getdbt.com/guides.md) dbt platform Orchestration CI Intermediate [Menu ]() #### Introduction By validating your code *before* it goes into production, you don't need to spend your afternoon fielding messages from people whose reports are suddenly broken. A solid CI setup is critical to preventing avoidable downtime and broken trust. dbt uses **sensible defaults** to get you up and running in a performant and cost-effective way in minimal time. After that, there's time to get fancy, but let's walk before we run. In this guide, we're going to add a **CI environment**, where proposed changes can be validated in the context of the entire project without impacting production systems. We will use a single set of deployment credentials (like the Prod environment), but models are built in a separate location to avoid impacting others (like the Dev environment). Your git flow will look like this: [![git flow diagram](/img/best-practices/environment-setup/one-branch-git.png?v=2 "git flow diagram")](#)git flow diagram ##### Prerequisites As part of your initial dbt setup, you should already have Development and Production environments configured. Let's recap what each does: * Your **Development environment** powers the Studio IDE. Each user has individual credentials, and builds into an individual dev schema. Nothing you do here impacts any of your colleagues. * Your **Production environment** brings the canonical version of your project to life for downstream consumers. There is a single set of deployment credentials, and everything is built into your production schema(s). #### Create a new CI environment See [Create a new environment](https://docs.getdbt.com/docs/dbt-platform-environments.md#create-a-deployment-environment). The environment should be called **CI**. Just like your existing Production environment, it will be a Deployment-type environment. When setting a Schema in the **Deployment Credentials** area, remember that dbt will automatically generate a custom schema name for each PR to ensure that they don't interfere with your deployed models. This means you can safely set the same Schema name as your Production job. ##### 1. Double-check your Production environment is identified Go into your existing Production environment, and ensure that the **Set as Production environment** checkbox is set. It'll make things easier later. ##### 2. Create a new job in the CI environment Use the **Continuous Integration Job** template, and call the job **CI Check**. In the Execution Settings, your command will be preset to `dbt build --select state:modified+`. Let's break this down: * [`dbt build`](https://docs.getdbt.com/reference/commands/build.md) runs all nodes (seeds, models, snapshots, tests) at once in DAG order. If something fails, nodes that depend on it will be skipped. * The [`state:modified+` selector](https://docs.getdbt.com/reference/node-selection/methods.md#state) means that only modified nodes and their children will be run ("Slim CI"). In addition to [not wasting time](https://discourse.getdbt.com/t/how-we-sped-up-our-ci-runs-by-10x-using-slim-ci/2603) building and testing nodes that weren't changed in the first place, this significantly reduces compute costs. To be able to find modified nodes, dbt needs to have something to compare against. dbt uses the last successful run of any job in your Production environment as its [comparison state](https://docs.getdbt.com/reference/node-selection/syntax.md#about-node-selection). As long as you identified your Production environment in Step 2, you won't need to touch this. If you didn't, pick the right environment from the dropdown. If you point CI at a non-production environment (staging, QA, UAT, or similar) that runs many jobs, comparison manifests can change unpredictably, or lag behind merges to your integration branch (for example, `develop`). Use CI to test your metrics If you've [built semantic nodes](https://docs.getdbt.com/docs/build/build-metrics-intro.md) in your dbt project, you can [validate them in a CI job](https://docs.getdbt.com/docs/deploy/ci-jobs.md#semantic-validations-in-ci) to ensure code changes made to dbt models don't break these metrics. ##### 3. Test your process That's it! There are other steps you can take to be even more confident in your work, such as validating your structure follows best practices and linting your code. For more information, refer to [Get started with Continuous Integration tests](https://docs.getdbt.com/guides/set-up-ci.md). To test your new flow, create a new branch in the Studio IDE then add a new file or modify an existing one. Commit it, then create a new Pull Request (not a draft). Within a few seconds, you’ll see a new check appear in your git provider. ##### Things to keep in mind * If you make a new commit while a CI run based on older code is in progress, it will be automatically canceled and replaced with the fresh code. * An unlimited number of CI jobs can run at once. If 10 developers all commit code to different PRs at the same time, each person will get their own schema containing their changes. Once each PR is merged, dbt will drop that schema. * CI jobs will never block a production run. #### Keep comparison manifests stable Deferring to Production is straightforward when one primary deploy job owns artifacts and merges are infrequent. With high merge volume, Production’s manifest can still move while other pull requests stay open, so CI may select extra `state:modified` nodes until those branches incorporate the latest merges (or you add a fast [merge triggered](https://docs.getdbt.com/docs/deploy/merge-jobs.md) `dbt parse` in the deferred environment). Busy staging, QA, or UAT environments add another failure mode as many jobs with different settings overwrite `manifest.json`. Any deferred environment can also be stale for a short window right after a merge until the next successful run refreshes artifacts. #### Enforce best practices with dbt project evaluator dbt Project Evaluator is a package designed to identify deviations from best practices common to many dbt projects, including modeling, testing, documentation, structure and performance problems. For an introduction to the package, read its [launch blog post](https://docs.getdbt.com/blog/align-with-dbt-project-evaluator). ##### 1. Install the package As with all packages, add a reference to `dbt-labs/dbt_project_evaluator` to your `packages.yml` file. See the [dbt Package Hub](https://hub.getdbt.com/dbt-labs/dbt_project_evaluator/latest/) for full installation instructions. ##### 2. Define test severity with an environment variable As noted in the [documentation](https://dbt-labs.github.io/dbt-project-evaluator/latest/ci-check/), tests in the package are set to `warn` severity by default. To have these tests fail in CI, create a new environment called `DBT_PROJECT_EVALUATOR_SEVERITY`. Set the project-wide default to `warn`, and set it to `error` in the CI environment. In your `dbt_project.yml` file, override the severity configuration: ```yaml data_tests: dbt_project_evaluator: +severity: "{{ env_var('DBT_PROJECT_EVALUATOR_SEVERITY', 'warn') }}" ``` ##### 3. Update your CI commands Because these tests should only run after the rest of your project has been built, your existing CI command will need to be updated to exclude the dbt\_project\_evaluator package. You will then add a second step which builds *only* the package's models and tests. Update your steps to: ```bash dbt build --select state:modified+ --exclude package:dbt_project_evaluator dbt build --select package:dbt_project_evaluator ``` ##### 4. Apply any customizations Depending on the state of your project when you roll out the evaluator, you may need to skip some tests or allow exceptions for some areas. To do this, refer to the documentation on: * [disabling tests](https://dbt-labs.github.io/dbt-project-evaluator/latest/customization/customization/) * [excluding groups of models from a specific test](https://dbt-labs.github.io/dbt-project-evaluator/latest/customization/exceptions/) * [excluding packages or sources/models based on path](https://dbt-labs.github.io/dbt-project-evaluator/latest/customization/excluding-packages-and-paths/) If you create a seed to exclude groups of models from a specific test, remember to disable the default seed and include `dbt_project_evaluator_exceptions` in your second `dbt build` command above. #### Run linting checks with SQLFluff By [linting](https://docs.getdbt.com/docs/platform/studio-ide/lint-format.md#lint) your project during CI, you can ensure that code styling standards are consistently enforced, without spending human time nitpicking comma placement. Seamlessly enable [SQL linting for your CI job](https://docs.getdbt.com/docs/deploy/continuous-integration.md#sql-linting) in dbt to invoke [SQLFluff](https://docs.sqlfluff.com/en/stable/), a modular and configurable SQL linter that warns you of complex functions, syntax, formatting, and compilation errors. SQL linting in CI lints all the changed SQL files in your project (compared to the last deferred production state). Available on dbt [Starter, Enterprise, or Enterprise+ accounts](https://www.getdbt.com/pricing) using [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). ##### Manually set up SQL linting in CI You can run SQLFluff as part of your pipeline even if you don't have access to [SQL linting in CI](https://docs.getdbt.com/docs/deploy/continuous-integration.md#sql-linting). The following steps walk you through setting up a CI job using SQLFluff to scan your code for linting errors. If you're new to SQLFluff rules in dbt, check out [our recommended config file](https://docs.getdbt.com/best-practices/how-we-style/2-how-we-style-our-sql.md). ##### 1. Create a YAML file to define your pipeline The YAML files defined below are what tell your code hosting platform the steps to run. In this setup, you’re telling the platform to run a SQLFluff lint job every time a commit is pushed. ##### GitHub GitHub Actions are defined in the `.github/workflows` directory. To define the job for your action, add a new file named `lint_on_push.yml` under the `workflows` folder. Your final folder structure will look like this: ```sql my_awesome_project ├── .github │ ├── workflows │ │ └── lint_on_push.yml ``` **Key pieces:** * `on:` defines when the pipeline is run. This workflow will run whenever code is pushed to any branch except `main`. For other trigger options, check out [GitHub’s docs](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows). * `runs-on: ubuntu-latest` - this defines the operating system we’re using to run the job * `uses:` - When the Ubuntu server is created, it is completely empty. [`checkout`](https://github.com/actions/checkout#checkout-v3) and [`setup-python`](https://github.com/actions/setup-python#setup-python-v3) are public GitHub Actions which enable the server to access the code in your repo, and set up Python correctly. * `run:` - these steps are run at the command line, as though you typed them at a prompt yourself. This will install sqlfluff and lint the project. Be sure to set the correct `--dialect` for your project. For a full breakdown of the properties in a workflow file, see [Understanding the workflow file](https://docs.github.com/en/actions/learn-github-actions/understanding-github-actions#understanding-the-workflow-file) on GitHub's website. ```yaml name: lint dbt project on push on: push: branches-ignore: - 'main' jobs: # this job runs SQLFluff with a specific set of rules # note the dialect is set to Snowflake, so make that specific to your setup # details on linter rules: https://docs.sqlfluff.com/en/stable/rules.html lint_project: name: Run SQLFluff linter runs-on: ubuntu-latest steps: - uses: "actions/checkout@v3" - uses: "actions/setup-python@v4" with: python-version: "3.9" - name: Install SQLFluff run: "python -m pip install sqlfluff" - name: Lint project run: "sqlfluff lint models --dialect snowflake" ``` ##### GitLab Create a `.gitlab-ci.yml` file in your **root directory** to define the triggers for when to execute the script below. You’ll put the code below into this file. ```sql my_awesome_project ├── dbt_project.yml ├── .gitlab-ci.yml ``` **Key pieces:** * `image: python:3.9` - this defines the virtual image we’re using to run the job * `rules:` - defines when the pipeline is run. This workflow will run whenever code is pushed to any branch except `main`. For other rules, refer to [GitLab’s documentation](https://docs.gitlab.com/ee/ci/yaml/#rules). * `script:` - this is how we’re telling the GitLab runner to execute the Python script we defined above. ```yaml image: python:3.9 stages: - pre-build # this job runs SQLFluff with a specific set of rules # note the dialect is set to Snowflake, so make that specific to your setup # details on linter rules: https://docs.sqlfluff.com/en/stable/rules.html lint-project: stage: pre-build rules: - if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH != 'main' script: - python -m pip install sqlfluff - sqlfluff lint models --dialect snowflake ``` ##### Bitbucket Create a `bitbucket-pipelines.yml` file in your **root directory** to define the triggers for when to execute the script below. You’ll put the code below into this file. ```sql my_awesome_project ├── bitbucket-pipelines.yml ├── dbt_project.yml ``` **Key pieces:** * `image: python:3.11.1` - this defines the virtual image we’re using to run the job * `'**':` - this is used to filter when the pipeline runs. In this case we’re telling it to run on every push event, and you can see at line 12 we're creating a dummy pipeline for `main`. More information on filtering when a pipeline is run can be found in [Bitbucket's documentation](https://support.atlassian.com/bitbucket-cloud/docs/pipeline-triggers/) * `script:` - this is how we’re telling the Bitbucket runner to execute the Python script we defined above. ```yaml image: python:3.11.1 pipelines: branches: '**': # this sets a wildcard to run on every branch - step: name: Lint dbt project script: - python -m pip install sqlfluff==0.13.1 - sqlfluff lint models --dialect snowflake --rules L019,L020,L021,L022 'main': # override if your default branch doesn't run on a branch named "main" - step: script: - python --version ``` ##### 2. Commit and push your changes to make sure everything works After you finish creating the YAML files, commit and push your code to trigger your pipeline for the first time. If everything goes well, you should see the pipeline in your code platform. When you click into the job you’ll get a log showing that SQLFluff was run. If your code failed linting you’ll get an error in the job with a description of what needs to be fixed. If everything passed the lint check, you’ll see a successful job run. ##### GitHub In your repository, click the *Actions* tab ![Image showing the GitHub action for lint on push](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAR8AAAC9CAIAAAAFlu5LAAAQp0lEQVR4Aeycg5MkSxDGz7Zt27Zt27Zt27Zt27bvv3m/3e+9iomZN9M755vNiC82srKrq7o76jeZld13ceImSGr6GTKZjC6TyegymYwuk8lkdJlMRpfJZHSZTKaIp8tkMrpMJqPLZDIZXSaT0WUyGV3JUqVHsmvWazJk5ISESVJix0IlTpaGRxE/UfJfMBezMBcz2gKNWLqy5cr/9O1XlDZD1qQp0j558wV74PCxsfNJ3X36ltvv2W/IL5hr4LCxzHX78WtboBFLV/bcBRxdNC/efPj49ecqNep5Dnr60i3WYtdeA7AjRvefv/t1dA2PZXQZXSiGedGdJ284a/iYyUZXjGR0GV2Xbz8iItVp0Az76t2n2F16Drhw/T4Z48OXH9ds2gV7BYuWxq+zHr36hF24eNnQV5A+U45te4+wkhjnyu3HC5atlT/4LN6Eb951kBOnzl60dc9hzkJ7j5zOU6CYgz/6wsqoOWrCNJo79h9Ts32X3szLdATqS7ceVq/d0Jeu0ROnE5k5ev/5+/Xb9vrN26RFO4Zi/LQZs8lTr3FLedhHuas6cPw8g/N8uNnM2fMcO3tFd7dj3zHuzuiKjXSxGrBbte+KzeLA1k7MqWGzNrCkbjqKXaxU+RBTpEid4dq9p45GGaxCDoWYxfOuTl64EXjijfvPVSdQs3T5quo8b+kammev3sVmcB3lqoQTyleouOgKHHPWguV+RSB1GDB0jDwQSJOkOthVAZ6vE96MrigZXfzGp0qXmZhw+/Ermpt2Hgg3M5w+d4nWlta6VhWqUKWW1yzedKHeA4YTDfir5tBRE0PTxeDYx85d1aGNO/YzVNtOPUSXVnymbLmp9xBmaRJU/aY+dPIC/kOnLrpHoYjnrooYlTt/kZRpMjKUu3cu8tSlm27AgUaX0dVvyCj1XLZmM83r956FS9eR05fpuWXXIecRQmMmzfSaxZsu1qtvoQUPtISmSzYrHsw6dusD1X77rskzF6g5eeZ8mjhdB5dY4ufKAaZA4ZKaS4miroqRHbo0uTA1x02ZRfPWo1dGV5SMrsYt2qknv81KvcKlSyxNmbXQDwx485rFm651W/c4zwYyNBC6cic0XWnSZ7l44wFNp92HTimf9KtqdOs9MJqu94GvqnTZ7LhAEePM5du+V8VvhJprt+ymye+LmsPGTPqRdJmMLoWUtZt3OQ+hCQ8Z4/fTpfRMOnrmCp6dB46LLq1+HVq5fpujSypTodqMeUsdZvOWrI45XYjihG7q3LW7GLy8+mF0mYwu7SiWrd3iOf2ilRvo+eDFB03B5yBu6X8/XSR4cCJaVDaYMG2uu5EV67Zi85Zc8VN0TZg6Z9XGHS3adPYdZ9fBE550UdicNGM+o2HXbdRCh3QNVG5+GF0mo0u1AVXeVHALJmoV7PJVMKSwJgbY2fO91ffTJQl1xERZc+RVwHSHAFu26KLiLyT2HT1DTqhDvBYPTZfbX1Hc11GhhYiZNI2uULIvoWgKgxZtO7t174rjrCrfekOJMpVgwG97E0ylylVR/Q0xLMtRP/Zes3jTBeQQq5FvPnhBYNTRnHkLKV6pIL5971FtyVRS5+WbIBfz46fO9v0Sqkfff+kCOVfVoABIT/dD4Ch1ZEonzl/Hs2TVRjVJHX3TV+qZji4K+tF0Rdkm+0b+f0T8SZQsNVkW70+DSZmbvpH1fO/s5DmmixJ6W00NPXAQ4liwuKpyH6E7rG98VRiUDp44r2jJE7BFZnT9LO0/dpZ1FkyUoX/GmL50/WINHjHeBUbFPZPR9bPE2yo+Rwomvqv6GWPyl5yNV1K//oFSLFHGqNcMJqPLZDL96XSZTCajy2Qyukwmo8tkMhldJpPRlS5T9tIVqv/4kU0mo6tKzQYLVob7OthkMrqMLpPRlbdQiWnzli9bv3PGwlXFy1TGkz5zjiVrt7fv1m/puh1zlqyrXKO+ejZq0SF3/qLuRNezeduu9Fy8emurDj3kn79iY8GiZWRPmLGwWp3GGMXLVJq3bMPyDbtmLVqTM29hR9egkZOYnVMKFSsbOc/dZHQlTZEOKoaPn1G4RLm+Q8awytkLZcqaGwagokjJCoNHT6aDPnvF2WvgSGwn9YSZEmWrNG7ZAbtMxRr4GadoqQrqA5/1mrTBWLRqS7e+Q7PlKjBy4qyJMxeLLk5hXsCbOncZPSPnuZuMrhJlK0NCgsQpseMlTAYANeo1FTPZcxdU4QE7a8782Fly5EucPE0gXcVKVVRz1KRZ/YeND0YXzl4DRyVPlcE3M8TJvNjlq9RmKAyTKULoatyy49yl611z8uwlnXsNEjMOJOy8BYu7PoF0pUyTSc2OPQZMmbM0GF3V6zQhjaQ/iWiBIqX89l3knEaXKaLoImKwZXJNcjxICJeuPP8dHTRq0pDRUzAYs2K1unIuWLlJdCk8whUMA5XRZYpwulKny0Kcad2xJ/8EsG7jVqxv9kXB6OrSaxA7MT+6tENLnjoDZQyGYhD8s5esHTFhJv8FRbnKtegAXcQ3gljJclU52qpjT1JQoyvyZTVDAGCts7JJ22rVb4bHjy5FJ6oaRKQO3fsH0jVg+Hj+ooEjJ8pfvmptSMND2gk/dRu3Vt6IBz8Tlf+HvXtAqgUA4zg6fH7Ztm03zFpKi2kz7a5/tl1n5tS1v981phav1NXQ8uXqQl3HcuPzkJ+curLOWYe//hZf+QB87gT+L664vg/HD9LuAl5NvnIr95JAXbktKquqzxJQF6AuUBeoC1AXqAtQF6gL1AWoC9QF6gLUBeoCdQHqAnWBugB1gbpAXYC6QF2gLkBdoC5QF6AuUBeoC1AXqAvU9Q2AukBdgLpAXaAuQF2gLlAXoC5QF6gLUBeoC9QFqAvUBeoC1AXqAnUB6gJ1gboAdYG6QF2Vtc3dA+OD43NDE/NfHmTUM/AZ+9et6+ffohxM9+BEe89wc0f/NwEZ+Ix9hj8JvFZd2feu/rHvef5Chj8JvEpduWVMvt/5zIUkkBBevq5U6w4h7iImhJeva3B8zpkLCeHl6xqamHfOQkJQ1zOBukBdoC51gbpAXaAudYG6QF3d6xNDu4uTe8uL+zuRJVn53JMH6kpIKepGT28M1HU1rScF1tE70tU/dqylc+CAnbPQjWNZAui/hJkTMzNvzMwvzMycmJmZmcPMzAx/845uXZVWmSwovPaVSlF3dU1vz6TOFNjen/uM5i9xSspYExIV/82d+cTkzHVOHv7GIyHWjyoyfc5i9JUNHfuP5dk8TOaazY0dA5UN7TINCo9h3DkwsXiFm7mZ7rZx+/4L1++7egdNOef+jy5F6wcBu//8/ZM3X0Qev/584/5zePjxpwMJvSMXzHfese+YrvoERV6580RX7z59e+hkofmRHrz4YP2oIjXN3ejZ/NKtR9bPA8ZyjLGLN5nOW7yKMZpbD1+udPVRM/PdeobPY5C9bpuVbbmR8rq26MTMKQgArzbent8EY/aCFY5NF+TYKRRm1unCn86UVOeV1nYPnWOM4DQ/+HSIErjm4MSVLbsOQc6dJ2+YmuLTWPILjnr48iPTrqFz+46eqahvF2yO55fZpIuzFVc1qSSm/89OumCAj8hcs0WmXMi0rrVXpka6JLSGRydZ33btlt3sk19WO9W42rjzUFF1K1JQ2RwUGS96v9BopqLftOswCYtD0mU9cLEKUXaFL4Mrl9W2qrtcvvXo4s0H5pZdg2cZ4NaMYeb249f44rW7T400ssQ+muBhAEUSFTsGJlg6VVSpxl4B4UK1TbrQW+EBWrDRTG/04o2bD18yAHKJVFB97tq9k4WVgjdK7PlTom/ull9Wx2pA6GqciQHHHvhnH67lCWBw+GSR+T7iTMQ6LkfDR58oKEejT4xXGFGaF42j0xUZl5Zf0bTSzZdbzl6/g7FkK0CVtX4Hj8vJ3R/MYlNyHJIuGoNW0PoKP4ztp6uhvR932Xv4tCzhDbqEfuLKHQbNXYPiUsPnr0lQ6ugfN99THBr9hRsPqHaEMRWST5ZmzF1itMeVf4SuvrFLbLLc2Uv01+8/E2KJimSALHHgwop6AOBGmPJeaOsb1eMZWcUmzJTAURkgvDJILDGQT0nP3Qirug+XRMamCH5MMdYnKU8M4WNbe0YcnS43n2BP/3CNV0AFUX4hq2Ug+t1H8o7mVTCYMXfplj1HC6ta4C13024HoEvJEXisoCViMzM8fKoI6Rm58K/ruHjbpIs3MWPeXlxidHp6FVyLmQhemJq9Qd1Xt1UprmrEbNfBEzbrLmKCioBkky5FRTM9UkSmJMMytYcumJEoV1Bez5TQbcwMpdQMjoyTjg6X3HjwQp9YeV0r40kjXgERpviM06X1a7bsZerqFQRd85c6y6okjQyik7IIbq7ewYERcTAGlo5El5RVMGZES8XOroZEpOx1W9HbogtPTRY92SNT486At2bTTnoDoCKb87JHj88Z4alt6cFgzaZdNuki9Kn8TrrOXb0rBnEp2Uyrm7qMdMmd0mkUYYzwHHhi+umTRo7klUNLXnmjT7BJ/rvPlDUQr0KiEsgPoYtV9CnZG6HL3TfUITNDxtbrMZuZIe7u4RtCs0GKn1Vufka6eHZf0RUQFi1L56/fY2q9CMYXsSE2Ko1SHeFwdPNwX9mEmPnjmaH2AMHvJ9IlzUaE3owluqQMGxi/LELg4l8yT31ik629MXcJ8EDRrAXLmS5z8tp/vIiQBXgAdqKoRppD+44VSijL3bgLR3Kwrgb8GJXf0dWQ9Izy3bwzIekQP7Oyny7cESzRqyY2OQsbepKMaaOLCzImv2IslZgc40footphH4m9nAFf/z10acpHH4XpUidPmVJGUokxmGR0RcWnewdGKmCQExAey1fNuHgFqQ1NRUovnc5b4hSXmoulKSHjb6dLWoJ2CvZ20gVI8valicd06Nw1KZnoCkj3wv7YNX7pFsqJy7f3HjlNj06Sug3b9klBItySO1G8MRDRwkxKQforKum5m+yhiyYe+3DU08XV0rf41XR5+oUx5mC0QOUH0FIZctcgJw2PyUdX1rrtlFtEKgKRMEO5RYwiiKXmbMTAN8SEkoYH44w1W3cePAV7/Iwxv6LRlJA5tX6ajOgUv8QPaD0zJl3UzgSdLqWrqXNQ+3sI+BnpIvEDLfNyDsbMex7khyjVQNNCQymI4MqdelQrPJCDyWHErekTsqR1naCiv7chjRn76SIuiUFUXKoeSTsZiOQ83KbWmQBOe83wxBxeZi9YvuvQGf15V0RsqujDo1MATCouMkNRrnD1OVVSK8Z7juSRQ06h34SyKZRhi5a7ft+1BMPoxAxLXoXXmuLTaULIz3YBA+WPC6eV0vG3CTfy//bugANhIIzjMCAEASEQIKpFNmRA3/879YeJjkBnunt4BAi5n13nnUvYH0em5+X0rFXZE17uj/I/9jQ/81lOveVo3hTvOvKsu43evmH1ugrlGyhfp59AXaAudYG6QF2gLnWBukBd6gJ1gbpAXW52hUVCcCs5/M+t5PvD8TQYEaRrSSAh/L6uSLUZ1/cT06cs/uXBVaGuzXaXb0++3W0RsSEcpiz+JFCrrvcW8ToO45zDk+ZBlnoWfLEhrFAXoK42oS5AXaAuUBegLlAXqAtQF6gL1AWoC9QF6gLUBeoCdQHqAnWBugB1gbpAXYC6QF2gLkBdoC5QF6AuUBeoC1AXqAvUBagL1AW8APd3BlmN0NC+AAAAAElFTkSuQmCC) Sample output from SQLFluff in the `Run SQLFluff linter` job: ![Image showing the logs in GitHub for the SQLFluff run](/assets/images/lint-on-push-logs-github-d1b1d9efc65a86cf416ce9fd081cc1e1.png) ##### GitLab In the menu option go to *CI/CD > Pipelines* ![Image showing the GitLab action for lint on push](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAATAAAAChCAMAAAB3aBOTAAACl1BMVEX////8/Pz29vby8vLw8PD7+/v09PT6+vr9/f1LS0twcHDHx8eCgoLR0dGAgIBAQED+/v7m5ua1tbXi4uK4uLgmJiYAAAANDQ1NTU3S0tLs7OwcHByPj4+kpKQyMjLe3t7Ly8tmZma+vr6QkJCIiIj5+fnFxcVoaGh4eHjKysrT09Ph4eF1dXVJSUnr6+vv7+/c3Nzo6OiLi4vd3d3b29vY2Nh3d3eGhobg4OCBgYGWlpbMzMxubm7GxsZiYmKYmJgqKipTU1OHh4dtbW0WFhY4ODicnJxVVVU7OzuVlZX4+PhWVlaTk5MiIiJzc3OhoaFCQkK7u7t8fHzt7e1ra2uzs7PX19eDg4POzs7IyMjQ0NDW1tZ9fX25ublxcXGUlJTx8fHj4+OysrLa2tpsbGyFhYVvb2+qqqrn5+fq6uqsrKxqamqEhIQuLi6wsLD19fWmpqZjY2N6enqnp6e8vLzBwcGOjo6Xl5egoKDk5OSenp6/v7+rq6vNzc3u7u7ExMStra1FRUW3t7eSkpL39/fPz8/p6emfn5+dnZ15eXlfX19dXV2vr6/CwsLV1dWlpaVcXFxpaWk1NTWampqxsbHAwMCMjIyjo6O2traurq7Jycnf39+bm5taWlqpqanz8/N2dnZlZWV+fn7l5eWNjY1mZsRhYWEwMDA6OjpXV1dGRkaKiopQUFBycnJeXl5BQUGZmZlZWVmioqJKSkpnZ2e9vb1/f3/Z2dnDw8PS6N2FwKFLo3UlkFgQhUjV6t9drIMTh0s9nGtSpnoUh0uezrVZqoC22sf8/f2NxacnkVqz2MU7m2nk8eq6urpSUlJgYGCHwqNMpHaRkZFPT09MTExbW1tkZGRHR0eJiYnD4NG0tLSoqKiNxaj5DcKcAAALs0lEQVR4AezMxQHEMAADMIdzWNx/1TIzPGsNIBARERE9lpBKN5QU9687mbHOeQDu9cbM5/vDP7DYExqUvMeiKMYWE6KVpFgg9YjEnDCVnNa6cG4b3aIAflzZYfhK1yn3GiqnYWYu6r2woei4EN4Nc8rMzAz/6FpWFcNIo8UTMH1z7vgntP7+epd5mUW2E5GzYAd20i4gNbtpD/bSPlhlP7sAwO2BYbJZwDxePgA9LtlgQZo9JQZbUOZYfEhO4UGzriLzMiuw4rwSchahqNQYrKwcsAbjiv8MTN8nNldyleluUV1TU1tXU1OP5DQ0Gnc1ZXFzi3GZNZgEtFKbo70D2e2dXYcOHwFw4Kjz2HENrLW9TGn/3/+7D/VkAL19zoJ+I7CegQwNbPC43xcAEPTIoRMAmhrkugALnPTJvlPmYKc98pmzKlh22D8UAc7lyOHzMWabXcuFi8MbYHYbABjh6OO1cQPMI1ViNCwPjyGxa7x5z+DE5GbzMiuw36ghnX5HP5HzGNEUTjudre00FAPLp53TRIfyiU5ihrpnD1GhAVg9T8TAvHM9rnlewJjcubjE9Sji0OIys9jMK6fcHDEDW5WHF9d4HS72b9niDwnRFcq9xE2JO0VhYCwOtskETB//a9zlnryTkov7C30+JHXJO+1XLi4al1mD2a7uon4NzLkZ1+g6uugGpEO7EsCKcJPORBeP4TLNGoDdWuByFew23wHWirEn7EBG7V30jiiAh8VJroCjZtoMbI+6bKgWLhWpkFcj7AKqdgKAZNdzJw4mmYDp4/Vx6nv3RhyI1IjEri33o/9CE8ZllmBqjqZrYA8AGznRTtcLCojGN8DagSZ6GCEqiL6/3QhM1IWVKNgjXl5eZk4fiT32IGcIwDYWZX75eIsXZmAhddljTnexAtziFuSxZ1sEauxGYHYTMH28Pk59L5dHep+IpK6nw9F/u+8ZllmDZT5rHvVCA5MBOEl0Ez9//uJ5xQbYXuAgPbxD9Dz6fq0RGK7IL6Ng8zwzM3OzV/Grj0tbcKYNgIsFyl41cFgyAdOWZfMtF9sAB7+G8voN84W/DKaP18ep76HobQ9XJoG980X/Nbz/m2AS1Ghg3dM4ScfQSieA0ydtKWA4dGgTxIeDhmA4wOxBNk8DyhXcGwCQJuFlrQDaWCjTQDVfRUqezAgUcS7uqcvehuHicuACf7RJAlLXy798SOrj9XENN4FpBZjiwcSuJnncPjh5daPs74PRi/AhcqtyQwPUnpEKlkfXP32mZmMw7GYPIv668uo5H57w23OX5bfI5ZX6bcziLGdH1NNcSnJ5S9ESS3gSXbbOo3Bx8ZMnxT2Oczx/JSgHku8071if9PXx+riXXUFlqau8rE12JHbtzOod/HLxzt886X+FGkcMbPY6ObMEsGc7UV8V3DQTBRvcpINlfDtE9H3aBEzye4CqOeYfFcDrMHOjArzzc3iehbLMrAKk5iUzrwN4HGZ5RcAlTzHP/QS2yMzLCgDYjMBsRmDL2Bivjysf4eB4HvPIDiR1XW7mi9kpZX8j/VSLMgdi2ZkGo4ixdFjk1i3tUdKahIRYMiIwSlnRr4FfhfbomNYe/2DHHrjlBgMgDNe2bdt2j+o2SW27U9u2bc930+Qa+/O6Ol3XzrzHWD27m2Syqlb6mElU5Qsvn3i5RpETzfQamcNoaOqTfT/Yn+47p9F3P9f3P1m4UW/3/XGd7xzfiX7++P6udHtHKaWUUkoppdS7n5LABJarYeqrE9j3gEF9dQITmMAEJjCBKYEJTGACE5gSmMAEJjCBCUwJTGACE5jAlMAEJjCBCUxgGVW1V62qgRzdXXkeKc18b2PiHERrurJZEMFakTROa2TN4VCklMdBoIto9bk7mGAfZvak6fh1YOfXQmADAIs3qnn+GPcAFvsm/wESYKcvF7xcgMZeA2CRVwjLOx8FO+cXvLobYLAxXGmTLHpw3RjLY3ECjJNL2KpNU5YCdTkF79k4Ata4gK08BhbsRV2X92xyEbCNZRhQ4KLqwHA14HAbXrznvHSwk9xQAy+DChbJX2WzFQCPrxyHfDORjAA6HAtsZGE62BA2A44EFWzriPXzq8FmHoASlo8eXTR69XwnXAUc1gd2cGVTvgJ6JMAiP0T0CPAxDIiDWTwHXO/9ArEcPgV6scl5mhq4kQDbwynAfoEB91hS3JVejU9gXObTrIbL0v1MgA0je+1mUMHaINLxKBiutiJ3DUO8qbxg6PUAQoZmfxRsUwQM81qx4CktbclwjW0k9+I8ItVoWhUpDa6h8Z01jW8lMIEJTGACUwITmMAEJjCBKYEJTGACE5gSmMAEJjCBCUwJTGACE5jAPlJDBxgUAkEYgEembQlKB1gQWKw1i+4REDCCTjFA3SAodNUH752h/31H+CzrJAWNTJoNMCz1orMbPaHxo5tV+oQVlvS2mnDVdmsCCtsCEzoOG0oYx8MTPn9EhggzYfoPLAYQZsURoiqve3ue7b7mir5csdfDWCC/nnAtAzdd1/CwXOH5jQm/HRaZ8HyosQvmxo0wjOOP5wI+DpUhpxmHOQ7DcJmOmawcMzMzMzOvylyHNszM+GWqVSxnL0oqF7f53YX5L+3Ou7bsnuLcCLeNzim7LdB4RQkOtuoEXpaXX1BIaWFBfh6ECUnK9AJ4XplJIdCcWCU0WFywFbyiYkpLSsvKSsspLS6CGMeSL/hhCL8LycfAWIPjRAZzeIFXQWllVTVU1VWVlFZAiKQLGMaFJGi8HAKDxd0Fr4bW1sGtrpbWQIDdmX4Yhl/mbmjuxokL5n/g5furvgGchnoR99jDO15wkW9I0o0bTgzwuvMQzAF/ccHs3vz+RWv5XqxYLS3Cfy1YDyRLkiyzZjKcN6ByBoPxtgsLdsABTjGtwxB1tBhu13KwbS881ngdf0XInY16L9n9iixBtfFOCBjHAVHBEmbw8wSthEElzYNu6kr8+B3ctmfiDzU1Y6jYfRthJmE+30u/1SQw8xPAzEgQFczBD/n5tAqcFjBVNJ8P9pD7grA5JnuRMc5xsgpmcgLBOLlebFmCCcwB4+MQFSwtBIMKSqr5XrQVquqSAj5YYhuuha1sn2qzoJmQjtnQxHds71Si7cDytunKHVzrUsbtB2xHgfnfKjOzAcz/XPnxEKZNJd2TYOK4l+uugu6GpALjdRxMSJqoYHYrBhWWgu81EAylhXyw6Cd4h3x3JJVEILRJ2rsUmgVEef9aU4cPssi4aycaSWJQFrHj80lwKOOCTpMcRClrj/SQIP/3yaH5MHHQ13Vbwcj3IBirXVSwJHBomStVL9cLZdQQLBbouMEvyQXkJOAgvyLrAyvw3XQgpL2NBfuo3RfWH9/GR93HgKs/ebIkLdf9MCK/69Ak/Z+CUUp79V7DBpsK4Mm3A8HGdXR0dKrBZgNo/wFZEwAL6QOwTmLBOsi6detIE9ZOAjNKg5ksyRbKsF4jLMkOAJNcwYI2bdp0SA1mB/BBG7LWAR+TRgA3vmPBppKMjIzT/ehaZx6MW5JGsixsSZpv+i1cL37TNwR7MrgkZwFjSIAWDE1vA37f2ViwG80AxizFsyYrEHpeDRYIDzd9SXUDOumGyaYvdKxo0XsZxwo+mK17/sfuTf/ggixl9UCwXWRW0lGSyYLNJZ/eDVWO4g2SGH+EbMVq8ukqD8cKNZfMDWJOsWOFyeDaO+Lg+ugJLrJgE74FFnSQN/Rg7yik6Q4wjgXzOzqVtPeBBcORdkLmfAxMU18m+gI2stbDwVWS+MFVhoDB9W8ejYz8QqAHS7B68R9YClVzD1RLvaFZagXjbfHwaCTJ/NEIMByNRvHhmwWDgUQi/s7h+4YTsrYsZf1W4w/fo/zhnRn9G2HQn/S3H95x3rjBmsHl9/btkAoAGAZiqH+PFTNevOwN/EgIvKYfzDt7QJw9II4B0UTtCHIXZzaH3BapgBilRe4kqGuRbIqCE2TnHhueCgNhhBFGGGHIhBEGwggjjDDCcABMDQkF7368eAAAAABJRU5ErkJggg==) Sample output from SQLFluff in the `Run SQLFluff linter` job: ![Image showing the logs in GitLab for the SQLFluff run](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAmgAAABhCAIAAADtHcr/AAAl6UlEQVR4AezVJQICQRiGYfK6a8fqTsMjVCi4c/8j8Ccq7u+z7jLyldyLAQCAPwxOAAAITgAACE4AAAhOAAAITgAACE7f9+v1eqfTqVQqnued9odhqJTq9XpFUcRx7OLRjL6hj3Qnd1zcwa7a8hvNpukCeAGCM0mS7XZ7OBwmk4nMZT3Pc9kfRdFqtZI9y+VS5rKeZdltz9aHujE4knMWzq3tuB+/SRps6DEzMzMzMzPz+8EyMzMzMzMzM+Pfs5/OZ0fTPalOe9LzJnfeejx3HMexbEnWV5LdO6zZVh456b6ya12+fXl2wHSn6eCBQfelKwMwrMMrhjF+dNaowuKvWe69sMevhhcO0/6krEu08YfGln9uGZ8y3lrUZMdtVlb74m7JkO4r3FG395zeQtY4Pn4M9cF9g+jpP9yHja0vtarpzJOg5ApZVUyelG122Ho32H+kH+Kb7jOtnwAhw3b8m2jdMePJQZO1wg7qGlNN9p/UTjTcRIzJ7PzMgyWZ7l7OmbJlW5iZI5YTzafaBNHprlMWX1C/6V7T0ZmjyRET+JMsZnHAee6554KLe++9N+0DDzyQ9tlnn0379NNPp33EEUfQ3meffWgTes5Hu/2+dvc13drNBLbSuvT5pdkBSx9bigGj80edN3biY/9p/Q1SGTw4iF91X9JN+5OSE60MnNNdpq0vtpY+xU6fwDLdY7qy2r9vKTnnKwOsf1lM/n9w5wDqrW+1cuCsrjPVS+etHSQyPnaBTk+uqNXF5OCtOfnR/EkzNjU+oX62957eW1GMTxcVY3LopPGbhnRb324Fi1AheDt7XvA76Vnp/9sW5qyNqJ1/bAQTmt9pTvabhI8uUWrnTZ11XM8Txwxr/Lkx6zej0k4CIeCznGj3Zd3G7xv2g2RzEoUJL1phgpxcvm3ZTggFUaiMzh3li1kEcF5yySWAIhlac7O0gVLaV1555QMPPGA/hXiU0LMqVbJnSAJ2UBu/aJBPq2c3eignr0QemRFUjaDIsOnOU9wTGq2vtSoBJ7aY8d1Xd1cm2Xea9idlXaKj00YoBMsrX4bOl7qyWOCksCNSCFsVcCIC2Ih7Pr/OVC/YNX2yRYJJrpCK6ckEnCQM2BTbeWKB8zNFxdCCN37bkHTn7Z3gGI5v4bws37Qsz9vvajueZFtdRNk1sNF5R4d0goi19MklFRuQ5mPzh03luHzLclUMEwjF+8YvGzT4NyXqIn/QZJJNAicJEmdofr9pQyxk7wQ/eIHN7zYF8mwxiwFO7jUFy/333//CCy+kve+++64JnHysSlVBwgIara+0+k9dB7HIv6ExAm3zZ004KPxgJe1pf7ANv/r/23d8/3/6s0YQdocfRIMKPvkVEVtgWEQhDGh/uE2baWkjEtqMcRKFLdG0PysJUaNw10aNqAXrz0eAFh+T/fLD6d5T+lEpj2jsaHDvIM0PXz1kQPPHK2xsf4TTu8I3OYALKSc5Gzi5stckGOpIJ+dztSFYvmGZw8NHdgrngVX7x0eOZy0y38JGmcP43vPrTOSS4em8s8MKWSebEjg9SMFGTFUMH9wzANh0nhq/agwekl2pzjBPcKb1zVbYZfjPptiOVpLAjs7OWzoFHdCuJSXlDOfCZUMX60BWyv5ZMXEXzkhW/h8RCXRf30kUMhFTzpkATvSHhTEGbcw5U1uBEGvGPhpJkJqCXfRotdFbT0SMVxaxjHJFxZNA+QUDwakqhpEp9URw9UOOwan8Cl6BWAXgRJr0EEKZLae99PGluogWEtf0N3/apD24fyDwkGmgx+zLHBimIvnwgobJsIwoRTcLzm8GODka9Hdf1Z0cOFEte8/qmdwKVXcxcCNZzIIeB/H2B7y0nHDCCXaaqj388MMjVUvZfvvtKxk7IRC930iqlly26tJ9bZdsA6cXH5N+fms/5zySUZkR1G9a+txSZORoE4tkGDa4axBWmGlpA2m0uQPjh/pxOFa0l29ezvsrAyfWk9+qKKGdw8uGfFQbJEESkv72e9qtL7f8il9Rh1cOywMy4ntnwNjxLz4ayV5tE4xFfT1pXgY7syFUGAJOgm1st5z3eGcW2UgOv2eF/59dwn7VqL647UzuvmwossnhE7gxaywwN2T2QDhdASq6lOkM51+GtL7Rkl0wyjSAH6nwHxOsbiDKkB2AR7v3jNxLyDkTOTEkQgOHj841xTQ5ZEIDj4FLIOwIwABU0NN7bi9VyERMGWds81sbRE7lnKmrrE67wUwCCJyVUEICejUzB85UUYUTXWEEh/jmAE7jIabFjYhcKHjpt7PAiStDT+cNHVmt+OolalEcEjJSBLO5z45jMgeG+VuUja9CHzKils0Dp7E1akxkJVE0s5D6ppNIgHa2mAUAp6h53nnn8bCWtK33mj4Ouueee+Jx0L333hsZ3Y0XLH5ctKC+JpHKgZOKoSHHWwhbsRQ66eXAWVCCuG2uBpwF0pFhzvsrAGd+xylwYgTxtjwM2MqqqVqBkwNpiph43UsUVNOwiclNyVJxKcDgOGk8p9IQhCfIJATlZAL+TX371CLjWWu7B3cPdPmzokHEPSrU4UVDjN1sPydZe+SJcmHld5wY3/5j/c6bO0hTK8DdSaYz7hpIYKdUk2NcGoWry5mP3Hv1VG3KGSdH1TGUmng6MzGhFfgHCFc7qzXBn0gVMhFTxpkAMGOOcs6kLsJLirJDjYkkZmVaiHQ5hnGXXxU4M0UlInHxJLoQ5XypWsyOpwPLg10SxpgtA05CZBegRMS/eolSdHpALBYgQqvSjOw9u+euq2IYF5x06sDhwWg94GFGtC7gxKHR0KHhoGAhRkfh5TBZej5mi1kAcN5444133nmnbXARgLz66qtpi50EoLwVOuyww84//3y+qkpVJRDt1s1amARzpJrH66k4Whw/2mjYkx44sY/mjQW8+YATznjP73kAVPRGiUhW233ycjynDIQmaayahidYqDg3mUXGZChoZygPwgi4ZyeH/x7XQsV6krumYaa0///9cuAEcvBPI3up3QEtMp3RRylUxgNLzrCpO86cM6Gi46PHQSgTk1cY2gvMIpEZDZ5QVgPOnDM0orKGcs5kO3XCqO6XFHShM8QXdFHvmKQqcGaKikmFCu3AgzmA0wUIY0jKNjF3BpwUoMvA19PN8uol2nlbx698WhyGy6BTpQWN5sAwxce/JHIIlE08ZETrAk5lZNBJap0GtMKUGUjE5NliFgCcwOFtt90WH4kyb7/99vgYgEo/t55z/OWA93m4EqZty8fr1OPnIvgASNv6uQrJfu9EjeI3CJxkyQrRPbfoYpXf1gWcCdH5gTOCDxLg8wEnl1thgyI9izZzNoyHghuKCS/HI43WWokImU0ZFQ5z2GXOPDt1EgZnBfeZSLdQOa5gzGw/e/eomPAn7VwOnCS44u2lNzea+0xn+MqAb/VOuVkUeKgszJGr96txLH/6Uc6ZiI1WA2cmJnxKIZNt2kB5yhWyIKZyztg/vGqo9cSFKuFM/nfJRdmh0uR7Z2XqpXvQJdBcnTmPTtOtAZyuWU/Cj+WKqi8IkhkrI4KqGBY8JCswOmcUjkgROJNH9QV7ghHjXLPCuYnqOSHuSF2oMy6D9XC9ql+YE00xTPcObwnOA1cqXkK0DDglAVFxt5xo+wPtcC+4ETBbph1z4xFr5hxYBHBeccUV5GBPPfVU/iLlzDPPpH3SSSf51RlnnMGLIV4PXXvttfTTrkpVC4Wu6yeSxS5P1YKR5KCwg/AxwIb7Tjlo52rgBHLiLhOGYg7KgZMjpIaR3mE8IBQ6qgtfN3AmRA+a0KDSKWna2KwS4NQfNCXFYExAVeDEkecwCDneFKLxGHTtlDM7QEPA7mxjmEizMxvjg0q84mMxHg+8XVyE/uN9Y6aIh2opsRgEHSGLDwRYAG0dfNokITEf8oq/a2RVESdlOoM4GK9+YjXYBbsDHnxqoe6hFVxzcvVQuGKBJwC5oXBSEs4kwJmJSbwX2xigTDOFzMRUwpl4HGT4wj0oippyppaSAKcWXKLwpACcSEFZs35fkWSKSjYYlcYKcxbwMwJuy4GTDcIuq+8EzXZAka/ivoDrAwZE+O54zhoboQGvXGchOPZIkqmej6hpfANKB7Nl/TnFpMJ4N1lCVAyjBlHG+Le8chL+O1sJUWfmo6ThOe2wS6RVnH9dovwk5rfBjR46Fi8AHCxSZotZAHDusssuF198sf8HAtHnBRdcEC+AvNek8O2RRx5ZlaTehCqrnSr/gwESszqGVvLsJnAwK0yiRDnPAZyW/lP6qkvBcIRViqKHiAqqiwGrRoRUj0H7vTPAeV1Mm/RnJSHqRUihcmDKgZPzH5f27Hod4Hy8CJwGBM6gosdrXg6VnZgq+Ryn2lNh5WHL6tfIBKn2EwyFfx3qTkK1RvUl9gq3ScQSOF1wodIfQQZsxwoID5nO6JUb/Vv5if4sD2eUdSGH6YE36AzOJCXhTAKcmZj4N8JH8MDIIFPIVEw5Z+IFCvppf+8FvZQz9ZU1Uxf9/+ureCYMAzjlA9yON3Qlirp867KdbhZbtJFUbdQIBMk3rP6TSp0exRE1dkFdnabmMK4mwXa8tZmPKK5ADAtxOE+4U1R8x3KiYlhUdc9TptEQsxlWTjS0NGqEmLyS1eaXE43nPwWOgQWFwQJqtpgFAGckY/fcc881O/fYY4/NkQY+K/wHCDi5xASIsJDpQlPpxOUP4KyxcIH63/P/3pGBhMmFTm4LZh9QyHnG4zVvhD+42yTnnaf2Yk5i41ZbnQGWKsy/P7pX/MNco0D6y0nXzZkQU/2lBs488QV941RueHiiqATNR07o5NtNFpBsg6E2ekKumAQj1Nd4ffMXvYSaicoBdAzXZ5NE0XY4uUl2idZcb2/cM/4Xe3egwSAUBWDYBG5AhiBABERAhHqF3uDCBRj2dHuBvdfKZbMBDLbt+xBACD+c09k/09FP3p9HfvJi0/se4QT4HnlGLJzCb780O1z3hVrXUT5FOIfiUpSxrABAOAFAOAFAOAFAOAEA4QQA4QQA4QQA4QQA4QSAvw1n0zTjOC7L0vf9y6nqruumaRqGoa7rCgCEs23blNJ2AiXGuD3Xdb23c57nG3tnwEpNFgbgH2FtUoAIEQVFVBCBaEkhvwARqSqqoghRKigoSlS1KWxFbSFShEJLyUJEIKwg7NM97Wm6t++0I9/W3e99qu2dd87MGVPtc99zzvlG5/nYNd9R+VzfPzmwZkZHR393wLc/zXkzP1sADw8P69nx8fGMjAwCM/pTMN3d3QRm+Cjp5OQkwSdobGxsaWmxZnhg9WYEQRAEtxFnUVER4vT29ibmS5w4Mjw8nJgMsRJPSEgIcUpKCrFd+Mbnx8fH29sbYtPJubm53d1d8gMDAxwa8gY6OztpaaW/v9/agIxVVGZmZ2cfHx8JzLS3t3Nbgk/An3ZycmLNbG1tqU4FQRAEtxEn39osLCzUBRyC5POcxAkJCcR+fn7EWVlZqvQktsvExMT7+zuyqaurczpFXgnSnDeLs6qq6pd/CA0NtTYoKSmxZszExsbm5ub+x+Lc3t5+enoiEARBcGNEnBiIOD09nViXm8XFxerQLjc3N5ubm7e3txsbG99DnAjP9dTBwcFfDmpra3Xy9PR0fn6e5+H+Ozs7+sKamhrVmKs41HR0dFxcXKhymQut4pyZmXl9faVY7Orq0q+OIdzn52fO3t3d6R8Z+fn5qkfewPX1tYjzf4ggCDJUW1paSpyXl4dTmYRjdjMtLS05OZlTXl5etnplzREiYW5vamoK/TCf9+XiRId/Otjf39enKJExvdNQLYoiMzg42NDQQBfMgJJUvwxojN0RobUApTGzrampqViwr6/PKk46raiowL7cR02sYk3yzNHSmCfBoCTh/Pz84eGhvr6eW9HA7cUpCIIg4gwODsaIehEQssQi5LOzsxEqviTv6empFgoR2OoVkWALX19filqCysrKLxfnwsLCbw6mp6edGriK8+joSMWIltgwxxkdHc3li4uL8fHxrkO1/NogpjQnVq+LAvT+/r6np6e3txclk6+urg4ICCDgOWmg2jiJc3V1lasI3AZBEAQRJ/j4+CQmJuLIwMDAsrKynJwcklRaSqhhYWEcskqImMAWl5eXLy8vvzpAIXjiew/VmsWJX1W8trZ2dXVlECcMDQ3x8NyE8rG1tdUqTqtcmWSlECegsvzDAlUmVSn5goIC1f7s7MxJnCMjI+vr6wTuhCAIgohTw1Ig7KhqLPZuEjNgSwzsD2Gtja0uuQPawJ37DphEpOSy7hjhkKVD+tCQ/ypxMjf578WpoPI+PDzE5ex2/ZY4iRmI1mPFGpYm0aC5uZkYELASpxsjCIIg4qTcZGqTQcWIiAhGZRmqVeOxzEcyVEsBShnKv4GAROPi4mx1OTw8jDbQp16DwyH31A0oy1gyw/KZmJgYDg15W+LEzUgOOMsCHwIqQoM4OUsb5iBpQABqDyuODAoKYpx5eXkZcbLh1SDOlZUV2lCY8uODRxobG1MLelkQxDQn5TtbZVznOLkzGbVRxz0QBEEQcaJMpKgoLy9X+0/0qhk8Sh4yMzPtdokwWF9KoP2EWpaWlnQGNR4fH6MTSkwODXlb4mxqaiJvZW9vT4lTz4MiTkphFVMKO7VnRTGTsjwtMXAhAlaN29ratDijoqL0xC2vkU2ZHOpLIiMjyXNW3Yf/0hF/F0kNo7ucUl53DwRBEEScQInp7+//rf99U3LRgOBHg5qbujMpKelv9u4QV0IYCsPoBuC9SROWQXUxQ4Pq/lc0v0SiZpLmHEVwXPMFc2+eH8qssloof6L3lxnsGEMdAeYJJwAgnAAgnAAgnAAgnAAgnAAgnACAcAKAcAKAcAKAcP7Y374v1/WfzezTKyVfupynsc9kPY5M8vVwUyPblnGtWZDJdwhn7oG01nrvtdZSyv3Gde50vj/snQOT5UAUhefZ2JrS2rZt27Zt27Zt2979b/u9PVNdU5lKxk7q4ebmvk73Sd9zunvR/fqxSUgxmT3ZujUvDM5iEyeGV6zQK81WJKVzBA8cyPr7NzpnTpV/tHArLeVV9reODxjAQ0x0714NYS8sMraPr3796LRpkeXLI4sX5/Z73r0DSQYi2CWYNTwalVB2CJTJkWrcGLg8P34UDYFU06YG3ni+G1q4hyuc7JbFVl/sf8KeYnxOnDhR2tmzZ09OubRw4UJdrVWrVlESdfr0rF+/xOzeN2/w+O7fx9YLNXWFsyIIZ2zSpMCpU/Fhwwr1q9COHdw3smRJ2cAe2riRSqbYWb3iHjbI2B8SSF6kib1wlljWeD5/pgQwLDMEKpZw2iDAgo2Blz6WTxHu4QrnlClTUEeJIhtzopGtWrXCbt26NTt0KoaZKP5hsGph6Xjs2BzJfPXKf+GCujWDO+jA8/VrqQpnqlmzZJcuDOerw1ItLU2yd2nxaW7p0qKRY9nA7n35ktslunatMsKZ7NiRMKg82amTJReS7duDZA3t1eqYNa5wajMj4ALPoiHA6BN4mW4WSDjdwxVOdtxk70nZ7KiMQLIHVt4wxFVhhTrQSzpiaPNmw6qW7muhAP6Uwvv0KUNvrgb37En/l3M6tO/uXZyimPDatVryDR4+zBQ2tHMnTm4UGz3aLGrh1ys2fnzuymTit2+nKP+NGybHwitXoui8mC0R4Jw2LKkRE506FTu4bx+2/nDRf/MmdnjDBqiNV2TePOfRMcG+O3e4I5XBTvTurUvRWbMyldy6lQIzOHz4kGrQgPZGFi7UFATxiI0aZYryPXqkluI3TnCjbsBCCb7Hj0HVSGxk0SKVQ3uZwOGLjxjBz4mUE9t3+7bTBLdePf/FiyqZwZDI0QF2CR5OwmgO8eHVq3Hy6P1XrlAON/Vfvmx6AmtoqjyRVDU2bhzO8MaNlIBHfQBbncoBmbw3BQc81DkH6hkz1H9sm5qdrdGepoPBQ4dSDRvihqBVjv/q1Yz/yBHqbI+M0zhAzdRiDD1ffqpkkBS/O2SNQUBwRebPx5OvbJAFtAs7Oneu/KSVgOLlv37djE4YkJERGcFWR+3b1z5VC4cACUiB9AGd0go9L5vK2GcHWfDggeBiXq5Ie96wQUDlT5liFU77jhobM4a64edFGtL2aq0wrnBOmjTJElO3bl38/CFooW4J76jrJ3r1ksOZAhh05zDU0aPe58+xA8eO4Y8PHgwjMIA16QRH44ffVb4MYkxKB06cINKyZqhg/JJzREvZJT8yJoOct2uQSM3QAemELbI2DEgKyUjXrWsrnM2bK0ZCJdI3Km7qGTh3jmKhTvjCUklBKvEOnDljWaoNnD/PKbwGgBhmxY8/RdMp4wYoQEyNZgOX0R7s0K5/7JyFj+xGDIcFj5kZV1BmZmYUlJmZKygJC8L3N/e7fnduOrv2QsrJaHTKZrMzHo/tn2Fy32aUS6HbJBvlRsF2OR9GkKnxeHDw5Rh2Fkpcr9GVH+msC2YKbDteeIHBmVTiud7x0kvcTziTTHrkiCPIDWVMfys7b8IGgZfbPvwQE6zMyDFHxl7zgAKZciZvIC4AoBiwIibyPuDHxyC11hoe5g7Axs+NR6mYzgVOxaMrqDAQeUPF0L6uNimfW7/7jtXBxh3YilxVl+IAMCb3CKb5hN5xzemelJhEO9QCcTpStandSDiQAWcmqNDsSnHItn32Get1FcNtY6r2xRdf/KN4H3j22WfB16NHjy41Jc6gcqk/XgOn0s8djD4xHN6cvwV9sV+77r8fAcWGWunB5Qzg1MXWyjBjUWxzwD1XX82BhSi4EoFpF/RzewKn1lwFYwlzgXPP5ZezQIk3valpgAZWbQKKB7RNqj1honYqq3Hu22A7oSSclEiDPM0HYBOx4wqpWiGKHYHt2se6xhmcj3AQACOO1AZBIV2Lj9G0UhWCIZFFqrbgzPSka898/TU3+bvnyiu1s2sczhtzbX/3XaITjKmmVuD0t3AAjnGN6KacKRsBnACc1T5r4BTY3F/YSMJDH2IucBpmETHrOpgNIk1CHI8urKvt8ePKjAvcfeedrKtW1WU5QKiq1qC5wj8/TIlJtCOrcSZ2I+FAApyZoAqcdLK7mJQRYAYHnBydBSzjcBAACUx2H3jooYe4f/HFF69Qe1O2ULlFgJPIgDtNR0Bx6CI+o3Ohby5wmod0NABpLnCibOIWP+GmBpGsHdfoT0/g9AyhBKP2c4Ezzn1obsI0AGPd5121zjXTcU0AlAGnbkHTibfkgOP0qXE6CPDMNSi1OHCazbOBRtNEEs0QL7o1dY2z5kw2KY0nFSR2xz2qz4U6iFlBDajAGWhH0p5rAqOEM385cLZslMisJYIKbsUJPuel+04RuCiT/YoNKlR1WQ6AQDyDHslGUx0pMYl2ZMCZ2I1EVRPgzATV8oEfdVZM4w+ljcBpJpbXTu64447JZPLSSy891kle3XXXXaDm1au6VFo6/FMdwygr6ik3mIpYGzGQCLLvvvlmcE4t4nSurm4DnFi3pYATSgI4zXQFruCQ1sAZIQs64xIa4ETxuDZ3ahhUA6dFX7Uazdc0TBt0nXcnIm2obmfAiYnxI/ZdNlKmElokMggzNWojqOIrZue6buaiLcoSgS0BnJ3pOMRr0B97jSEjuIQhYSiniTS+xHbXnMkmtRlkFD6EzcyBAokLaORRA2fOmf7AmWqNiI4Yy0a3uwbOMP0hqObwGYooDQBotgAemocX5ApVXZYDMJYYzmFjZzNiEu1IgXOm3cg4EGXL8L1qQY1XiRAMV+1KB9RG4Ix29uxZYPLGG2/kOt5IuaHHIUYFUTWwnu/9EFmsAyZbO4tyxikMzuBgAc2MUVkR20irqgw1cJKcQbXEbCblmqpPBpxeS8kiNU6jUrTdULUBTsizvkjPPNBmUrQ08saZaSAdpFGACf6Q90Yy4PSshNVWWcHgbET4yJ5nwZR0T1LAW9dFSM10GeUOIvfEFU1GxvYMwyKqoFKFU7/lhx9YHfYISyrxUoKQdMMLZ0QqcF8wWBlnauDkgSiN19KLEMpGFgIPa+DMObMccMIZWEeXP6yLa5Ay0xonhTB4whawKOsOWQtBdSj63skE2JONwIkYFljFVm779NOdTz/NRggquaquwgGiTJ+MPG1CTAqcu2+5hTtCo9/SCX9n2o2MAw5lAl/3i0GI7DNBZXwEA4wHdI2/4dIIMwMCTsJNSpvnz5/n5ROytaRqD/9WP7jqqqtAzeeee+7KjbZCttazi817nJE02/zLL3HIIk7N+bCdB8znaI7p2PoaOCOA63ZUKANObR/jE7tofCUma6T+hDpmkSpKiUFA2AvNfQ2crFSQoAMDBXDiy4ddmDZGlJAb4AyrESYJ42L0FuO0Yevp0x4R9PmaeMy09FvTwnwUbA/Od2NH4UdnxQ7/rWNBvMPa43yyFUeDTmWj5kxMmkU5Hv2ti/TKmLm4GjgLztS5yhgq3kLhTtMBp0xrWGAUAp1ajtXA6VGayI4aRLr7npUNrAoO+BVbkKvqKhwwOR8qkxKTA+f2V16Z5hjom9iNlAPx3rnHlCwDZYKKW6wUhahbFBhKG4ETyAQgba+88gpBp/fvvffeuB/f9nmrEqXSo6wbz/Awvh6BVDOCd/70RjYG7EHurcoIJ3XDuAPAM18Ow3v1kF7RArk1hREV1Y3lc5ho+jALyD0T7VBvPGhSWA3bMbUkAPu8asmAkC0lPRtkQOQ0x9gRiGQJq3MmaWQy9XvIBC5IIRHG8pz5WxszojLI1eJTU4lkaQ0brR3M3Av+1qranwM5Mas37UbJgdUFFdeKmwrS4NqYqiXEPHfu3DHEa5Ct8VUJYf/q98qbkLfPf0cCL393+cc2D13cpvXjYKu1sY1tbCNwjm37G29s+emnzRcu/MreHeIwCIMBGDWMiYllwyIQaAgOyQW4A6YcAgsKgSHcd6tfsmVLpt7TFU3N17+m52mKT5pfSec5OY5rUXx42X8uTpfl152HkOz7aV0vff9+bCXL4nFtW6xmnML/ABBOABBOAEA4AUA4AUA4AUA4AUA4AUA4AQDhBADhBADhBADhBADhBADhzPO8bduu66qqur/6NaKu66ZpbgAgnGVZhhDGcRyGB3vn4CRL1kTxMrqLD2N75tm2bdvv+9a2bdu2bdv8e/bkZr87rjV6I0/c6Mg6lZM3K/SLvNPYiddVq1YpdipqwofS36eBXdaA7F/DD90L3fyWfFrk8o/y8SBxW/yrsktSJLtnu6lIJBKJigKca9euBTgrKioQT5gwAYDs6OhQd0tLS/fs2bN79+7fDU79c137UeMFQmTx5khf+0GzHrbSIpf+oY4HCeeGvyY5qU6QrH2npUUhkUgkEnCCi2vWrOG4pKQEgFy4cKG6u2jRovXr18+dO/d3g9O+wzafM/UviCXuRf85cAo4RSKRSMC5evVqvmxpacFlXV3dHwEny77JBh7UxJlfm9c/0EELANW92E0HdYLTeMUw3jJwy3jNiDvibOTYd9n6Z0Qp/SvdO81j33zWhBlOCRFjF8TI5ENj70yPJmCA6lvNucYp1KmjOqgAH3dz+3OF5DMomU0qPiDN8MNZIT+R+STtrsAZjYvMF0342NF8wcQlb+r/z8ezUxvXOcUNTpFIJJKj2s2bNxMgBg7csWPHnDlzEP8J4LyxE5xAGmIsBqTy/SMInAwk9u1b7Iya7gUuFXnZcM91rUcsjLbs6x93covrAI2UfxblM5vBMHCO8xGzbz5lGu8a9q1UxzvFI/NrzbrPwiti/Ocywwfmmd9scgNJZcI8Np83eRfAMilLktKEc7jVYgKnSCQSCTibm5sBRfXmIAygOJuFP2PGDMSDBw/+08HJsXWvxaMn84bByUzCAArqkP8J/H5F6AIFXzdyO3I8U2aDk4+LveM8ToubYnrtiBld0fioq89TI6bh/Mq8c5VDG71h9OeDhVwEbcStsQJnbm+OYYlkLGZqsCwIlgfkv09PhxwBZzFJJBIJOKHq6urJkydPmzatoaFhy5YtS5Ysgblr1y68J2jtz0IMcCLA4e0fB6f5tKl9r/nH+HxMihgrHUjgRIAJMoUGp+wnFUnajzC5gpGcBvB4x3vs6x+RGc4pgBMxg5Mzo9ERYqVgUQCTya3EyT0W5uD+fOxFwaeFIqjGDWAU7p3v/993zyHfvttGMvpBLOAsJolEIgGnUm1tLQA5fvx4njjnHdLWrVvhIwBZ01+tYHEQzgg5xuko8OCdTGyzb7cR84kocggbX2k8cSo+RRMjFWdpQAo+OZc7VORbLS0hD5McLvNr8nFLjECBU/+SfP+Az3+alJEZDY84Jx4ad/WxNTV8mhdODwtrZtifn1Qz/qkBFaMxMJKBqpKD+UHcGOfX5+HjdJdm7o35QvNFIZFIJBJwYtzEvzbr6+uHDh2K01p1PKvER7X4P2j6G+UfTdgw3jSwmCXAJMFyYcCX1oMWkIkA86gCJw+dOBEl/y47e4oFg/3DfPC4AMiqhPzbbMT6e7ral8HpXF3gq/WAZd9pG28ZjF7jbYPhbd9i46099k02JzO5nesd9xIXkEPZDJ+LGO8YjG0GJx3bfouY/nuK6RPdat9oAGdSmXCO+YwJR8ApEolExQROIHP/IW3btg1DJ8w/Dk4+BeX5jJd9g93J1GN9JgqW9bAFkJB5OIETBGKWgGTh5DCjPg2a32m8AFp+kw4EYuESJsDJd5PahEdJ8FL9ifm4yfnxsNh43VA+MMzJIKIy0VK2T0Mkd/KhzruHs6j5YGmAibOzz48Kb/HFf0nZAVO5TioSiUSiYjmqxYhZV1dXVlaW/gWK2+JoQtTHvyoH0Uc1mCJ93BoT4fUXlZQnyIzb495HuNGwqHc+kw/9xE1xT78uiSb26qeEmoxGRAiyfFV5VN+bYjsq3tCteFKf9NH5T+3cIQEAAADCsAr0T0sHDGYrcXcATN4BQDgBQDgBAOEEAOEEAOEEAOEcBACuCmr/QJP0VtbpAAAAAElFTkSuQmCC) ##### Bitbucket In the left menu pane, click on *Pipelines* ![Image showing the Bitbucket action for lint on push](/assets/images/lint-on-push-bitbucket-746ed122c51527e3a775d29d3506ad5f.png) Sample output from SQLFluff in the `Run SQLFluff linter` job: ![Image showing the logs in Bitbucket for the SQLFluff run](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAoYAAABhCAIAAADENvc9AAAeh0lEQVR4AezXAQaAQBCG0blHAHSCEBASgKQrJHT/A3SBwYKdxeMBBgN8/pjmtRwAIMkAIMkAgCQDgCQDAG1JXrZ7P5/jegGAPiLtcf1nACDJQ+xjAJDk5A4AkGQAkGQAQJIBQJIBAEkuAQCSDABIMgBIMgAgyQAgyd/PnjkoWRfDcPzpPtta27Zt27Zt27Yf5fvNduasu7p79i5mMnfaFGmanPzTXPfYqIx6Qd8N3c6dY2gbHJvdVFg/kVHe/8/c1zsiX8yPSKu9rjhEhCRW5teNZVcOOvgmK/y3v2zYFmZezUhwYoVkB7n0l1/NU4u73UOytdbGpo7hHJ5zSuYoFvEIzbn3A7sFZ/lEFijdbwauOdXD+taBCufGxnpkZO+ThO8V1I3zi31voLvwjYei728TL8VRzV2i7usYhnYh3Py7X7ZXnI8DOwek48B3Kp2Y6Rqcae0R/+GPnWSJ+tIVUl/6Jx1HG88E4omOpd8zJMuif/PQduvIbsf4Afd1dgJMhphQ2b7ATHC0uGmKRvvYPnwx5+ofA/uwsLJjoXFgI6mgQ8FROOxW071S17Na1bEo2UQu/f1v28N9lq9+KjIA78h81WxMcsMJX3+3lMxBQYhptd0r9+6UmKOhf0PpOgekcbDE/LarrL2xq6hvLI2YFa+u6lzEyXHpG+gel9PCZJGuaT+RzSuOKr7lO6KW4R0iw7nxREgXdDxx5A6TCtph1vWuHl+SXNihzG8a3AQkNC7dziepbXRP4eMAhFDJEtWkQ0b2oWgNHyilq6b00pYZwRREV+bnz4VrW6/EiyCZcIxhzmZbyUWd142zmRX9LCG3OsUnpYIfllItX35F6bjdN32Xq2+FgiQEWgXJgmp7VrUPkkUsjnvzw0ojxhKkWWOpT3wdKFjWMiPX/ZFBskIqQPK5HwJMRDv4pnzWdcIxaOtZBYjnO7kRXQHJp7LJ3JqRL7rOMVlNtCva5jUu3SUoI6Woi2oidvQMz4Vf3bUkWaKa9OjMRgURRYVSTd3zakd5H7/9af3T2KPmcLmlW8wzJN8EkuFTstZInK3qXMLGZ/nxh5GImrb2R/lnSD5Lz5BM2RYFA2JLniFZNUgm7iM3p2pIdMtaZ+lmlPXRBgx4qwXFl1MtOw7JALAoYIiuwBLSd81KP0XNg1tNg1uSJepIh2hQ6eS/sOOQrJr04+QVnscSTvIkIJmEnWxdlMvwv+LGKWWIlw25IVeMv1J24LJefbOQQDLJJi9a+LnVwzSgr3rOkljz3cCNPfmfRqkd0fWPKaZNWspyaiYN/f/JuQsd2Y0lDMBPePmGmZmZmZmZmTmCoCAMwtDb5NP5pZLjOVvx8c44MFJpZfe63d3V1fUXeX7Kqw467hLtjCbXUstelfYjT7tmtpb3fiMiqm3Qzjx8E7ztd/QFL7z3vV4eOO7sG2uBWOG/LkKNQnR69T37ivt0IW0HHHvRGx//hJn88sTMn3z988RtLDM24CrnszWBZGNZSEJAIvZJsfSQbIa0TExdHW99+K1NCKKl3f/sh/iZ5QSSrcjyQ6tHWszqqTe/JHLIA1fd/ly/Wc3a/3nwGUQiQ3uV4MqMzQo59hkigcpMm5x7rWziwy9+ahR01R3PVxc7RROFvdR6dJDnjz3zBtcOl7m5IE4aJdKOPPVq4XTPX3n7c5mVwoh+7cib6cc6iV4ygmRM8AABO+TEy3p2rYu4L2Zy1uX33fbIOxkoTqTKDO38oYpXu+UV9ZDsyJOHbOLTb37ldMwQ4AYYbrjvVW+Iejn3qgdri/ew7sycrxEkG26YFFO2osvl3rDW0UdZVe3ErOmyzOiI9s4pHkLyYqMP6dm3v/Yv27QVkEyTWq3owUXXP3bHY+9FnYUcjLi86k3EEwIMDSTnmSHJxDS6htLMVpXOdcspdH3HY++OXkWxaj/xvFuqpWg2JFMllFpiU8N2YpHzjywBKMZ2cz5XR6dKGqDyAC3jqOeFUS747L9pZAOZRjQRpd9z/qEXPslsxZR0yax6SH7j4x/N36mQv7HXov2bCMNGI3v/TQ+87lgGkllR2PvgnjmrXVotODB/eJa9FvloNqtZOzPRtX/pdenNT6n4c4BnbFYsIW8Ww+S/gkB2/emX3JWCtSgRnDSNobckRx7Mw+EEP6++8wWGYy7+eciZGRcSMz11DyCNKOeoF1SN3P2b7n89ssHPHkKyHGdGZ//17Fojgfmkw2OFMyuz7xauhcHturQwo7aBZJyPoOIn/8kDSgJnCHADDFF09L69YAU6fU4WztQDq5A8eiAFNHi+odFtonOku4umyzKjhxpIXmb0QnSacFvKu+IIKt0ctUPoqLkKn/aQHDudYaVdBakLxPyfB8mUmu5OuA3Oq7w8z7imdj2Zdsb4LmOhhGOvkEwXUPel3YI9RqTdiJSLUO87ljGR4gUM5C6T0WCSgYbaDZY0nOdm5ZlS09F0PSRbnRGVV2xOCkGXmRCJvQauDzv5ylVINiuH0y7XvveQ3KwdSq0GgWdsFmISxUjKxEKB5GwfiU0CLMIPSIJJo3VFfbNFkh5LzA00kij/IrECcYHPzAoHmrVnVsJ9uQXtI0i2uTFKGBA9uzYByZXke/adb2ZDsu3T6NuK3Cp5cztDgBtgEJGK+RIbUSTj6be+ctFAsvl4gIYUFHG4sljdNzG6qIAIBxk79KQr+i4LjN5A8sKjq830JIW/RRXXUu4RNayEjhg31LN4NB2SQ9r5PXU7D5LLQKZPd/Ls63ZDkFw6hVPrdkZ6MpCsGM11PIBkaMxf4XqU8tAGZyc2nL/s5qdcJBKLooUp9x6S73z8fY9FQQc1e2JF5fkiCr0WPmxPTC8ris07EZKjypNbmgjJzdqtK/xZSy6ZvAV9yUAEIJAcWC0llb1LmuaB5z4aBZZY+soghJGFOk3Pzgp4CF/73imPCfZ6TMhniqDyC7Uw1HaC5IQoxIensGuN+559rMMubCCUPQ+Sk4oCh5kwjiVQOV2Ae2CIYZ1plLCJbdjoBpKROLm3RT3KrbjQsvbR2XZW7TauUd9lgdEbSF5y9Ni+EH3rvktmgQIJWjUmf1wEZkvs/RDHd0OQHP8m3sOfB5JLYsrJgBzzIPn4c24KJIu8FSQTzVENoXYC3XD+2rtfimMtOFwkRddDckrKAxUJMPZztkzHb0hxDZGFDNuPOfP6nKgk/+oNlFoPyYwwLc5bJRF/F5KbtUvW+hefeC2QnHr+5BTQ5bc8XZCc9aIoF9AIFysNERJu1cJJNdUkPuCTRjJGDYnSz4BkQK4FZBbcjiCZe+ovI28Ku9a17wXJkeqiguSqvvRAD8llV9VUQ5ynXoCnA0PVD9uIqolJYr6H5GJOSky8QeRj7aMLzIRFU7osMHoDyYuNbtNjh23vT4VQ+vKmpYYSqKFwSwetHZKZ1VoEHiP0iVb9+SHZuZU22yUkR8tjeFlF8T8azgfbyPe8imvIB5wCGGsUwfi48Gbo0vWQ7Dr6N92Df5yhZrOate8UuN7XzVotNqFfKI4GktkBlZgIeT4zTwEXAmwB4HSZAcmEU0vKGLmexDXGXCA5shEesnV6dq2RGkimQLQrQMsZj8vbQ3LyheHwfAFugUG1nb7F9vgDvrSZAskhYY/y3Uv4BQZ2OXrilA7OxC4LjN5A8mKj24jktrYOkikUNrUqTZyKQxb3RZFFggYpD0EjSA6aunD+p0Oy5xEEKm2O9Uo0BfoMsWlIrtFHkKxSVHu0Xg/J+W+4RCUxnEUvZ0Cyt+V3IcLDfLenCLnnfL7P4ydRzT725T9FNddWKkHKGvM8HYex2EuLWV2MnqRU10jceq/lAuY7chNehWS/w2BWznDKvrQ4vSKBlGBUNuBpNqtZO28svycj4ceyFiaNM7qvmxV7wmv9ihAWiQN7rYU0kFx5MqE5jbRMKiWjyLLGGLuuEXmYCMm19sqzOghSm/mVnjJxqryLQ2mqWWnPrgUgmaWe+iz7a8LkYRWSRfItML5+mRrYpa8NJbHQd7YAx5ZioBTZghJUAVLo7rXZ08TnwnA8lLfOtUZmsap4u8b2otNqc4vojajB2aNHRANLwy72tOmyzOi2DB9iLTk7rnVfZnSFYDH0h+26b0t5V85MyHcIaa8azkTARrnknLpQBS1DwyDkTr+DU9+QcJXq84aUsUSfFiTbv11CcjN6QfIoSQZaGkimMsKZEH3XfCVSymgAyb8wfYJVrus9193z8l45jyHhfAro4g+FaKhKOZd7PdT+Bx9/aWq8ixywTXxla3VJwfpEhDJdheQQKRqyNEXXJukkY36zWf3aGZQ58yGPadzXzaqvd+phggGY66dp4P1vIfn++oawukge0+Npd1vJ4xfe/97tqIAD36YIKsGL+kupUQwanBxJJqDCxry2Z9e6CNtXIXn000u2NeWKQ31asxomaLCXEijOi1jOFuDwYUhK32MiZ4hQfXdXVXhFKR3Qa9jCVg6ri2IiJAwze/RYciNy3psuy4xeZ6GIvlpm9CQxR0Qtb1HgGssoGibMKDQhfJcs3RCS10v2LB9N/rVIUAHHdh9XAdtcNymWnTi/SkJnbEzPTI8tyyMaJcm5TZBMJ5U6fUpxjNRe9s9MX3tASIjMTHa3WbLjJ3sPALMp02cFcjbH3rik/zvi3Nns+kMOCC9/Ri9IvzkB9iqScO5VD+LMxN/YZz8xvwItq29jfTI7NjR63+VX9u4AA0AABsPopQIgdIDQFbr/GRqDgkDrh8cDYAw+ZWpy+rv8dF/veiQZgHqt0gdfpvN3kuueU5IB7td8H/2Jy3RJBgAkGQAkGQCQZACQZABAkgFAkgEASQYASQYAJBkAJHnbT3vJA0CSl/WwlzwAJLmrHHhWBgBJzgIAJBkAJBkAkGQAkGQAQJIBQJLnXOyYA4+1SwzHv8hr27Zt27Zt27a5tm3btr0x7i+Zm8XB854ne5icpElmOui00/Y/0837zt95+g1av/OU8ujY6Svvv/xl4RR0/8XPXkOnM01Q1/4TNdm8fc/R+07c/Gnr+93Ke97KXU38xev2v/vu9MchYP/JWyoXSst69sF66fqDBjfduJmrr97/oG50+sLN4vynrjzV88GOXXi4YPUe0e4zfCbGHzR2vnw3MEVSdDlZep278ZJbMzaNzl5/IVSYMm+DfiRit1VbjnbqO/6fpp6/cjeGJUK1KKt9rzGL1+7fdeTahFlrFHLR5r3neg+boXtZiqRTWVPmb4TJfCaYYdUMyaOevLMIjMpKzauz94ykq+AraQUN4YmFdh4Rbv4JQ8YtYCbdjMLGoRMWabL5X4fA9IIGJ58Yr5CUJvQFMNjBOzTVxTf29VcHlQulZSVlV7/55khDmgaOmWflHNJv5Gwdme7Iuftop250+8HLqJCQWZmQVaXnO41LL3/1xU60Zy3dxiWSy2S7gXw6f/MV+KF7BWW4nCy9UnJrrz34SMOoyNE7GhUIBO3aFkPFppcrMHm6RaeU4DDJubVYY+7ynYLPCy8sPp8zbN1/oSm4IpOKmClo9dbjWpG1bMMhLis5p4YYR9wPGx8BZg6eUXQF8/TVZ7qTBZFw8BY4X/560NWpXiQH+PEZFTDDEwr+f9yYyVy49g1LU85Z7386KzscHqYhJONwzDx05q4Cn9jGlaXXai5LgvhVsJy/rEEgWdDD178NC8ltdwPNySMoydU/XucKync59DJdSBZEitcuJFs4BuEnCkyeMsBDj8FTsST4QfiLmhbSidmWkOwekMCLGQjp3G9CUHQO8MOStssCz0ZNWSYmUEhDYsc+4y7cek2gzVqyDSZ/A5ETdCSLNhjpE5oGdjZBso5kUWZoqh2u3XGyybxmMjFI5tWsCUr9sPbBXXBlIIFSnmAu33SYWCL14Cs85dTlLMJsz7HrYCdEA6JepA4mnX1iAZ6mH3BIbC6N2cu2X7rzlpmX775jOVENc8TkpbRFeNPYduCiXEj+be/PvxNqSk8iMN79cKaaTV5AL76nMFduPnrl3nuWX7z9RqggURi0dQtHC8xCFQ5zhcTlwafoRGmd+OTA/OkJKgUbwse8TZ/RwOhsupyN6JKAZNRx8YvjqEwmXIdPXKwtNHr73ZE75cBoLSCZd7cwl0LNc+Pus2gk3CAUZdVBlxq9SCUvP9uxVqjAr4U/ExZGHLvRgKYt2CT9nyahsxw7kKEExz8ic8/xG2yIYV98tIEp7hGlhBsjSHxNnr6zFKd98cmWBtcEDCu7nIReFA+p/aACB0A1iqItIZluVHKxtUuomExJhgTNqezcw0Upsu30xeI/9u1CR5pbiQLww1xmJsHlMDMzMzMzMzMzRxRmZlE4D5C3yCcdpTTpnvY/s7MdWK1UGnncbkO5XKfA/SxX6fwr77IuqzvvijtUUuXlbjq/Yu99SG5vTYex88IJ7klRpR9l5Az+9Pf/jfc2iRn6v+yGB1OW2/JIJGbxsSYbvPLuF3ZBgVAR18TtLE1Lx2qksVDiamIABcnjjNVNxmkpzfTDw8JVSCaUa8wq0VA2+JjTLmdaHn/mVVI+UUMONgUnu3nbA0/rZ0hniRs75xogBUTfDcEkFVC646xLbnVWFW69/6mEaPx6/ZEn31R58LHnK6uk+xTeYxPMCckWss/hZ3j94mvvq8qX3vncuqhXGJxYcazpyQkgZumQI2uZXtStQrwByWyL8tdTbNQ5U2OIh86wBnqA6DiJA8YagmT9G2j7PY+GGWdfetuv/rquysXpvsdfzuQpTRMIJINe87Sc2j6krKVdO+Kki+kUkxkSg6F1xcy312x8OU7xYcvpCExQcyox7zQDSGtvtrt3Yy3R7yrxFv5RhfoH88wglZwwi2I2qTTQvY+9xIkxN3+jEKE4TOqL3NC6eDDAz9JMngfmFQeqINkWWyn+aFbiAeyPOuXSws7FiVmW2KbrCBXVMJO9Dz1NIQKMOpDc3po+Y+eCEwctoOv4Gy6cqXxTB5KZX8+9/lHsMyZsLKFlHIsYlGfsBAUdDcoo99fvSGOFupA88lhuPFRiYpVWJiTDLXtMQU86YZQaKeFOVdC1HdmTjeM71t+5IBnRC1pWuKaIG8dzXSRw7Ux2ILl4omdrnCtwHZ37239sAHStJaqHn6QfGay0oYhTOZWHjB5PKUcjHnbChbGLhyCZ9+Mt7hHnw982UXNUdoj3P1QJvQx6xU0P9wPXuNeB5Gdf+6j2CAI1IHloXeZ/x0PPLjlwzdzRFW5vuv2BVQmSVYI9ZTKjDO/tskLJsKlCMoYXQGVPxLzT2HxYNg2R66wLEpdgAOBJSGZmYY7gUO2O7WMBaIBAgq4W369AsmknbunGA3U8CyS3t6bD2HnhRGDD6y5OwlrLZJEUTvQh+ZDjzveXbcoyyEYQ6eUaixrxlxKrfRc10b+t0VJXdnOksdqQPMZYTHOScMu9T/7AIHAVkln9pB/Fy1SgOIYaJ7ToGOciVZSU85NDvvIgWRCyIuefzQ/J8YD5HEJwUT05WrT/5EwYs1N5yG/z1I4UcdqGIBl4SD1oH4emfaGD//Hm1wQFhyolJnT47w12nAWSOZEwYBZInrou5r9KfF4klyyKgy36ITyuoQaS1dQt8eg1NuVH35ye9i6yYt2jT73F4TAlNx5IlKezQ7JsXzRpH5L1HHZVBsffxGBCVO3i+xVIZht1Xp8Rkhsi12HsXHASRGd0YsI//r81Pneym52/gF+uRPxAKscjQrj4WAX2ImrV2LtES0vGRyxCYDnSWC1IHmGsf62/I+llpv/wIHAVkqWOZGuQzXY4FZyE9ivUPZ1ry2XpgmSVHoPTywXJdV2LHSDG+N1CMuep1sWp9XpiRG1INu1A8rmX31Fesn4kKSvfrNIXMlN5yBBRsPzZr3dZVMKzrgQvLhvxmZIDg2TKDUjGfzHe7BeQGILkxrrAg+/i+tNwhT45vxkJ/HzwyZd6E3QtSE5kO5AsVTyZYNMAIkbreYu+Y6daDviZC5JNkr9bdxi9IkmpnISIMDiEK6kjz0lsLy+BZDDTh2SKO5kjk2lAcmdrGoydKw+aiwhJOlAdHWYO3T+Sf/EIAi0+VuK9yVIX2XRPKbEyoaRpRxqrBckjjGUU6msVTVd+4BrmOZZOrNsoTn7khgDRL4AcqBCUDiTTEYK34nWzQDInUmP9R8HRiY4K58OjxSG5MVapZjEAlQLIDUhGyo6Bd+PezQXJGEItepG+dh1DsFQ/U3mocVxe3LY1bmqUetI5r9RUC1eAZQBSDdWvw8VFQv8mc8t9T/nA0fT6kCwFG8BGHFmRRl9lQFDTHhKDxrrAG/1CwGh8lmKFSQ1qdGEArG58B4ylQrgJD1x49d2GgLJTIdnWJ/9i8kCoItv2RdnuS5cqmE9T5LrrMnpiHvRm3GLoXrnkP/9r01yUNSuVjA8Sq2cT5uKIQo8HyQbyRTXvlryZVQeSIZ8lcNyHtmYqYxtwguGOWIic5EZCGKgTSiNX//DZuCi32TMH0ps7cdhFwifXgrEEDK/mHatyCmK51fjHv/tPgkDsTg3igNrHkcZKnt4a3XSRsVYQfB5prIg3wa7K3CFdpRUIyXyvqC1EaGiufFyorIYFHSu72suWpbG7xJPGKZTt9AxO0rIipZRv9Bo9Ip1ZkMynVOmWdV/vuPXQmHxjrFBWgXbe7/goXFdh+5Bc9xiRyuRB+8Qx6kMyBQdX8i0monSk/Ro8dFU41nFihrRVdZ5K21EByQChSsc++LE4UR9ZphH7getQIWJgm6dIvyhQBH0xaKwrdl4qJx19TEtaEcH7oakK8OBhZqvz+BOXXv+AdzuQHOMyvEIQqAIz2TLY4KkJNEWuuy7rDebhA3uiPnXNZav0AwKTzrBYAqZN2nPcx4NkxycDMTJMtQPJeQSYh7ZmKmOHKNcYi5gC4Ty/LfJpsXXAq1lIz24ORqqTvZ68pei+cSqXMBa57Yzlyn3ATznrJbSjjtWp5J2PNBbqVObYrtKK/S6ZAdtRT/w8R7ebS16YWHzp9ntIjFxoyqxeMg8FAKbysE/MXqbxjN9MO7rLvt02YpaWvNuhJcy+LjWSAv2Pgugg+ftZLnklXLxGMgQnNc7KMlImOSOBnLQfmxiCLuvNvTXzM7ZtCkOLRtA7p8C58B3jVEESXetnypc8FsqlP3YAMRh/rEkaf6yv2DsDTymiOIz+O4GAoiABSEgSkIJ4hCIkFS8EoRRSFsgD5YFUQCQEhYJQAIT+iE4+1s/s7LjWbHdxOLh73+zsCL7m3pnviFUhs0dyEBHhLjY7/f5WK2Iks4qbZZYZERFh65d7RH9LjGQREREjWURERIxkERERI1lERESMZBEREdmRSOYdRHphAq8nVuFgJlMqshm8dUqHbePBlHKk22H6MW+sShtcCeVTFCMw2BVERMRIpmO2On8oFqBziu6eQRc0YgPmG4VfHFahliHzNBnRztPYPkG/FRqA6WNqaWI7KTWkram2avNzDEREpBtGMj37pCaFbS16hlpVP0GtcZ+b+SM5zZdp7RYRkW4YyZSmYllIAG8vklGGRUQfx071ltA7T/F1NP7pZ6ZdOQcPfgvjGMFJRxgxHHFbIpklaDr9OcPzl2+WnZc01VFgy005i+2ZBJqHSWLm47GvPgw6ihmIiIj0ieR496iWJaKQ6s8Yydhn7zx4BhFU0AiNZ5tM5QyDEwK6AoQ8/OeAbzF59uIeB0dsx8dAcnNado7ZkEbhwjGJZK6cM7DEHTUh1xxvBAvRnCSmvOgUoxdFGIDd6NOXXzWSyW8ctAxERET6RPLB4QfuTRkgJ0FIMmMkc/PKDTFUrSzfXY1kDEjRw5GLRCmDMHAYk6/R6h07da4uXMevlxJ5xgghoipCScuTaHTok9kIDTmA226SeHThmtX7/LSIiEifSCalkJShXsHhFbX7VveSRyM51lJAGMye7rpIRhSD0jhGOYTKBPBgL5mLj4CPdexo75DUBh5eYyk7JtHRSKadO946ERGRDpEc33WFHP3/kRyzLBy++zwRyYH3snDCsxkc5+iopp596wwyH/AMMnnz3iPGePHI6RrJON5jyBcREekQyTwMxa1kxsAKNvusNZLJ7KMnznB7WhMU4TmTLDVvHMl8nbefWyKZ2+J/F3DkJB8xz1+9vs/FRHfPc17rIpkxf2V/mujFFLv/cHHl2t1Mcv4Ll2/wyhNH1kjmxjpPgYmIiHSI5K8/frOHmjEsDt6y7RqBPMowQisgkK+RnEmCeTqSebBrNJLzdUJxecJb95+ui+QcfGnvNh+fLF7zwFdm+FOeGnv84hVr76uRzFL8959/cjCDHMwl5frff/zGInmNZPJb2XMrIiJGsmT9+fjp840HE8zcZNcZdpTpQhk9LUsC/vP+ba+OaQAAAAAE9W9tCU82OgCAkgFAyQCAkgFAyQCAkgFAyQCAkgFAyQCAkgHgoWQAUDIAoGQAUDIAoGQAUDIAoGQAIFuSF99xMhs6AAAAAElFTkSuQmCC) #### Advanced: Create a release train with additional environments Large and complex enterprises sometimes require additional layers of validation before deployment. Learn how to add these checks with dbt. Are you sure you need this? This approach can increase release safety, but creates additional manual steps in the deployment process as well as a greater maintenance burden. As such, it may slow down the time it takes to get new features into production. The team at Sunrun maintained a SOX-compliant deployment in dbt while reducing the number of environments. Check out [their Coalesce presentation](https://www.youtube.com/watch?v=vmBAO2XN-fM) to learn more. In this section, we will add a new **QA** environment. New features will branch off from and be merged back into the associated `qa` branch, and a member of your team (the "Release Manager") will create a PR against `main` to be validated in the CI environment before going live. The git flow will look like this: [![git flow diagram with an intermediary branch](/img/best-practices/environment-setup/many-branch-git.png?v=2 "git flow diagram with an intermediary branch")](#)git flow diagram with an intermediary branch ##### Advanced prerequisites * You have the **Development**, **CI**, and **Production** environments, as described in [the Baseline setup](https://docs.getdbt.com/guides/set-up-ci.md). ##### 1. Create a `release` branch in your git repo As noted above, this branch will outlive any individual feature, and will be the base of all feature development for a period of time. Your team might choose to create a new branch for each sprint (`qa/sprint-01`, `qa/sprint-02`, etc), tie it to a version of your data product (`qa/1.0`, `qa/1.1`), or just have a single `qa` branch which remains active indefinitely. ##### 2. Update your Development environment to use the `qa` branch See [Custom branch behavior](https://docs.getdbt.com/docs/dbt-platform-environments.md#custom-branch-behavior). Setting `qa` as your custom branch ensures that the IDE creates new branches and PRs with the correct target, instead of using `main`. [![A demonstration of configuring a custom branch for an environment](/img/docs/dbt-platform/platform-configuring-dbt-platform/dev-environment-custom-branch.png?v=2 "A demonstration of configuring a custom branch for an environment")](#)A demonstration of configuring a custom branch for an environment ##### 3. Create a new QA environment See [Create a new environment](https://docs.getdbt.com/docs/dbt-platform-environments.md#create-a-deployment-environment). The environment should be called **QA**. Just like your existing Production and CI environments, it will be a Deployment-type environment. Set its branch to `qa` as well. ##### 4. Create a new job Use the **Continuous Integration Job** template, and call the job **QA Check**. In the Execution Settings, your command will be preset to `dbt build --select state:modified+`. Let's break this down: * [`dbt build`](https://docs.getdbt.com/reference/commands/build.md) runs all nodes (seeds, models, snapshots, tests) at once in DAG order. If something fails, nodes that depend on it will be skipped. * The [`state:modified+` selector](https://docs.getdbt.com/reference/node-selection/methods.md#state) means that only modified nodes and their children will be run ("Slim CI"). In addition to [not wasting time](https://discourse.getdbt.com/t/how-we-sped-up-our-ci-runs-by-10x-using-slim-ci/2603) building and testing nodes that weren't changed in the first place, this significantly reduces compute costs. To be able to find modified nodes, dbt needs to have something to compare against. Normally, we use the Production environment as the source of truth, but in this case there will be new code merged into `qa` long before it hits the `main` branch and Production environment. Because of this, we'll want to defer the Release environment to itself. ##### Optional: also add a compile-only job dbt uses the last successful run of any job in that environment as its [comparison state](https://docs.getdbt.com/reference/node-selection/syntax.md#about-node-selection). If you have a lot of PRs in flight, the comparison state could switch around regularly. Adding a regularly-scheduled job inside of the QA environment whose only command is `dbt compile` can regenerate a more stable manifest for comparison purposes. ##### 5. Test your process When the Release Manager is ready to cut a new release, they will manually open a PR from `qa` into `main` from their git provider (e.g. GitHub, GitLab, Azure DevOps). dbt will detect the new PR, at which point the existing check in the CI environment will trigger and run. When using the [baseline configuration](https://docs.getdbt.com/guides/set-up-ci.md), it's possible to kick off the PR creation from inside of the Studio IDE. Under this paradigm, that button will create PRs targeting your QA branch instead. To test your new flow, create a new branch in the Studio IDE then add a new file or modify an existing one. Commit it, then create a new Pull Request (not a draft) against your `qa` branch. You'll see the integration tests begin to run. Once they complete, manually create a PR against `main`, and within a few seconds you’ll see the tests run again but this time incorporating all changes from all code that hasn't been merged to main yet. --- ### Getting started with the terminal [Back to guides](https://docs.getdbt.com/guides.md) dbt CLI Beginner Beginner [Menu ]() #### What is the terminal? The terminal (also called the command line, shell, or CLI) is a text-based interface for running commands on your computer. Many dbt tools — including dbt v2, dbt v1, and the dbt Wizard CLI — run from the terminal. You don't need to be a terminal expert to use dbt. This guide covers the basics. #### Open the terminal ##### macOS **Option 1: Spotlight** Press `⌘ Space`, type `Terminal`, and press Enter. **Option 2: Applications folder** Go to **Applications** → **Utilities** → **Terminal**. **Option 3: VS Code integrated terminal** In VS Code, press `` ⌃` `` (Control + backtick) to open a terminal panel directly in your editor. ##### Windows **Option 1: Command Prompt or PowerShell** Press `Win + R`, type `cmd` or `powershell`, and press Enter. **Option 2: Windows Terminal** Search for "Terminal" in the Start menu. Windows Terminal supports Command Prompt, PowerShell, and WSL side by side. **Option 3: VS Code integrated terminal** In VS Code, press `` Ctrl+` `` to open a terminal panel directly in your editor. Using dbt on Windows? dbt runs natively on Windows via PowerShell. For the best experience, consider using [WSL (Windows Subsystem for Linux)](https://learn.microsoft.com/en-us/windows/wsl/install). ##### Linux Open your distribution's terminal emulator — usually found in the applications menu, or press `Ctrl + Alt + T` on most desktop environments. #### Navigate your file system When the terminal opens, you're in a directory (folder). These commands help you move around: | Command | What it does | Example | | ------------- | ------------------------------------ | ---------------------- | | `pwd` | Print the current directory | `pwd` → `/Users/alice` | | `ls` | List files and folders (macOS/Linux) | `ls` | | `dir` | List files and folders (Windows) | `dir` | | `cd <folder>` | Change into a folder | `cd my-dbt-project` | | `cd ..` | Go up one folder | `cd ..` | | `cd ~` | Go to your home directory | `cd ~` | **Tips:** * Press **Tab** to autocomplete folder and file names — saves a lot of typing. * Press **↑ / ↓** to scroll through previous commands. * Press **Ctrl+C** to cancel a running command. #### Navigate to your dbt project Your dbt project is a folder on your computer containing a `dbt_project.yml` file. Before running any dbt commands, navigate into that folder: ```bash cd ~/path/to/your-dbt-project ``` Verify you're in the right place: ```bash ls # macOS/Linux — you should see dbt_project.yml dir # Windows ``` #### Run your first dbt command After installing dbt, you can run your first dbt command to verify it's installed and working. Make sure you're in your project folder before running any dbt commands: ```bash cd ~/path/to/your-dbt-project ``` Then run your first dbt command: ```bash dbt --version ``` You should see output similar to the following: ```bash dbt-fusion 2.0.0-preview.45 ``` .... and that's it! Congrats, you're ready to start using dbt in the terminal! 🎉 #### Common issues **`command not found`** The tool isn't installed or isn't on your PATH. Double-check the install instructions for [dbt v1](https://docs.getdbt.com/docs/local/install-dbt.md) or [dbt Wizard CLI](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md). **`Permission denied`** You may need to run the command with elevated permissions, or check that the file is executable. **Nothing happens after I type** Make sure you pressed **Enter** after the command. #### Next steps * [dbt Wizard quickstart](https://docs.getdbt.com/docs/dbt-ai/wizard-quickstart.md) — use the dbt Wizard CLI in your terminal * [Install dbt](https://docs.getdbt.com/docs/local/install-dbt.md) — set up dbt v2 or dbt v1 locally * [dbt commands reference](https://docs.getdbt.com/reference/dbt-commands.md) — all available dbt CLI commands --- ### How to use prompts for dbt Wizard [Back to guides](https://docs.getdbt.com/guides.md) dbt Wizard AI Best practices Beginner [Menu ]() #### Overview Learn how to write effective prompts for dbt dbt Wizard to generate accurate SQL, models, metrics, and macros. Each recipe is self-contained with its own realistic example. dbt dbt Wizard is an AI assistant that generates SQL, YAML, documentation, tests, semantic models, and macros based on your project's context. The quality of output depends on the clarity of your prompts. This cookbook provides independent recipes for common prompting tasks. Jump to any section that matches your current need. This cookbook covers the following topics: * [Prompt best practices](https://docs.getdbt.com/guides/prompt-cookbook?step=2) * [Generate SQL queries](https://docs.getdbt.com/guides/prompt-cookbook?step=3) * [Use what you already have](https://docs.getdbt.com/guides/prompt-cookbook?step=4) * [Create semantic models and metrics](https://docs.getdbt.com/guides/prompt-cookbook?step=5) * [Create reusable macros](https://docs.getdbt.com/guides/prompt-cookbook?step=6) * [Troubleshoot errors and issues](https://docs.getdbt.com/guides/prompt-cookbook?step=7) * [Conclusion](https://docs.getdbt.com/guides/prompt-cookbook?step=8) #### Prompt best practices Writing effective prompts is about giving dbt Wizard the right context and clear direction. Follow these principles: * [Provide rich context](#provide-rich-context) * [Break complex logic into smaller steps](#break-complex-logic-into-smaller-steps) * [State the business question, not just the output](#state-the-business-question-not-just-the-output) * [Be clear and explicit about the result](#be-clear-and-explicit-about-the-result) ##### Provide rich context In your prompt, include table names, column types, and example values to describe how they relate to each other. Include the following: * Table relationships (such as `orders` connects to `customers` on `customer_id`) * Data types (such as `signup_date` is a timestamp) * Sample values (such as `plan_type` can be "monthly" or "annual") tip The following example uses SQL terminology (like data types and joins) because it's generating a SQL query. However, the principle of providing rich context applies to all dbt Wizard tasks—whether you're generating macros, documentation, or YAML configurations. **Example: Santi's neighborhood café** Let's say you run a neighborhood café and folks get a free drink after 10 visits: **Without rich context** (vague): ```text I need a query using customers, subscriptions, and activity tables to see weekly regulars. ``` **With rich context** (specific): ```text Context: I run a café loyalty program where customers earn a free drink after 10 visits. Tables and relationships: - customers (customer_id INT or integer, name STRING, email STRING, signup_date TIMESTAMP) - subscriptions (subscription_id INT, customer_id INT, plan_type STRING, start_date DATE, end_date DATE) - Joins to customers on customer_id - plan_type values: "monthly", "annual", null for non-subscribers - activity (activity_id INT, customer_id INT, visit_date DATE, visit_count INT) - Joins to customers on customer_id - visit_count tracks cumulative visits (resets after redemption) Business question: Show me which customers visit weekly (3+ times per week for 4+ weeks) and compare conversion rates: do high-frequency punch-card users convert to our 'beans of the month' subscription at a higher rate than casual visitors? ``` **Why it works:** The AI now knows exact data types, how tables relate, what values to expect, and the specific business logic (3+ visits/week defines "regulars"). ##### Break complex logic into smaller steps Common misconception Many users try to ask for everything at once in a single prompt. Breaking your request into smaller, sequential steps consistently produces better results. For multi-part tasks, write them as a sequence of clear instructions. dbt Wizard handles step-by-step logic better than complex, all-in-one requests. **Example:** ```text 1. Filter the dataset to active users in the last 90 days. 2. Calculate their average session duration. 3. Join to subscription data and group by plan tier. ``` **Why this works:** Each step is clear and actionable. You can always iterate on your prompt to refine results — start simple, then build complexity. ##### State the business question, not just the output Describe the decision or insight the query supports, and avoid only technical-like prompts. This means, instead of "count users", you can say "count active users per week to analyze engagement trends." **Example: The sneaker drop** Let's say you run an online sneaker shop and just launched a new feature: customers can view 3D previews of sneakers before buying. ```text We launched a 3D preview feature with our latest limited-edition sneaker drop. Did customers who used the 3D preview convert to buyers at a higher rate than those who only saw photos? Show me weekly conversion rates: browsers who became buyers, segmented by whether they used the 3D preview. If preview users convert 20%+ higher, we'll add 3D to all products. If not, we'll improve the feature before expanding. ``` **Why it works:** You've described the feature, the behavior you're measuring, specific success criteria (20%+ lift), and the decision you'll make based on results. ##### Be clear and explicit about the result Define the expected output clearly. Mention the expected columns in the final result and state whether results should be sorted, limited, or filtered. **What to specify:** * Expected column names and formats * Sort order and any limits (for example, "top 10 products by revenue") * Output format examples (for example, "`conversion_rate` as a percentage") **Example: The fitness challenge** In this example, you run a fitness app with a 2-week challenge, Kimiko's kettlebell challenge. ```text Give me a weekly trend with the date, active folks, and a simple 'engagement per person.' Then a summary by launch week with 'trial starts,' 'upgrades in 30 days,' and an 'upgrade rate' as a percentage. Each week, show active challengers and total workouts. By challenge start week, show how many upgraded to paid within 30 days and what their average workouts looked like. ``` **Why it works:** Specific metrics that are ready to present. #### Generate SQL queries Let's say you want to build a query to find top-spending customers. ```text Context: I have two tables: - customers (customer_id, name, email) - orders (order_id, customer_id, order_total, order_date) Relationship: orders.customer_id connects to customers.customer_id Business question: Show me the top 10 customers by total spending in 2024. Output: - customer_id - customer_name - total_spent - order_count Sort by total_spent descending, limit to 10 rows. ``` **What dbt Wizard generates:** ```sql select c.customer_id, c.name as customer_name, sum(o.order_total) as total_spent, count(o.order_id) as order_count from {{ ref('customers') }} c inner join {{ ref('orders') }} o on c.customer_id = o.customer_id where year(o.order_date) = 2024 group by c.customer_id, c.name order by total_spent desc limit 10 ``` **Why it works:** * Clear context about tables and their relationship * Specific business question with a defined time period * Explicit output requirements and sorting logic **Pro tip:** Start simple, then iterate. If dbt Wizard's first attempt isn't perfect, no worries! Refine your prompt with more specific details and let dbt Wizard do its magic, it usually gets there in the end ✨ #### Use what you already have You don't need to write everything from scratch. Pull in documentation, definitions, and sample data you already have—it helps dbt Wizard understand your specific business context. dbt Insights integration When using dbt Wizard in [Insights](https://docs.getdbt.com/docs/explore/dbt-insights.md), you can easily cross-reference between dbt Wizard's generated SQL and metadata from [dbt Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md). This embedded integration makes it seamless to access documentation, definitions, and sample data while building queries. ##### Define your business rules Instead of just saying "active customer," explain the rule: ```text Active customer = at least one paid purchase in the last 90 days, excluding refunds Net revenue = gross sales minus discounts and returns ``` **Pull from:** Metrics glossaries, KPI catalogs, product requirement docs, data dictionaries ##### Show sample values Give dbt Wizard examples of what the data actually looks like, especially edge cases: ```text Order statuses: - `customer_id: C-12, created_at: 2025-05-03T09:07:00Z, status: 'completed'` - `customer_id: C-14, created_at: 2025-05-03T09:02:00Z, status: 'cancelled'` - `customer_id: C-13, created_at: 2020-01-02T06:40:00Z, status: 'pending'` ``` **Pull from:** Data profiling reports, QA test datasets, BI dashboard filters ##### Start with a draft, refine later Frame your model first, then iterate. Start with a clean outline that gets the basic structure right: ```text From stg_orders and dim_customers, draft a minimal model with order_id, customer_id, order_date, net_revenue = gross - coalesce(discount, 0), and join to dim_customers on customer_id. Filter to the last 30 days for preview only. ``` **Pull from:** Source-to-target mapping sheets (join keys and transformations), data dictionaries (primary and foreign keys) #### Create semantic models and metrics Fast-track your semantic layer strategy with AI-generated YAML using dbt Wizard. dbt platform provides built-in generation buttons that automatically [generate code](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md), [documentation](https://docs.getdbt.com/docs/build/documentation.md), [data tests](https://docs.getdbt.com/docs/build/data-tests.md), [metrics](https://docs.getdbt.com/docs/build/metrics-overview.md), and [semantic models](https://docs.getdbt.com/docs/build/semantic-models.md) for you with the click of a button in the [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-studio-ai.md), [Canvas](https://docs.getdbt.com/docs/platform/build-canvas-copilot.md), and [Insights](https://docs.getdbt.com/docs/explore/dbt-insights.md). These features understand your model's structure and generate YAML in the correct location. **How to generate semantic models:** 1. Navigate to the Studio IDE and select a SQL model file in the **File explorer** 2. In the **Console** section (under the **Editor**), click the **dbt Wizard** icon to view AI options 3. Select **Semantic model** to create a semantic model based on your SQL model 4. Review and refine the generated YAML as needed You can also use dbt Wizard to generate documentation, tests, and metrics. These built-in features automatically understand your model's columns, data types, and relationships, which means you don't need to manually describe your schema or copy-paste between file types. **Typical workflow:** 1. Build your SQL model using dbt Wizard conversational prompts 2. Use built-in buttons to add documentation, tests, and semantic models 3. Refine the generated YAML as needed For more details, check out the [dbt Wizard](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) docs. #### Create reusable macros In this section, we'll look at how to create reusable macros using dbt Wizard. * [Turn repetitive code into reusable logic](#turn-repetitive-code-into-reusable-logic) * [Lower the barrier to entry](#lower-the-barrier-to-entry) * [Accelerate complex logic design](#accelerate-complex-logic-design) ##### Turn repetitive code into reusable logic A junior analyst keeps copy-pasting CASE statements across models. **What to give dbt Wizard:** ```text Turn this CASE pattern into a reusable macro: CASE WHEN amount >= 1000 THEN 'high' WHEN amount >= 500 THEN 'medium' ELSE 'low' END Macro requirements: - Name: categorize_amount - Parameters: column name, high threshold (default 1000), medium threshold (default 500) - Include docstring with usage example - Handle null values by returning 'unknown' ``` **Why it works:** Clear input (the CASE statement), clear requirements, clear output expectations. ##### Lower the barrier to entry You need a macro but don't know Jinja syntax well. **What to ask dbt Wizard:** ```text I need a macro that calculates the number of days between two date columns, excluding weekends. Parameters: - start_date_column (required) - end_date_column (required) Include a docstring explaining how to use it. ``` **Outcome:** dbt Wizard generates proper Jinja syntax, handles parameters, and includes documentation. You learn Jinja patterns while getting working code. ##### Accelerate complex logic design This is best for advanced users who are comfortable with Jinja. **What to ask dbt Wizard:** ```text I need a macro that builds a grouped aggregation with optional filters. Parameters: - relation (the model/table to query) - group_by (list of columns to group by) - metrics (list of columns to aggregate) - where (optional filter condition) Include defaults and guardrails for empty lists. Add a docstring with parameter descriptions and usage example. ``` **Why this works:** You've outlined the interface (parameters) and edge cases (empty lists), letting dbt Wizard handle the Jinja boilerplate while you focus on design. This approach accelerates iteration so you can refine the structure without getting stuck in syntax details. #### Troubleshoot errors and issues dbt Wizard acts as a fast, context-aware reviewer for failing SQL and macros. It reads errors, inspects your query structure, and suggests minimal fixes. Troubleshooting with dbt Wizard gives you: * Faster diagnosis by using plain-language translation of errors with likely root causes * Safer fixes by biasing toward small, targeted changes * Better learning by generating explanations you can paste into docs or PR descriptions ##### Troubleshoot errors When something breaks, give dbt Wizard the error message, your code, and what you expected to happen. Here are a couple of examples to show you how to use dbt Wizard to troubleshoot errors. **Example: SQL error** ```text Error: "SQL compilation error: Column 'product_name' must appear in GROUP BY" Query: SELECT product_id, product_name, SUM(quantity) as total_quantity FROM inventory GROUP BY product_id Warehouse: Snowflake Expected: Group by product and show product name. What's wrong and how do I fix it? ``` **Example: Macro not working** ```text This macro should calculate discount but returns wrong values: {% macro calculate_discount(amount, rate) %} {{ amount }} * {{ rate }} {% endmacro %} When I call {{ calculate_discount(100, 0.1) }} I expect 10 but get an error. Show me the rendered SQL from target/compiled and explain what's wrong. ``` **Tip:** Include your warehouse type (Snowflake, BigQuery, Databricks and so on.) — this is because the syntax can vary across data platforms. #### Conclusion Congrats, you've now learned some tips on how to create and use prompts for dbt dbt Wizard 🎉! You can: * Boost your prompting skills by providing rich context and stating clear business questions. Applicable for SQL, macros, documentation, tests, metrics, and semantic models. * Amplify your workflow by using existing documentation and project context * Generate Jinja macros to build more scalable and maintainable systems * Troubleshoot your code to diagnose issues fast and apply safe, explainable fixes ##### Quick reference checklist When writing prompts for dbt dbt Wizard: * ✅ Provide rich context: Table names, columns, data types, relationships, sample values * ✅ Break down complex logic: Write multi-part queries as a sequence of steps * ✅ State the business question: What decision or insight you're supporting, not just "write a query" * ✅ Be clear and explicit: Expected columns, sort order, filters, and output format For troubleshooting: * ✅ Include complete error messages: Full warehouse error with line numbers * ✅ Show the failing code: Both the dbt model and compiled SQL (from `target/compiled/`) * ✅ Provide sample data: Representative rows that trigger the issue * ✅ State your warehouse: Snowflake, BigQuery, Databricks, etc. ##### Next steps Start with one task—automating documentation, generating a test, or refactoring a model—and build the habit from there. The more you use dbt Wizard, the more you'll discover ways to accelerate your analytics engineering workflow. Check out the following docs to learn more about how to use dbt Wizard: * [About dbt Wizard](https://docs.getdbt.com/docs/platform/wizard-platform.md) * [dbt Wizard in Studio IDE](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md) * [Generate resources](https://docs.getdbt.com/docs/dbt-ai/wizard-ide.md#generate-resources) * [Build visual models with dbt Wizard](https://docs.getdbt.com/docs/platform/build-canvas-copilot.md) * [Build queries in dbt Insights](https://docs.getdbt.com/docs/explore/navigate-dbt-insights.md) --- ### Hybrid development with dbt platform and dbt v2 [Back to guides](https://docs.getdbt.com/guides.md) dbt v2 dbt platform Local development Beginner [Menu ]() #### Introduction Hybrid dbt deployments are becoming increasingly common. dbt v2 adopters are frequently working in several places at once: in the dbt platform for production runs and IDE-based development, and on their local machine using the dbt platform CLI or the dbt VS Code extension. These paths are fully supported for dbt platform users. Keeping the environments in sync across credentials, environment variables, and engine versions is one of the first operational challenges teams encounter. This guide walks through credentials, environment variables, dbt v2 versions, and Mesh or deferral, with concrete, copy-paste-ready steps to keep everything aligned. #### Prerequisites * You have a dbt platform account with at least one project using dbt v2. * You have either the [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) or the [dbt VS Code extension + local dbt](https://docs.getdbt.com/docs/local/install-dbt.md) installed. #### 1. Managing credentials How you authenticate to your data warehouse locally depends on which self-hosted tool you use: * [dbt platform CLI](https://docs.getdbt.com/guides/dbt-platform-local-workflow.md?step=3#dbt-platform-cli): For a CLI-only development experience (without the dbt VS Code extension), use the dbt platform CLI with dbt v2 set as your platform release track. Warehouse credentials are managed centrally in dbt platform and passed through automatically — no `profiles.yml` required. * [dbt VS Code extension](https://docs.getdbt.com/guides/dbt-platform-local-workflow.md?step=3#dbt-vs-code-extension-profilesyml-required): For IDE-based local development, the dbt VS Code extension runs dbt v2 and its LSP features in a local process. This path requires a `profiles.yml` to connect directly to your warehouse. ##### dbt platform CLI The [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) is the lowest-friction path for dbt platform users who want a self-hosted CLI-only workflow without VS Code. It authenticates using your dbt platform session, and your warehouse credentials are managed centrally in dbt platform and passed through automatically. For detailed installation instructions, refer to [Install the dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md?version=1.10). The dbt platform CLI is installed from your local command prompt. The configuration file downloaded from your dbt platform **Account settings** will facilitate the connection and authentication with your existing credentials. This is the lowest-friction path for teams that don't need full IDE integration locally. ##### dbt VS Code extension (profiles.yml required) The dbt VS Code extension runs dbt v2 and its language server in a local process and connects directly to your warehouse. For this reason, you need a `profiles.yml` for local extension development sessions. Download your [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md) from your dbt platform **Account settings** and dbt v2 attempts to hydrate non-sensitive credential metadata from dbt platform automatically. If you get access to a new project, re-download the `dbt_cloud.yml` file before working on it locally. To switch between projects already listed in your file, update [`context.active-project`](https://docs.getdbt.com/reference/dbt_cloud.yml.md#update-or-switch-projects). To avoid manually recreating your warehouse configuration, use `dbt init`. ```shell dbt init ``` dbt v2 pulls down fields such as your **username**, **role**, **warehouse**, **database**, and **schema**, but never sensitive values like passwords or tokens. If your authentication mechanism is passwordless (such as `externalbrowser` or SSO-based OAuth), dbt v2 configures that too, so you can work without storing secrets locally. note This hydration happens once during initial setup and does not stay in sync automatically. When your warehouse configuration changes in dbt platform, run `dbt init` again to refresh your local `profiles.yml`. The dbt VS Code extension first-time setup flow prompts you through this process, so you usually don't need to run `dbt init` manually. Coming soon We're working on a solution that lets you develop locally in the dbt VS Code extension while you manage credentials entirely in dbt platform, without a local `profiles.yml`. We'll update this page when that ships. #### 2. Managing environment variables Environment variables you set in dbt platform apply to production runs and the Studio IDE sessions. For local development, you manage environment variables separately. ##### dbt platform CLI When you use the dbt platform CLI, dbt platform injects the same environment variables you use in production into your dbt platform CLI session. You don't need extra setup. ##### VS Code extension (.env file) The dbt VS Code extension runs dbt v2 as a local process, so environment variables from dbt platform are not automatically available. Instead, use a [`.env` file](https://dotenvx.com/docs) at the root of your dbt project: ```shell # .env DBT_MY_DATABASE=my_database DBT_MY_SCHEMA=my_dev_schema DBT_TARGET_SCHEMA=analytics_dev ``` dbt v2 and the dbt VS Code extension automatically load values from this file. You can also view and override individual environment variables from the extension's settings UI. Reference these variables in your `profiles.yml` or elsewhere in your dbt project using the [`env_var` Jinja function](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md): ```yaml # profiles.yml my_profile: target: dev outputs: dev: type: snowflake account: my_account database: "{{ env_var('DBT_MY_DATABASE') }}" schema: "{{ env_var('DBT_MY_SCHEMA') }}" ``` For a full walkthrough of `.env` file usage and variable precedence, see [Environment variables](https://docs.getdbt.com/docs/build/environment-variables.md) for more information on where to find your configured variables and [Set environment variables locally](https://docs.getdbt.com/docs/configure-dbt-extension.md?version=2.0#set-environment-variables-locally) for local configuration instructions. ##### Keeping platform and local variables in sync Environment variables in dbt platform and a local `.env` file don't sync automatically. Commit a `.env.example` file with the list of required variables to your repo. Each developer copies it locally and fills in their own values and their copy is never committed. ```shell # .env.example (committed to version control) DBT_MY_DATABASE= # Your development database name DBT_MY_SCHEMA= # Your personal dev schema, for example dbt_yourname DBT_TARGET_SCHEMA= # Target schema for dbt output ``` ```shell # Developer setup: copy the example and fill in your values cp .env.example .env ``` Do not commit .env dbt v2 and the dbt VS Code extension only load from a file named exactly `.env`, so each developer needs their own copy. Make sure `.env` is in your `.gitignore` so credentials are never committed. Running `dbt init` adds this automatically. ```shell echo ".env" >> .gitignore ``` When environment variables change in dbt platform (you add variables or rename values), update `.env.example` in the same pull request so local developers know to update their own `.env`. For teams with strict security requirements Consider a script that fetches variables from your secrets manager (for example, AWS Secrets Manager or 1Password) and writes them to `.env` at the start of a session, instead of storing values in a file long term. #### 3. Managing dbt v2 versions The **v2 Stable** release track on dbt platform updates continuously as dbt v2 ships new releases. If your local version falls behind, you might see inconsistent behavior. The same query could compile differently locally than in production, or a feature might exist in dbt platform but not in your local binary. Stay current to avoid these mismatches. ##### Versions on the dbt platform On dbt platform, dbt v2 follows a versionless release track model. The default release track is **v2 Stable**, which always runs the most recent stable release. For details on release tracks and their stability levels, see [dbt v2 releases](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md#dbt-v2-release-tracks). ##### Versions installed locally By default, the dbt v2 [installation script](https://docs.getdbt.com/docs/local/install-dbt.md) installs the latest stable release, the same version that ships with the **v2 Stable** release track on dbt platform: ```shell # macOS / Linux curl -fsSL https://downloads.getdbt.com/install/dbt-fusion.sh | sh ``` To update your self-hosted installation to the latest stable release at any time: ```shell dbt system update ``` To check your current version: ```shell dbt --version ``` ##### Keeping versions in sync: dev containers (recommended) Use a [VS Code dev container](https://code.visualstudio.com/docs/devcontainers/containers) for the most reliable match between local dbt v2 versions and dbt platform. A dev container runs your environment inside a Docker image that rebuilds at the start of each session and performs a fresh dbt v2 install each time, so everyone on your team uses the same version as dbt platform without manual updates. Our friends at Brooklyn Data have published a ready-to-use dbt v2 dev container: * **Dev container template:** [brooklyn-data/dbt-fusion-devcontainer](https://github.com/brooklyn-data/dbt-fusion-devcontainer) * **Blog post:** [Why you should use dev containers with dbt v2](https://www.brooklyndata.co/ideas/2025/06/11/why-you-should-use-dev-containers-with-dbt-fusion) To get started with their template: ```shell # Clone the devcontainer template into your project curl -fsSL https://raw.githubusercontent.com/brooklyn-data/dbt-fusion-devcontainer/main/setup.sh | sh ``` Then open your project in VS Code and select **Reopen in Container** when prompted. VS Code builds the image and installs the latest stable dbt v2 release automatically. Coming soon We're introducing additional dbt v2 release tracks on dbt platform beyond **v2 Stable**. When they're available, we'll update this guide with steps to pin your dev container to a specific track. ##### Without dev containers: update at the start of each session If dev containers aren't an option for your team, run `dbt system update` at the start of each development session instead. That installs the latest stable release, the same version as the **v2 Stable** track on dbt platform, so your local binary stays current: ```shell dbt system update && dbt debug ``` Pinning to a specific version number does not work long term here: the **v2 Stable** track on dbt platform keeps advancing, and a pinned self-hosted installation falls behind. Aim to stay on **v2 Stable** instead of locking to one release. To make this easy to remember, add a `dev` target to your project's `Makefile`: ```makefile # Makefile .PHONY: dev dev: dbt system update dbt debug ``` Then developers start their session with: ```shell make dev ``` You can also document this convention in your project's `CONTRIBUTING.md` so it's part of your onboarding checklist. *** #### 4. dbt Mesh and deferral If your project uses [dbt Mesh](https://docs.getdbt.com/docs/mesh/about-mesh.md), referencing models from other dbt projects via cross-project refs, dbt v2 handles this automatically during development when a [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md) is present. ##### How it works When dbt v2 detects upstream projects defined in your `dependencies.yml`, it downloads the publication artifact for each upstream project from dbt platform before resolving cross-project refs. Then `ref('upstream_project', 'model_name')` works locally without manual setup. Your logs include lines such as the following while dbt v2 resolves cross-project refs: ```text Downloading publication artifact for <upstream_project> (resolving cross-project refs) Downloaded publication artifact for <upstream_project> to <path> (resolving cross-project refs) ``` dbt v2 caches downloaded publication artifacts for up to one hour, so subsequent runs in the same session skip the download and resolve refs from the local cache. Auto-deferral is also on by default. When a [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md) is present, dbt v2 defers to your project's configured deferral environment, so you build only modified models and their downstream dependencies while the rest resolve against the production state. ##### Disabling deferral * **In the VS Code extension:** Disable auto-deferral in the extension settings. Search for `Dbt > Flag: Defer` and uncheck the option: [![dbt VS Code extension deferral settings](/img/fusion/vsce-defer-settings.png?v=2 "dbt VS Code extension deferral settings")](#)dbt VS Code extension deferral settings * **On the CLI:** Pass `--no-defer` to any command to skip both deferral and the publication artifact download: ```shell dbt run --no-defer dbt compile --no-defer ``` #### Reference table The following table summarizes the key differences between the two development paths covered in this guide: | Area | dbt platform CLI | dbt VS Code extension | | ------------------------- | ------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------- | | **Credentials** | Managed through your dbt platform session, no `profiles.yml` needed | `profiles.yml` required; use `dbt init` to hydrate from dbt platform | | **Environment variables** | Same env vars as in dbt platform automatically | Use a `.env` file at the project root | | **Version management** | `dbt system update` to stay current | Dev container recommended for automatic sync | | **dbt Mesh / deferral** | Auto-enabled when [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md) present; `--no-defer` to disable | Auto-enabled when [`dbt_cloud.yml`](https://docs.getdbt.com/reference/dbt_cloud.yml.md) present; toggle off in extension settings | #### Related docs * [Install dbt v2](https://docs.getdbt.com/docs/local/install-dbt.md) * [dbt platform CLI installation](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) * [dbt v2 releases and release channels](https://docs.getdbt.com/docs/dbt/dbt-releases.md) * [About profiles.yml](https://docs.getdbt.com/docs/local/profiles.yml.md) * [Environment variables (local)](https://docs.getdbt.com/docs/local/configure-environment-variables.md) * [VS Code dev containers](https://code.visualstudio.com/docs/devcontainers/containers) * [dbt Mesh overview](https://docs.getdbt.com/docs/mesh/about-mesh.md) * [Deferral in dbt](https://docs.getdbt.com/docs/platform/about-defer.md) --- ### Integrate with dbt Semantic Layer using best practices [Back to guides](https://docs.getdbt.com/guides.md) Semantic Layer Best practices Advanced [Menu ]() #### Introduction To fit your tool within the world of the Semantic Layer, dbt Labs offers some best practice recommendations for how to expose metrics and allow users to interact with them seamlessly. This is an evolving guide that is meant to provide recommendations based on our experience. If you have any feedback, we'd love to hear it! 📹 Learn about the dbt Semantic Layer with on-demand video courses! Explore our [dbt Semantic Layer on-demand course](https://learn.getdbt.com/courses/semantic-layer) to learn how to define and query metrics in your dbt project. Additionally, dive into mini-courses for querying the dbt Semantic Layer in your favorite tools: [Tableau](https://courses.getdbt.com/courses/tableau-querying-the-semantic-layer), [Excel](https://learn.getdbt.com/courses/querying-the-semantic-layer-with-excel), [Hex](https://courses.getdbt.com/courses/hex-querying-the-semantic-layer), and [Mode](https://courses.getdbt.com/courses/mode-querying-the-semantic-layer). ##### Prerequisites To build a Semantic Layer integration: * We offer a [JDBC](https://docs.getdbt.com/docs/dbt-apis/sl-jdbc.md) API and [GraphQL API](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md). Refer to the dedicated [Semantic Layer API](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md) for more technical integration details. * Familiarize yourself with the [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) and [MetricFlow](https://docs.getdbt.com/docs/build/about-metricflow.md)'s key concepts. There are two main objects: * [Semantic models](https://docs.getdbt.com/docs/build/semantic-models.md) — Nodes in your semantic graph, connected via entities as edges. MetricFlow takes semantic models defined in YAML configuration files as inputs and creates a semantic graph that you can use to query metrics. * [Metrics](https://docs.getdbt.com/docs/build/metrics-overview.md) — Can be defined in the same YAML files as your semantic models, or split into separate YAML files into any other subdirectories (provided that these subdirectories are also within the same dbt project repo). ##### Connection parameters The dbt Semantic Layer APIs authenticate with `environmentId`, `SERVICE_TOKEN`, and `host`. We recommend you provide users with separate input fields with these components for authentication (dbt will surface these parameters for the user). ##### Exposing metadata to dbt Labs When building an integration, we recommend you expose certain metadata in the request for analytics and troubleshooting purpose. Please send us the following header with every query: `'X-dbt-partner-source': 'Your-Application-Name'` Additionally, it would be helpful if you also included the email and username of the person generating the query from your application. #### Use best practices when exposing metrics Best practices for exposing metrics are summarized into five themes: * [Governance](#governance-and-traceability) — Recommendations on how to establish guardrails for governed data work. * [Discoverability](#discoverability) — Recommendations on how to make user-friendly data interactions. * [Organization](#organization) — Organize metrics and dimensions for all audiences, use [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md). * [Query flexibility](#query-flexibility) — Allow users to query either one metric alone without dimensions or multiple metrics with dimensions. * [Context and interpretation](#context-and-interpretation) — Contextualize metrics for better analysis; expose definitions, metadata, lineage, and freshness. ##### Governance and traceability When working with more governed data, it's essential to establish clear guardrails. Here are some recommendations: * **Aggregations control** — Users shouldn't generally be allowed to modify aggregations unless they perform post-processing calculations on Semantic Layer data (such as year-over-year analysis). * **Time series alignment and using metric\_time** — Make sure users view metrics across the correct time series. When displaying metric graphs, using a non-default time aggregation dimension might lead to misleading interpretations. While users can still group by other time dimensions, they should be careful not to create trend lines with incorrect time axes.<br /><br />When looking at one or multiple metrics, users should use `metric_time` as the main time dimension to guarantee they are looking at the right time series for the metric(s).<br /><br />As such, when building an application, we recommend exposing `metric_time` as a separate, "special" time dimension on its own. This dimension is always going to align with all metrics and be common across them. Other time dimensions can still be looked at and grouped by, but having a clear delineation between the `metric_time` dimension and the other time dimensions is clarifying so that people do not confuse how metrics should be plotted.<br /><br />Also, when a user requests a time granularity change for the main time series, the query that your application runs should use `metric_time` as this will always give you the correct slice. Related to this, we also strongly recommend that you have a way to expose what dimension `metric_time` actually maps to for users who may not be familiar. Our APIs allow you to fetch the actual underlying time dimensions that makeup metric\_time (such as `transaction_date`) so you can expose them to your users. * **Units consistency** — If units are supported, it's vital to avoid plotting data incorrectly with different units. Ensuring consistency in unit representation will prevent confusion and misinterpretation of the data. * **Traceability of metric and dimension changes** — When users change names of metrics and dimensions for reports, it's crucial to have a traceability mechanism in place to link back to the original source metric name. ##### Discoverability * Consider treating [metrics](https://docs.getdbt.com/docs/build/metrics-overview.md) as first-class objects rather than measures. Metrics offer a higher-level and more contextual way to interact with data, reducing the burden on end-users to manually aggregate data. * **Easy metric interactions** — Provide users with an intuitive approach to: * Search for Metrics — Users should be able to easily search and find relevant metrics. Metrics can serve as the starting point to lead users into exploring dimensions. * Search for Dimensions — Users should be able to query metrics with associated dimensions, allowing them to gain deeper insights into the data. * Filter by Dimension Values — Expose and enable users to filter metrics based on dimension values, encouraging data analysis and exploration. * Filter additional metadata — Allow users to filter metrics based on other available metadata, such as metric type and default time granularity. * **Suggested metrics** — Ideally, the system should intelligently suggest relevant metrics to users based on their team's activities. This approach encourages user exposure, facilitates learning, and supports collaboration among team members. By implementing these recommendations, the data interaction process becomes more user-friendly, empowering users to gain valuable insights without the need for extensive data manipulation. ##### Organization We recommend organizing metrics and dimensions in ways that a non-technical user can understand the data model, without needing much context: * **Organizing dimensions** — To help non-technical users understand the data model better, we recommend organizing dimensions based on the entity they originated from. For example, consider dimensions like `user__country` and `product__category`.<br /><br />You can create groups by extracting `user` and `product` and then nest the respective dimensions under each group. This way, dimensions align with the entity or semantic model they belong to and make them more user-friendly and accessible. Additionally, we recommending adding a `label` parameter to dimensions in order to define the value displayed in downstream tools. * **Organizing metrics** — The goal is to organize metrics into a hierarchy in our configurations, instead of presenting them in a long list.<br /><br />This hierarchy helps you organize metrics based on specific criteria, such as business unit or team. By providing this structured organization, users can find and navigate metrics more efficiently, enhancing their overall data analysis experience. * **Using saved queries** — The Semantic Layer has a concept of [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md) which allows users to pre-build slices of metrics, dimensions, filters to be easily accessed. You should surface these as first class objects in your integration. Refer to the [JDBC](https://docs.getdbt.com/docs/dbt-apis/sl-jdbc.md) and [GraphQL](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md) APIs for syntax. ##### Query flexibility Allow users to query either one metric alone without dimensions or multiple metrics with dimensions. * Allow toggling between metrics/dimensions seamlessly. * Be clear on exposing what dimensions are queryable with what metrics and hide things that don’t apply. (Our APIs provide calls for you to get relevant dimensions for metrics, and vice versa). * Only expose time granularities (monthly, daily, yearly) that match the available metrics. * For example, if a dbt model and its resulting semantic model have a monthly granularity, make sure querying data with a 'daily' granularity isn't available to the user. Our APIs have functionality that will help you surface the correct granularities * We recommend that time granularity is treated as a general time dimension-specific concept and that it can be applied to more than just the primary aggregation (or `metric_time`). Consider a situation where a user wants to look at `sales` over time by `customer signup month`; in this situation, having the ability to apply granularities to both time dimensions is crucial. Our APIs include information to fetch the granularities for the primary (metric\_time) dimensions, as well as all time dimensions. You can treat each time dimension and granularity selection independently in your application. Note: Initially, as a starting point, it makes sense to only support `metric_time` or the primary time dimension, but we recommend expanding that as your solution evolves. * You should allow users to filter on date ranges and expose a calendar and nice presets for filtering these. * For example, last 30 days, last week, and so on. ##### Context and interpretation For better analysis, it's best to have the context of the metrics close to where the analysis is happening. We recommend the following: * Expose business definitions of the metrics as well as logical definitions. * Expose additional metadata from the Semantic layer (measures, type parameters). * Use the [Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md) to enhance the metric and build confidence in its accuracy: * Check if the metric is fresh and when it was last updated. * Include lineage information to understand the metric's origin. * Allow for creating other metadata that’s useful for the metric. We can provide some of this information in our configuration (Display name, Default Granularity for View, Default Time range), but there may be other metadata that your tool wants to provide to make the metric richer. ##### Transparency and using compile For transparency and additional context, we recommend you have an easy way for the user to obtain the SQL that MetricFlow generates. Depending on what API you are using, you can do this by using our `compile` parameter. This is incredibly powerful and emphasizes transparency and openness, particularly for technically inclined users. ##### Where filters and optimization In the cases where our APIs support either a string or a filter list for the `where` clause, we always recommend that your application utilizes the filter list in order to gain maximum pushdown benefits. The `where` string may be more intuitive for users writing queries during testing, but it will not have the performance benefits of the filter list in a production environment. #### Understand stages of an integration These are recommendations on how to evolve a Semantic Layer integration and not a strict runbook. **Stage 1 - The basic** * Supporting and using [JDBC](https://docs.getdbt.com/docs/dbt-apis/sl-jdbc.md) or [GraphQL](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md) is the first step. Refer to the [Semantic Layer APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md) for more technical details. **Stage 2 - More discoverability and basic querying** * Support listing metrics defined in the project * Listing available dimensions based on one or many metrics * Querying defined metric values on their own or grouping by available dimensions * Display metadata from [Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md) and other context * Expose [saved queries](https://docs.getdbt.com/docs/build/saved-queries.md), which are pre-built metrics, dimensions, and filters that Semantic Layer developers create for easier analysis. You can expose them in your application. Refer to the [JDBC](https://docs.getdbt.com/docs/dbt-apis/sl-jdbc.md) and [GraphQL](https://docs.getdbt.com/docs/dbt-apis/sl-graphql.md) APIs for syntax. **Stage 3 - More querying flexibility and better user experience (UX)** * More advanced filtering * Time filters with good presets/calendar UX * Filtering metrics on a pre-populated set of dimension values * Make dimension values more user-friendly by organizing them effectively * Intelligent filtering of metrics based on available dimensions and vice versa **Stage 4 - More custom user interface (UI) / Collaboration** * A place where users can see all the relevant information about a given metric * Organize metrics by hierarchy and more advanced search features (such as filter on the type of metric or other metadata) * Use and expose more metadata * Querying dimensions without metrics and other more advanced querying functionality * Suggest metrics to users based on teams/identity, and so on. ##### Related docs * [Semantic Layer FAQs](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-faqs.md) * [Use the Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) to learn about the product. * [Build your metrics](https://docs.getdbt.com/docs/build/build-metrics-intro.md) for more info about MetricFlow and its components. * [Semantic Layer integrations page](https://www.getdbt.com/product/semantic-layer-integrations) for information about the available partner integrations. --- ### Leverage dbt to generate analytics and ML-ready pipelines with SQL and Python with Snowflake [Back to guides](https://docs.getdbt.com/guides.md) Snowflake Intermediate [Menu ]() #### Introduction The focus of this workshop will be to demonstrate how we can use both *SQL and python together* in the same workflow to run *both analytics and machine learning models* on dbt. All code in today’s workshop can be found on [GitHub](https://github.com/dbt-labs/python-snowpark-formula1/tree/python-formula1). ##### What you'll use during the lab * A [Snowflake account](https://trial.snowflake.com/) with ACCOUNTADMIN access * A [dbt account](https://www.getdbt.com/signup/) ##### What you'll learn * How to build scalable data transformation pipelines using dbt, and Snowflake using SQL and Python * How to leverage copying data into Snowflake from a public S3 bucket ##### What you need to know * Basic to intermediate SQL and python. * Basic understanding of dbt fundamentals. We recommend the [dbt Fundamentals course](https://learn.getdbt.com) if you're interested. * High level machine learning process (encoding, training, testing) * Simple ML algorithms — we will use logistic regression to keep the focus on the *workflow*, not algorithms! ##### What you'll build * A set of data analytics and prediction pipelines using Formula 1 data leveraging dbt and Snowflake, making use of best practices like data quality tests and code promotion between environments * We will create insights for: 1. Finding the lap time average and rolling average through the years (is it generally trending up or down)? 2. Which constructor has the fastest pit stops in 2021? 3. Predicting the position of each driver given using a decade of data (2010 - 2020) As inputs, we are going to leverage Formula 1 datasets hosted on a dbt Labs public S3 bucket. We will create a Snowflake Stage for our CSV files then use Snowflake’s `COPY INTO` function to copy the data in from our CSV files into tables. The Formula 1 is available on [Kaggle](https://www.kaggle.com/datasets/rohanrao/formula-1-world-championship-1950-2020). The data is originally compiled from the [Ergast Developer API](http://ergast.com/mrd/). Overall we are going to set up the environments, build scalable pipelines in dbt, establish data tests, and promote code to production. #### Configure Snowflake 1. Log in to your trial Snowflake account. You can [sign up for a Snowflake Trial Account using this form](https://signup.snowflake.com/) if you don’t have one. 2. Ensure that your account is set up using **AWS** in the **US East (N. Virginia)**. We will be copying the data from a public AWS S3 bucket hosted by dbt Labs in the us-east-1 region. By ensuring our Snowflake environment setup matches our bucket region, we avoid any multi-region data copy and retrieval latency issues. [![Snowflake trial](/img/guides/dbt-ecosystem/dbt-python-snowpark/2-snowflake-configuration/1-snowflake-trial-AWS-setup.png?v=2 "Snowflake trial")](#)Snowflake trial 3. After creating your account and verifying it from your sign-up email, Snowflake will direct you back to the UI called Snowsight. 4. When Snowsight first opens, your window should look like the following, with you logged in as the ACCOUNTADMIN with demo worksheets open: [![Snowflake trial demo worksheets](/img/guides/dbt-ecosystem/dbt-python-snowpark/2-snowflake-configuration/2-new-snowflake-account.png?v=2 "Snowflake trial demo worksheets")](#)Snowflake trial demo worksheets 5. Navigate to **Admin > Billing & Terms**. Click **Enable > Acknowledge & Continue** to enable Anaconda Python Packages to run in Snowflake. [![Anaconda terms](/img/guides/dbt-ecosystem/dbt-python-snowpark/2-snowflake-configuration/3-accept-anaconda-terms.jpeg?v=2 "Anaconda terms")](#)Anaconda terms [![Enable Anaconda](/img/guides/dbt-ecosystem/dbt-python-snowpark/2-snowflake-configuration/4-enable-anaconda.png?v=2 "Enable Anaconda")](#)Enable Anaconda 1. Log in to your [trial Snowflake account](https://app.snowflake.com). 2. In the Snowflake UI, click the Create icon **+** in the upper left (under the Snowflake logo) to open a dropdown. 3. Select the first option, **SQL File**. #### Connect to data source We need to obtain our data source by copying our Formula 1 data into Snowflake tables from a public S3 bucket that dbt Labs hosts. 1. When a new Snowflake account is created, there should be a preconfigured warehouse in your account named `COMPUTE_WH`. 2. If for any reason your account doesn’t have this warehouse, we can create a warehouse using the following script: ```sql create or replace warehouse COMPUTE_WH with warehouse_size=XSMALL ``` 3. Rename the worksheet to `data setup script` since we will be placing code in this worksheet to ingest the Formula 1 data. Make sure you are still logged in as the **ACCOUNTADMIN** and select the **COMPUTE\_WH** warehouse. [![Rename worksheet and select warehouse](/img/guides/dbt-ecosystem/dbt-python-snowpark/3-connect-to-data-source/1-rename-worksheet-and-select-warehouse.png?v=2 "Rename worksheet and select warehouse")](#)Rename worksheet and select warehouse 4. Copy the following code into the main body of the Snowflake worksheet. You can also find this setup script under the `setup` folder in the [Git repository](https://github.com/dbt-labs/python-snowpark-formula1/blob/main/setup/setup_script_s3_to_snowflake.sql). The script is long since it's bring in all of the data we'll need today! ```sql -- create and define our formula1 database create or replace database formula1; use database formula1; create or replace schema raw; use schema raw; -- define our file format for reading in the CSVs create or replace file format CSVformat type = CSV field_delimiter =',' field_optionally_enclosed_by = '"', skip_header=1; -- create or replace stage formula1_stage file_format = CSVformat url = 's3://formula1-dbt-cloud-python-demo/formula1-kaggle-data/'; -- load in the 8 tables we need for our demo -- we are first creating the table then copying our data in from s3 -- think of this as an empty container or shell that we are then filling create or replace table formula1.raw.circuits ( CIRCUITID NUMBER(38,0), CIRCUITREF VARCHAR(16777216), NAME VARCHAR(16777216), LOCATION VARCHAR(16777216), COUNTRY VARCHAR(16777216), LAT FLOAT, LNG FLOAT, ALT NUMBER(38,0), URL VARCHAR(16777216) ); -- copy our data from public s3 bucket into our tables copy into circuits from @formula1_stage/circuits.csv on_error='continue'; create or replace table formula1.raw.constructors ( CONSTRUCTORID NUMBER(38,0), CONSTRUCTORREF VARCHAR(16777216), NAME VARCHAR(16777216), NATIONALITY VARCHAR(16777216), URL VARCHAR(16777216) ); copy into constructors from @formula1_stage/constructors.csv on_error='continue'; create or replace table formula1.raw.drivers ( DRIVERID NUMBER(38,0), DRIVERREF VARCHAR(16777216), NUMBER VARCHAR(16777216), CODE VARCHAR(16777216), FORENAME VARCHAR(16777216), SURNAME VARCHAR(16777216), DOB DATE, NATIONALITY VARCHAR(16777216), URL VARCHAR(16777216) ); copy into drivers from @formula1_stage/drivers.csv on_error='continue'; create or replace table formula1.raw.lap_times ( RACEID NUMBER(38,0), DRIVERID NUMBER(38,0), LAP NUMBER(38,0), POSITION FLOAT, TIME VARCHAR(16777216), MILLISECONDS NUMBER(38,0) ); copy into lap_times from @formula1_stage/lap_times.csv on_error='continue'; create or replace table formula1.raw.pit_stops ( RACEID NUMBER(38,0), DRIVERID NUMBER(38,0), STOP NUMBER(38,0), LAP NUMBER(38,0), TIME VARCHAR(16777216), DURATION VARCHAR(16777216), MILLISECONDS NUMBER(38,0) ); copy into pit_stops from @formula1_stage/pit_stops.csv on_error='continue'; create or replace table formula1.raw.races ( RACEID NUMBER(38,0), YEAR NUMBER(38,0), ROUND NUMBER(38,0), CIRCUITID NUMBER(38,0), NAME VARCHAR(16777216), DATE DATE, TIME VARCHAR(16777216), URL VARCHAR(16777216), FP1_DATE VARCHAR(16777216), FP1_TIME VARCHAR(16777216), FP2_DATE VARCHAR(16777216), FP2_TIME VARCHAR(16777216), FP3_DATE VARCHAR(16777216), FP3_TIME VARCHAR(16777216), QUALI_DATE VARCHAR(16777216), QUALI_TIME VARCHAR(16777216), SPRINT_DATE VARCHAR(16777216), SPRINT_TIME VARCHAR(16777216) ); copy into races from @formula1_stage/races.csv on_error='continue'; create or replace table formula1.raw.results ( RESULTID NUMBER(38,0), RACEID NUMBER(38,0), DRIVERID NUMBER(38,0), CONSTRUCTORID NUMBER(38,0), NUMBER NUMBER(38,0), GRID NUMBER(38,0), POSITION FLOAT, POSITIONTEXT VARCHAR(16777216), POSITIONORDER NUMBER(38,0), POINTS NUMBER(38,0), LAPS NUMBER(38,0), TIME VARCHAR(16777216), MILLISECONDS NUMBER(38,0), FASTESTLAP NUMBER(38,0), RANK NUMBER(38,0), FASTESTLAPTIME VARCHAR(16777216), FASTESTLAPSPEED FLOAT, STATUSID NUMBER(38,0) ); copy into results from @formula1_stage/results.csv on_error='continue'; create or replace table formula1.raw.status ( STATUSID NUMBER(38,0), STATUS VARCHAR(16777216) ); copy into status from @formula1_stage/status.csv on_error='continue'; ``` 5. Ensure all the commands are selected before running the query — an easy way to do this is to use Ctrl-a to highlight all of the code in the worksheet. Select **run** (blue triangle icon). Notice how the dot next to your **COMPUTE\_WH** turns from gray to green as you run the query. The **status** table is the final table of all 8 tables loaded in. [![Load data from S3 bucket](/img/guides/dbt-ecosystem/dbt-python-snowpark/3-connect-to-data-source/2-load-data-from-s3.png?v=2 "Load data from S3 bucket")](#)Load data from S3 bucket 6. Let’s unpack that pretty long query we ran into component parts. We ran this query to load in our 8 Formula 1 tables from a public S3 bucket. To do this, we: * Created a new database called `formula1` and a schema called `raw` to place our raw (untransformed) data into. * Defined our file format for our CSV files. Importantly, here we use a parameter called `field_optionally_enclosed_by =` since the string columns in our Formula 1 CSV files use quotes. Quotes are used around string values to avoid parsing issues where commas `,` and new lines `/n` in data values could cause data loading errors. * Created a stage to locate our data we are going to load in. Snowflake Stages are locations where data files are stored. Stages are used to both load and unload data to and from Snowflake locations. Here we are using an external stage, by referencing an S3 bucket. * Created our tables for our data to be copied into. These are empty tables with the column name and data type. Think of this as creating an empty container that the data will then fill into. * Used the `copy into` statement for each of our tables. We reference our staged location we created and upon loading errors continue to load in the rest of the data. You should not have data loading errors but if you do, those rows will be skipped and Snowflake will tell you which rows caused errors 7. Now let's take a look at some of our cool Formula 1 data we just loaded up! 1. Create a new worksheet by selecting the **+** then **New Worksheet**. [![Create new worksheet to query data](/img/guides/dbt-ecosystem/dbt-python-snowpark/3-connect-to-data-source/3-create-new-worksheet-to-query-data.png?v=2 "Create new worksheet to query data")](#)Create new worksheet to query data 2. Navigate to **Database > Formula1 > RAW > Tables**. 3. Query the data using the following code. There are only 76 rows in the circuits table, so we don’t need to worry about limiting the amount of data we query. ```sql select * from formula1.raw.circuits ``` 4. Run the query. From here on out, we’ll use the keyboard shortcuts Command-Enter or Control-Enter to run queries and won’t explicitly call out this step. 5. Review the query results, you should see information about Formula 1 circuits, starting with Albert Park in Australia! 6. Finally, ensure you have all 8 tables starting with `CIRCUITS` and ending with `STATUS`. Now we are ready to connect into dbt! [![Query circuits data](/img/guides/dbt-ecosystem/dbt-python-snowpark/3-connect-to-data-source/4-query-circuits-data.png?v=2 "Query circuits data")](#)Query circuits data #### Configure dbt 1. We are going to be using [Snowflake Partner Connect](https://docs.snowflake.com/en/user-guide/ecosystem-partner-connect.html) to set up a dbt account. Using this method will allow you to spin up a fully fledged dbt account with your [Snowflake connection](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-snowflake.md), [managed repository](https://docs.getdbt.com/docs/platform/git/managed-repository.md), environments, and credentials already established. 2. Navigate out of your worksheet back by selecting **home**. 3. In Snowsight, confirm that you are using the **ACCOUNTADMIN** role. 4. Navigate to the **Data Products** **> Partner Connect**. Find **dbt** either by using the search bar or navigating the **Data Integration**. Select the **dbt** tile. [![Open Partner Connect](/img/guides/dbt-ecosystem/dbt-python-snowpark/4-configure-dbt/1-open-partner-connect.png?v=2 "Open Partner Connect")](#)Open Partner Connect 5. You should now see a new window that says **Connect to dbt**. Select **Optional Grant** and add the `FORMULA1` database. This will grant access for your new dbt user role to the FORMULA1 database. [![Partner Connect Optional Grant](/img/guides/dbt-ecosystem/dbt-python-snowpark/4-configure-dbt/2-partner-connect-optional-grant.png?v=2 "Partner Connect Optional Grant")](#)Partner Connect Optional Grant 6. Ensure the `FORMULA1` is present in your optional grant before clicking **Connect**.  This will create a dedicated dbt user, database, warehouse, and role for your dbt trial. [![Connect to dbt](/img/guides/dbt-ecosystem/dbt-python-snowpark/4-configure-dbt/3-connect-to-dbt.png?v=2 "Connect to dbt")](#)Connect to dbt 7. When you see the **Your partner account has been created** window, click **Activate**. 8. You should be redirected to a dbt registration page. Fill out the form. Make sure to save the password somewhere for login in the future. [![dbt sign up](/img/guides/dbt-ecosystem/dbt-python-snowpark/4-configure-dbt/4-dbt-cloud-sign-up.png?v=2 "dbt sign up")](#)dbt sign up 9. Select **Complete Registration**. You should now be redirected to your dbt account, complete with a connection to your Snowflake account, a deployment and a development environment, and a sample job. 10. To help you version control your dbt project, we have connected it to a [managed repository](https://docs.getdbt.com/docs/platform/git/managed-repository.md), which means that dbt Labs will be hosting your repository for you. This will give you access to a Git workflow without you having to create and host the repository yourself. You will not need to know Git for this workshop; dbt will help guide you through the workflow. In the future, when you’re developing your own project, [feel free to use your own repository](https://docs.getdbt.com/docs/platform/git/connect-github.md). This will allow you to learn more about features like [Slim CI](https://docs.getdbt.com/docs/deploy/continuous-integration.md) builds after this workshop. #### Change development schema name and navigate the IDE 1. First we are going to change the name of our default schema to where our dbt models will build. By default, the name is `dbt_`. We will change this to `dbt_<YOUR_NAME>` to create your own personal development schema. To do this, click on your account name in the left side menu and select **Account settings**. [![Settings menu](/img/guides/dbt-ecosystem/dbt-python-snowpark/5-development-schema-name/1-settings-gear-icon.png?v=2 "Settings menu")](#)Settings menu 2. Navigate to the **Credentials** menu and select **Partner Connect Trial**, which will expand the credentials menu. [![Credentials edit schema name](/img/guides/dbt-ecosystem/dbt-python-snowpark/5-development-schema-name/2-credentials-edit-schema-name.png?v=2 "Credentials edit schema name")](#)Credentials edit schema name 3. Click **Edit** and change the name of your schema from `dbt_` to `dbt_YOUR_NAME` replacing `YOUR_NAME` with your initials and name. Be sure to click **Save** for your changes! [![Save new schema name](/img/guides/dbt-ecosystem/dbt-python-snowpark/5-development-schema-name/3-save-new-schema-name.png?v=2 "Save new schema name")](#)Save new schema name 4. We now have our own personal development schema, amazing! When we run our first dbt models they will build into this schema. 5. Let’s open up dbt’s Integrated Development Environment (Studio IDE) and familiarize ourselves. Choose **Develop** at the top of the UI. 6. When the Studio IDE is done loading, click **Initialize dbt project**. The initialization process creates a collection of files and folders necessary to run your dbt project. [![Initialize dbt project](/img/guides/dbt-ecosystem/dbt-python-snowpark/5-development-schema-name/4-initialize-dbt-project.png?v=2 "Initialize dbt project")](#)Initialize dbt project 7. After the initialization is finished, you can view the files and folders in the file tree menu. As we move through the workshop we'll be sure to touch on a few key files and folders that we'll work with to build out our project. 8. Next click **Commit and sync** to commit the new files and folders from the initialize step. We always want our commit messages to be relevant to the work we're committing, so be sure to provide a message like `initialize project` and select **Commit Changes**. [![First commit and push](/img/guides/dbt-ecosystem/dbt-python-snowpark/5-development-schema-name/5-first-commit-and-push.png?v=2 "First commit and push")](#)First commit and push [![Commit Changes button](/img/guides/dbt-ecosystem/dbt-python-snowpark/5-development-schema-name/6-initalize-project.png?v=2 "Commit Changes button")](#)Commit Changes button 9. [Committing](https://www.atlassian.com/git/tutorials/saving-changes/git-commit) your work here will save it to the managed git repository that was created during the Partner Connect signup. This initial commit is the only commit that will be made directly to our `main` branch and from *here on out we'll be doing all of our work on a development branch*. This allows us to keep our development work separate from our production code. 10. There are a couple of key features to point out about the Studio IDE before we get to work. It is a text editor, an SQL and Python runner, and a CLI with Git version control all baked into one package! This allows you to focus on editing your SQL and Python files, previewing the results with the SQL runner (it even runs Jinja!), and building models at the command line without having to move between different applications. The Git workflow in dbt allows both Git beginners and experts alike to be able to easily version control all of their work with a couple clicks. [![IDE overview](/img/guides/dbt-ecosystem/dbt-python-snowpark/5-development-schema-name/7-IDE-overview.png?v=2 "IDE overview")](#)IDE overview 11. Let's run our first dbt models! Two example models are included in your dbt project in the `models/examples` folder that we can use to illustrate how to run dbt at the command line. Type `dbt run` into the command line and click **Enter** on your keyboard. When the run bar expands you'll be able to see the results of the run, where you should see the run complete successfully. [![dbt run example models](/img/guides/dbt-ecosystem/dbt-python-snowpark/5-development-schema-name/8-dbt-run-example-models.png?v=2 "dbt run example models")](#)dbt run example models 12. The run results allow you to see the code that dbt compiles and sends to Snowflake for execution. To view the logs for this run, select one of the model tabs using the  **>** icon and then **Details**. If you scroll down a bit you'll be able to see the compiled code and how dbt interacts with Snowflake. Given that this run took place in our development environment, the models were created in your development schema. [![Details about the second model](/img/guides/dbt-ecosystem/dbt-python-snowpark/5-development-schema-name/9-second-model-details.png?v=2 "Details about the second model")](#)Details about the second model 13. Now let's switch over to Snowflake to confirm that the objects were actually created. Click on the three dots **…** above your database objects and then **Refresh**. Expand the **PC\_DBT\_DB** database and you should see your development schema. Select the schema, then **Tables**  and **Views**. Now you should be able to see `MY_FIRST_DBT_MODEL` as a table and `MY_SECOND_DBT_MODEL` as a view. [![Confirm example models are built in Snowflake](/img/guides/dbt-ecosystem/dbt-python-snowpark/5-development-schema-name/10-confirm-example-models-built-in-snowflake.png?v=2 "Confirm example models are built in Snowflake")](#)Confirm example models are built in Snowflake #### Create branch and set up project configs In this step, we’ll need to create a development branch and set up project level configurations. 1. To get started with development for our project, we'll need to create a new Git branch for our work. Select **create branch** and name your development branch. We'll call our branch `snowpark_python_workshop` then click **Submit**. 2. The first piece of development we'll do on the project is to update the `dbt_project.yml` file. Every dbt project requires a `dbt_project.yml` file — this is how dbt knows a directory is a dbt project. The [dbt\_project.yml](https://docs.getdbt.com/reference/dbt_project.yml.md) file also contains important information that tells dbt how to operate on your project. 3. Select the `dbt_project.yml` file from the file tree to open it and replace all of the existing contents with the following code below. When you're done, save the file by clicking **save**. You can also use the Command-S or Control-S shortcut from here on out. ```yaml # Name your project! Project names should contain only lowercase characters # and underscores. A good package name should reflect your organization's # name or the intended use of these models name: 'snowflake_dbt_python_formula1' version: '1.3.0' require-dbt-version: '>=1.3.0' config-version: 2 # This setting configures which "profile" dbt uses for this project. profile: 'default' # These configurations specify where dbt should look for different types of files. # The `model-paths` config, for example, states that models in this project can be # found in the "models/" directory. You probably won't need to change these! model-paths: ["models"] analysis-paths: ["analyses"] test-paths: ["tests"] seed-paths: ["seeds"] macro-paths: ["macros"] snapshot-paths: ["snapshots"] target-path: "target" # directory which will store compiled SQL files clean-targets: # directories to be removed by `dbt clean` - "target" - "dbt_packages" models: snowflake_dbt_python_formula1: staging: +docs: node_color: "CadetBlue" marts: +materialized: table aggregates: +docs: node_color: "Maroon" +tags: "bi" core: +docs: node_color: "#800080" intermediate: +docs: node_color: "MediumSlateBlue" ml: prep: +docs: node_color: "Indigo" train_predict: +docs: node_color: "#36454f" ``` 4. The key configurations to point out in the file with relation to the work that we're going to do are in the `models` section. * `require-dbt-version` — Tells dbt which version of dbt to use for your project. We are requiring 1.3.0 and any newer version to run python models and node colors. * `materialized` — Tells dbt how to materialize models when compiling the code before it pushes it down to Snowflake. All models in the `marts` folder will be built as tables. * `tags` — Applies tags at a directory level to all models. All models in the `aggregates` folder will be tagged as `bi` (abbreviation for business intelligence). * `docs` — Specifies the `node_color` either by the plain color name or a hex value. 5. [Materializations](https://docs.getdbt.com/docs/build/materializations.md) are strategies for persisting dbt models in a warehouse, with `tables` and `views` being the most commonly utilized types. By default, all dbt models are materialized as views and other materialization types can be configured in the `dbt_project.yml` file or in a model itself. It’s very important to note *Python models can only be materialized as tables or incremental models.* Since all our Python models exist under `marts`, the following portion of our `dbt_project.yml` ensures no errors will occur when we run our Python models. Starting with [dbt version 1.4](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.4.md#updates-to-python-models>), Python files will automatically get materialized as tables even if not explicitly specified. ```yaml marts:     +materialized: table ``` #### Create folders and organize files dbt Labs has developed a [project structure guide](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md) that contains a number of recommendations for how to build the folder structure for your project. Do check out that guide if you want to learn more. Right now we are going to create some folders to organize our files: * Sources — This is our Formula 1 dataset and it will be defined in a source properties YAML file. * Staging models — These models have a 1:1 with their source table. * Intermediate — This is where we will be joining some Formula staging models. * Marts models — Here is where we perform our major transformations. It contains these subfolders: * aggregates * core * ml 1. In your file tree, use your cursor and hover over the `models` subdirectory, click the three dots **…** that appear to the right of the folder name, then select **Create Folder**. We're going to add two new folders to the file path, `staging` and `formula1` (in that order) by typing `staging/formula1` into the file path. [![Create folder](/img/guides/dbt-ecosystem/dbt-python-snowpark/7-folder-structure/1-create-folder.png?v=2 "Create folder")](#)Create folder [![Set file path](/img/guides/dbt-ecosystem/dbt-python-snowpark/7-folder-structure/2-file-path.png?v=2 "Set file path")](#)Set file path * If you click into your `models` directory now, you should see the new `staging` folder nested within `models` and the `formula1` folder nested within `staging`. 2. Create two additional folders the same as the last step. Within the `models` subdirectory, create new directories `marts/core`. 3. We will need to create a few more folders and subfolders using the UI. After you create all the necessary folders, your folder tree should look like this when it's all done: [![File tree of new folders](/img/guides/dbt-ecosystem/dbt-python-snowpark/7-folder-structure/3-tree-of-new-folders.png?v=2 "File tree of new folders")](#)File tree of new folders Remember you can always reference the entire project in [GitHub](https://github.com/dbt-labs/python-snowpark-formula1/tree/python-formula1) to view the complete folder and file strucutre. #### Create source and staging models In this section, we are going to create our source and staging models. Sources allow us to create a dependency between our source database object and our staging models which will help us when we look at data lineage later. Also, if your source changes database or schema, you only have to update it in your `f1_sources.yml` file rather than updating all of the models it might be used in. Staging models are the base of our project, where we bring all the individual components we're going to use to build our more complex and useful models into the project. Since we want to focus on dbt and Python in this workshop, check out our [sources](https://docs.getdbt.com/docs/build/sources.md) and [staging](https://docs.getdbt.com/best-practices/how-we-structure/2-staging.md) docs if you want to learn more (or take our [dbt Fundamentals](https://learn.getdbt.com/courses/dbt-fundamentals) course which covers all of our core functionality). ##### 1. Create sources We're going to be using each of our 8 Formula 1 tables from our `formula1` database under the `raw`  schema for our transformations and we want to create those tables as sources in our project. 1. Create a new file called `f1_sources.yml` with the following file path: `models/staging/formula1/f1_sources.yml`. 2. Then, paste the following code into the file before saving it: ```yaml version: 2 sources: - name: formula1 description: formula 1 datasets with normalized tables database: formula1 schema: raw tables: - name: circuits description: One record per circuit, which is the specific race course. columns: - name: circuitid data_tests: - unique - not_null - name: constructors description: One record per constructor. Constructors are the teams that build their formula 1 cars. columns: - name: constructorid data_tests: - unique - not_null - name: drivers description: One record per driver. This table gives details about the driver. columns: - name: driverid data_tests: - unique - not_null - name: lap_times description: One row per lap in each race. Lap times started being recorded in this dataset in 1984 and joined through driver_id. - name: pit_stops description: One row per pit stop. Pit stops do not have their own id column, the combination of the race_id and driver_id identify the pit stop. columns: - name: stop data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: [1,2,3,4,5,6,7,8] quote: false - name: races description: One race per row. Importantly this table contains the race year to understand trends. columns: - name: raceid data_tests: - unique - not_null - name: results columns: - name: resultid data_tests: - unique - not_null description: One row per result. The main table that we join out for grid and position variables. - name: status description: One status per row. The status contextualizes whether the race was finished or what issues arose e.g. collisions, engine, etc. columns: - name: statusid data_tests: - unique - not_null ``` ##### 2. Create staging models The next step is to set up the staging models for each of the 8 source tables. Given the one-to-one relationship between staging models and their corresponding source tables, we'll build 8 staging models here. We know it’s a lot and in the future, we will seek to update the workshop to make this step less repetitive and more efficient. This step is also a good representation of the real world of data, where you have multiple hierarchical tables that you will need to join together! 1. Let's go in alphabetical order to easily keep track of all our staging models! Create a new file called `stg_f1_circuits.sql` with this file path `models/staging/formula1/stg_f1_circuits.sql`. Then, paste the following code into the file before saving it: ```sql with source as ( select * from {{ source('formula1','circuits') }} ), renamed as ( select circuitid as circuit_id, circuitref as circuit_ref, name as circuit_name, location, country, lat as latitude, lng as longitude, alt as altitude -- omit the url from source ) select * from renamed ``` All we're doing here is pulling the source data into the model using the `source` function, renaming some columns, and omitting the column `url` with a commented note since we don’t need it for our analysis. 2. Create `stg_f1_constructors.sql` with this file path `models/staging/formula1/stg_f1_constructors.sql`. Paste the following code into it before saving the file: ```sql with source as ( select * from {{ source('formula1','constructors') }} ), renamed as ( select constructorid as constructor_id, constructorref as constructor_ref, name as constructor_name, nationality as constructor_nationality -- omit the url from source ) select * from renamed ``` We have 6 other stages models to create. We can do this by creating new files, then copy and paste the code into our `staging` folder. 3. Create `stg_f1_drivers.sql` with this file path `models/staging/formula1/stg_f1_drivers.sql`: ```sql with source as ( select * from {{ source('formula1','drivers') }} ), renamed as ( select driverid as driver_id, driverref as driver_ref, number as driver_number, code as driver_code, forename, surname, dob as date_of_birth, nationality as driver_nationality -- omit the url from source ) select * from renamed ``` 4. Create `stg_f1_lap_times.sql` with this file path `models/staging/formula1/stg_f1_lap_times.sql`: ```sql with source as ( select * from {{ source('formula1','lap_times') }} ), renamed as ( select raceid as race_id, driverid as driver_id, lap, position, time as lap_time_formatted, milliseconds as lap_time_milliseconds from source ) select * from renamed ``` 5. Create `stg_f1_pit_stops.sql` with this file path `models/staging/formula1/stg_f1_pit_stops.sql`: ```sql with source as ( select * from {{ source('formula1','pit_stops') }} ), renamed as ( select raceid as race_id, driverid as driver_id, stop as stop_number, lap, time as lap_time_formatted, duration as pit_stop_duration_seconds, milliseconds as pit_stop_milliseconds from source ) select * from renamed order by pit_stop_duration_seconds desc ``` 6. Create `stg_f1_races.sql` with this file path `models/staging/formula1/stg_f1_races.sql`: ```sql with source as ( select * from {{ source('formula1','races') }} ), renamed as ( select raceid as race_id, year as race_year, round as race_round, circuitid as circuit_id, name as circuit_name, date as race_date, to_time(time) as race_time, -- omit the url fp1_date as free_practice_1_date, fp1_time as free_practice_1_time, fp2_date as free_practice_2_date, fp2_time as free_practice_2_time, fp3_date as free_practice_3_date, fp3_time as free_practice_3_time, quali_date as qualifying_date, quali_time as qualifying_time, sprint_date, sprint_time from source ) select * from renamed ``` 7. Create `stg_f1_results.sql` with this file path `models/staging/formula1/stg_f1_results.sql`: ```sql with source as ( select * from {{ source('formula1','results') }} ), renamed as ( select resultid as result_id, raceid as race_id, driverid as driver_id, constructorid as constructor_id, number as driver_number, grid, position::int as position, positiontext as position_text, positionorder as position_order, points, laps, time as results_time_formatted, milliseconds as results_milliseconds, fastestlap as fastest_lap, rank as results_rank, fastestlaptime as fastest_lap_time_formatted, fastestlapspeed::decimal(6,3) as fastest_lap_speed, statusid as status_id from source ) select * from renamed ``` 8. Last one! Create `stg_f1_status.sql` with this file path: `models/staging/formula1/stg_f1_status.sql`: ```sql with source as ( select * from {{ source('formula1','status') }} ), renamed as ( select statusid as status_id, status from source ) select * from renamed ``` After the source and all the staging models are complete for each of the 8 tables, your staging folder should look like this: [![Staging folder](/img/guides/dbt-ecosystem/dbt-python-snowpark/8-sources-and-staging/1-staging-folder.png?v=2 "Staging folder")](#)Staging folder 9. It’s a good time to delete our example folder since these two models are extraneous to our formula1 pipeline and `my_first_model` fails a `not_null` test that we won’t spend time investigating. dbt will warn us that this folder will be permanently deleted, and we are okay with that so select **Delete**. [![Delete example folder](/img/guides/dbt-ecosystem/dbt-python-snowpark/8-sources-and-staging/2-delete-example.png?v=2 "Delete example folder")](#)Delete example folder 10. Now that the staging models are built and saved, it's time to create the models in our development schema in Snowflake. To do this we're going to enter into the command line `dbt build` to run all of the models in our project, which includes the 8 new staging models and the existing example models. Your run should complete successfully and you should see green checkmarks next to all of your models in the run results. We built our 8 staging models as views and ran 13 source tests that we configured in the `f1_sources.yml` file with not that much code, pretty cool! [![Successful dbt build in Snowflake](/img/guides/dbt-ecosystem/dbt-python-snowpark/8-sources-and-staging/3-successful-run-in-snowflake.png?v=2 "Successful dbt build in Snowflake")](#)Successful dbt build in Snowflake Let's take a quick look in Snowflake, refresh database objects, open our development schema, and confirm that the new models are there. If you can see them, then we're good to go! [![Confirm models](/img/guides/dbt-ecosystem/dbt-python-snowpark/8-sources-and-staging/4-confirm-models.png?v=2 "Confirm models")](#)Confirm models Before we move onto the next section, be sure to commit your new models to your Git branch. Click **Commit and push** and give your commit a message like `profile, sources, and staging setup` before moving on. #### Transform SQL Now that we have all our sources and staging models done, it's time to move into where dbt shines — transformation! We need to: * Create some intermediate tables to join tables that aren’t hierarchical * Create core tables for business intelligence (BI) tool ingestion * Answer the two questions about: * fastest pit stops * lap time trends about our Formula 1 data by creating aggregate models using python! ##### Intermediate models We need to join lots of reference tables to our results table to create a human readable dataframe. What does this mean? For example, we don’t only want to have the numeric `status_id` in our table, we want to be able to read in a row of data that a driver could not finish a race due to engine failure (`status_id=5`). By now, we are pretty good at creating new files in the correct directories so we won’t cover this in detail. All intermediate models should be created in the path `models/intermediate`. 1. Create a new file called `int_lap_times_years.sql`. In this model, we are joining our lap time and race information so we can look at lap times over years. In earlier Formula 1 eras, lap times were not recorded (only final results), so we filter out records where lap times are null. ```sql with lap_times as ( select * from {{ ref('stg_f1_lap_times') }} ), races as ( select * from {{ ref('stg_f1_races') }} ), expanded_lap_times_by_year as ( select lap_times.race_id, driver_id, race_year, lap, lap_time_milliseconds from lap_times left join races on lap_times.race_id = races.race_id where lap_time_milliseconds is not null ) select * from expanded_lap_times_by_year ``` 2. Create a file called `in_pit_stops.sql`. Pit stops are a many-to-one (M:1) relationship with our races. We are creating a feature called `total_pit_stops_per_race` by partitioning over our `race_id` and `driver_id`, while preserving individual level pit stops for rolling average in our next section. ```sql with stg_f1__pit_stops as ( select * from {{ ref('stg_f1_pit_stops') }} ), pit_stops_per_race as ( select race_id, driver_id, stop_number, lap, lap_time_formatted, pit_stop_duration_seconds, pit_stop_milliseconds, max(stop_number) over (partition by race_id,driver_id) as total_pit_stops_per_race from stg_f1__pit_stops ) select * from pit_stops_per_race ``` 3. Create a file called `int_results.sql`. Here we are using 4 of our tables — `races`, `drivers`, `constructors`, and `status` — to give context to our `results` table. We are now able to calculate a new feature `drivers_age_years` by bringing the `date_of_birth` and `race_year` into the same table. We are also creating a column to indicate if the driver did not finish (dnf) the race, based upon if their `position` was null called, `dnf_flag`. ```sql with results as ( select * from {{ ref('stg_f1_results') }} ), races as ( select * from {{ ref('stg_f1_races') }} ), drivers as ( select * from {{ ref('stg_f1_drivers') }} ), constructors as ( select * from {{ ref('stg_f1_constructors') }} ), status as ( select * from {{ ref('stg_f1_status') }} ), int_results as ( select result_id, results.race_id, race_year, race_round, circuit_id, circuit_name, race_date, race_time, results.driver_id, results.driver_number, forename ||' '|| surname as driver, cast(datediff('year', date_of_birth, race_date) as int) as drivers_age_years, driver_nationality, results.constructor_id, constructor_name, constructor_nationality, grid, position, position_text, position_order, points, laps, results_time_formatted, results_milliseconds, fastest_lap, results_rank, fastest_lap_time_formatted, fastest_lap_speed, results.status_id, status, case when position is null then 1 else 0 end as dnf_flag from results left join races on results.race_id=races.race_id left join drivers on results.driver_id = drivers.driver_id left join constructors on results.constructor_id = constructors.constructor_id left join status on results.status_id = status.status_id ) select * from int_results ``` 4. Create a *Markdown* file `intermediate.md` that we will go over in depth in the Test and Documentation sections of the [Leverage dbt to generate analytics and ML-ready pipelines with SQL and Python with Snowflake](https://docs.getdbt.com/guides/dbt-python-snowpark.md) guide. ```markdown # the intent of this .md is to allow for multi-line long form explanations for our intermediate transformations # below are descriptions {% docs int_results %} In this query we want to join out other important information about the race results to have a human readable table about results, races, drivers, constructors, and status. We will have 4 left joins onto our results table. {% enddocs %} {% docs int_pit_stops %} There are many pit stops within one race, aka a M:1 relationship. We want to aggregate this so we can properly join pit stop information without creating a fanout. {% enddocs %} {% docs int_lap_times_years %} Lap times are done per lap. We need to join them out to the race year to understand yearly lap time trends. {% enddocs %} ``` 5. Create a *YAML* file `intermediate.yml` that we will go over in depth during the Test and Document sections of the [Leverage dbt to generate analytics and ML-ready pipelines with SQL and Python with Snowflake](https://docs.getdbt.com/guides/dbt-python-snowpark.md) guide. ```yaml version: 2 models: - name: int_results description: '{{ doc("int_results") }}' - name: int_pit_stops description: '{{ doc("int_pit_stops") }}' - name: int_lap_times_years description: '{{ doc("int_lap_times_years") }}' ``` That wraps up the intermediate models we need to create our core models! ##### dbt models 1. Create a file `fct_results.sql`. This is what I like to refer to as the “mega table” — a really large denormalized table with all our context added in at row level for human readability. Importantly, we have a table `circuits` that is linked through the table `races`. When we joined `races` to `results` in `int_results.sql` we allowed our tables to make the connection from `circuits` to `results` in `fct_results.sql`. We are only taking information about pit stops at the result level so our join would not cause a [fanout](https://community.looker.com/technical-tips-tricks-1021/what-is-a-fanout-23327). ```sql with int_results as ( select * from {{ ref('int_results') }} ), int_pit_stops as ( select race_id, driver_id, max(total_pit_stops_per_race) as total_pit_stops_per_race from {{ ref('int_pit_stops') }} group by 1,2 ), circuits as ( select * from {{ ref('stg_f1_circuits') }} ), base_results as ( select result_id, int_results.race_id, race_year, race_round, int_results.circuit_id, int_results.circuit_name, circuit_ref, location, country, latitude, longitude, altitude, total_pit_stops_per_race, race_date, race_time, int_results.driver_id, driver, driver_number, drivers_age_years, driver_nationality, constructor_id, constructor_name, constructor_nationality, grid, position, position_text, position_order, points, laps, results_time_formatted, results_milliseconds, fastest_lap, results_rank, fastest_lap_time_formatted, fastest_lap_speed, status_id, status, dnf_flag from int_results left join circuits on int_results.circuit_id=circuits.circuit_id left join int_pit_stops on int_results.driver_id=int_pit_stops.driver_id and int_results.race_id=int_pit_stops.race_id ) select * from base_results ``` 2. Create the file `pit_stops_joined.sql`. Our results and pit stops are at different levels of dimensionality (also called grain). Simply put, we have multiple pit stops per a result. Since we are interested in understanding information at the pit stop level with information about race year and constructor, we will create a new table `pit_stops_joined.sql` where each row is per pit stop. Our new table tees up our aggregation in Python. ```sql with base_results as ( select * from {{ ref('fct_results') }} ), pit_stops as ( select * from {{ ref('int_pit_stops') }} ), pit_stops_joined as ( select base_results.race_id, race_year, base_results.driver_id, constructor_id, constructor_name, stop_number, lap, lap_time_formatted, pit_stop_duration_seconds, pit_stop_milliseconds from base_results left join pit_stops on base_results.race_id=pit_stops.race_id and base_results.driver_id=pit_stops.driver_id ) select * from pit_stops_joined ``` 3. Enter in the command line and execute `dbt build` to build out our entire pipeline to up to this point. Don’t worry about “overriding” your previous models – dbt workflows are designed to be idempotent so we can run them again and expect the same results. 4. Let’s talk about our lineage so far. It’s looking good 😎. We’ve shown how SQL can be used to make data type, column name changes, and handle hierarchical joins really well; all while building out our automated lineage! [![The DAG](/img/guides/dbt-ecosystem/dbt-python-snowpark/9-sql-transformations/1-dag.png?v=2 "The DAG")](#)The DAG 5. Time to **Commit and push** our changes and give your commit a message like `intermediate and fact models` before moving on. #### Running dbt Python models Up until now, SQL has been driving the project (car pun intended) for data cleaning and hierarchical joining. Now it’s time for Python to take the wheel (car pun still intended) for the rest of our lab! For more information about running Python models on dbt, check out our [docs](https://docs.getdbt.com/docs/build/python-models.md). To learn more about dbt python works under the hood, check out [Snowpark for Python](https://docs.snowflake.com/en/developer-guide/snowpark/python/index.html), which makes running dbt Python models possible. There are quite a few differences between SQL and Python in terms of the dbt syntax and DDL, so we’ll be breaking our code and model runs down further for our python models. ##### Pit stop analysis First, we want to find out: which constructor had the fastest pit stops in 2021? (constructor is a Formula 1 team that builds or “constructs” the car). 1. Create a new file called `fastest_pit_stops_by_constructor.py` in our `aggregates` (this is the first time we are using the `.py` extension!). 2. Copy the following code into the file: ```python import numpy as np import pandas as pd def model(dbt, session): # dbt configuration dbt.config(packages=["pandas","numpy"]) # get upstream data pit_stops_joined = dbt.ref("pit_stops_joined").to_pandas() # provide year so we do not hardcode dates year=2021 # describe the data pit_stops_joined["PIT_STOP_SECONDS"] = pit_stops_joined["PIT_STOP_MILLISECONDS"]/1000 fastest_pit_stops = pit_stops_joined[(pit_stops_joined["RACE_YEAR"]==year)].groupby(by="CONSTRUCTOR_NAME")["PIT_STOP_SECONDS"].describe().sort_values(by='mean') fastest_pit_stops.reset_index(inplace=True) fastest_pit_stops.columns = fastest_pit_stops.columns.str.upper() return fastest_pit_stops.round(2) ``` 3. Let’s break down what this code is doing step by step: * First, we are importing the Python libraries that we are using. A *library* is a reusable chunk of code that someone else wrote that you may want to include in your programs/projects. We are using `numpy` and `pandas`in this Python model. This is similar to a dbt *package*, but our Python libraries do *not* persist across the entire project. * Defining a function called `model` with the parameter `dbt` and `session`. The parameter `dbt` is a class compiled by dbt, which enables you to run your Python code in the context of your dbt project and DAG. The parameter `session` is a class representing your Snowflake’s connection to the Python backend. The `model` function *must return a single DataFrame*. You can see that all the data transformation happening is within the body of the `model` function that the `return` statement is tied to. * Then, within the context of our dbt model library, we are passing in a configuration of which packages we need using `dbt.config(packages=["pandas","numpy"])`. * Use the `.ref()` function to retrieve the data frame `pit_stops_joined` that we created in our last step using SQL. We cast this to a pandas dataframe (by default it's a Snowpark Dataframe). * Create a variable named `year` so we aren’t passing a hardcoded value. * Generate a new column called `PIT_STOP_SECONDS` by dividing the value of `PIT_STOP_MILLISECONDS` by 1000. * Create our final data frame `fastest_pit_stops` that holds the records where year is equal to our year variable (2021 in this case), then group the data frame by `CONSTRUCTOR_NAME` and use the `describe()` and `sort_values()` and in descending order. This will make our first row in the new aggregated data frame the team with the fastest pit stops over an entire competition year. * Finally, it resets the index of the `fastest_pit_stops` data frame. The `reset_index()` method allows you to reset the index back to the default 0, 1, 2, etc indexes. By default, this method will keep the "old" indexes in a column named "index"; to avoid this, use the drop parameter. Think of this as keeping your data “flat and square” as opposed to “tiered”. If you are new to Python, now might be a good time to [learn about indexes for 5 minutes](https://towardsdatascience.com/the-basics-of-indexing-and-slicing-python-lists-2d12c90a94cf) since it's the foundation of how Python retrieves, slices, and dices data. The `inplace` argument means we override the existing data frame permanently. Not to fear! This is what we want to do to avoid dealing with multi-indexed dataframes! * Convert our Python column names to all uppercase using `.upper()`, so Snowflake recognizes them. * Finally we are returning our dataframe with 2 decimal places for all the columns using the `round()` method. 4. Zooming out a bit, what are we doing differently here in Python from our typical SQL code: * Method chaining is a technique in which multiple methods are called on an object in a single statement, with each method call modifying the result of the previous one. The methods are called in a chain, with the output of one method being used as the input for the next one. The technique is used to simplify the code and make it more readable by eliminating the need for intermediate variables to store the intermediate results. * The way you see method chaining in Python is the syntax `.().()`. For example, `.describe().sort_values(by='mean')` where the `.describe()` method is chained to `.sort_values()`. * The `.describe()` method is used to generate various summary statistics of the dataset. It's used on pandas dataframe. It gives a quick and easy way to get the summary statistics of your dataset without writing multiple lines of code. * The `.sort_values()` method is used to sort a pandas dataframe or a series by one or multiple columns. The method sorts the data by the specified column(s) in ascending or descending order. It is the pandas equivalent to `order by` in SQL. We won’t go as in depth for our subsequent scripts, but will continue to explain at a high level what new libraries, functions, and methods are doing. 5. Build the model using the UI which will **execute**: ```bash dbt run --select fastest_pit_stops_by_constructor ``` in the command bar. Let’s look at some details of our first Python model to see what our model executed. There two major differences we can see while running a Python model compared to an SQL model: * Our Python model was executed as a stored procedure. Snowflake needs a way to know that it's meant to execute this code in a Python runtime, instead of interpreting in a SQL runtime. We do this by creating a Python stored proc, called by a SQL command. * The `snowflake-snowpark-python` library has been picked up to execute our Python code. Even though this wasn’t explicitly stated this is picked up by the dbt class object because we need our Snowpark package to run Python! Python models take a bit longer to run than SQL models, however we could always speed this up by using [Snowpark-optimized Warehouses](https://docs.snowflake.com/en/user-guide/warehouses-snowpark-optimized.html) if we wanted to. Our data is sufficiently small, so we won’t worry about creating a separate warehouse for Python versus SQL files today. [![We can see our python model is run a stored procedure in our personal development schema](/img/guides/dbt-ecosystem/dbt-python-snowpark/10-python-transformations/1-python-model-details-output.png?v=2 "We can see our python model is run a stored procedure in our personal development schema")](#)We can see our python model is run a stored procedure in our personal development schema The rest of our **Details** output gives us information about how dbt and Snowpark for Python are working together to define class objects and apply a specific set of methods to run our models. So which constructor had the fastest pit stops in 2021? Let’s look at our data to find out! 6. We can't preview Python models directly, so let’s create a new file using the **+** button or the Control-n shortcut to create a new scratchpad. 7. Reference our Python model: ```sql select * from {{ ref('fastest_pit_stops_by_constructor') }} ``` and preview the output: [![Looking at our new python data model we can see that Red Bull had the fastest pit stops!](/img/guides/dbt-ecosystem/dbt-python-snowpark/10-python-transformations/2-fastest-pit-stops-preview.png?v=2 "Looking at our new python data model we can see that Red Bull had the fastest pit stops!")](#)Looking at our new python data model we can see that Red Bull had the fastest pit stops! Not only did Red Bull have the fastest average pit stops by nearly 40 seconds, they also had the smallest standard deviation, meaning they are both fastest and most consistent teams in pit stops. By using the `.describe()` method we were able to avoid verbose SQL requiring us to create a line of code per column and repetitively use the `PERCENTILE_COUNT()` function. Now we want to find the lap time average and rolling average through the years (is it generally trending up or down)? 8. Create a new file called `lap_times_moving_avg.py` in our `aggregates` folder. 9. Copy the following code into the file: ```python import pandas as pd def model(dbt, session): # dbt configuration dbt.config(packages=["pandas"]) # get upstream data lap_times = dbt.ref("int_lap_times_years").to_pandas() # describe the data lap_times["LAP_TIME_SECONDS"] = lap_times["LAP_TIME_MILLISECONDS"]/1000 lap_time_trends = lap_times.groupby(by="RACE_YEAR")["LAP_TIME_SECONDS"].mean().to_frame() lap_time_trends.reset_index(inplace=True) lap_time_trends["LAP_MOVING_AVG_5_YEARS"] = lap_time_trends["LAP_TIME_SECONDS"].rolling(5).mean() lap_time_trends.columns = lap_time_trends.columns.str.upper() return lap_time_trends.round(1) ``` 10. Breaking down our code a bit: * We’re only using the `pandas` library for this model and casting it to a pandas data frame `.to_pandas()`. * Generate a new column called `LAP_TIMES_SECONDS` by dividing the value of `LAP_TIME_MILLISECONDS` by 1000. * Create the final dataframe. Get the lap time per year. Calculate the mean series and convert to a data frame. * Reset the index. * Calculate the rolling 5 year mean. * Round our numeric columns to one decimal place. 11. Now, run this model by using the UI **Run model** or ```bash dbt run --select lap_times_moving_avg ``` in the command bar. 12. Once again previewing the output of our data using the same steps for our `fastest_pit_stops_by_constructor` model. [![Viewing our lap trends and 5 year rolling trends](/img/guides/dbt-ecosystem/dbt-python-snowpark/10-python-transformations/3-lap-times-trends-preview.png?v=2 "Viewing our lap trends and 5 year rolling trends")](#)Viewing our lap trends and 5 year rolling trends We can see that it looks like lap times are getting consistently faster over time. Then in 2010 we see an increase occur! Using outside subject matter context, we know that significant rule changes were introduced to Formula 1 in 2010 and 2011 causing slower lap times. 13. Now is a good time to checkpoint and commit our work to Git. Click **Commit and push** and give your commit a message like `aggregate python models` before moving on. ##### The dbt model, .source(), .ref() and .config() functions Let’s take a step back before starting machine learning to both review and go more in-depth at the methods that make running dbt python models possible. If you want to know more outside of this lab’s explanation read the documentation [here](https://docs.getdbt.com/docs/build/python-models.md?version=1). * dbt model(dbt, session). For starters, each Python model lives in a .py file in your models/ folder. It defines a function named `model()`, which takes two parameters: * dbt — A class compiled by dbt, unique to each model, enables you to run your Python code in the context of your dbt project and DAG. * session — A class representing your data platform’s connection to the Python backend. The session is needed to read in tables as DataFrames and to write DataFrames back to tables. In PySpark, by convention, the SparkSession is named spark, and available globally. For consistency across platforms, we always pass it into the model function as an explicit argument called session. * The `model()` function must return a single DataFrame. On Snowpark (Snowflake), this can be a Snowpark or pandas DataFrame. * `.source()` and `.ref()` functions. Python models participate fully in dbt's directed acyclic graph (DAG) of transformations. If you want to read directly from a raw source table, use `dbt.source()`. We saw this in our earlier section using SQL with the source function. These functions have the same execution, but with different syntax. Use the `dbt.ref()` method within a Python model to read data from other models (SQL or Python). These methods return DataFrames pointing to the upstream source, model, seed, or snapshot. * `.config()`. Just like SQL models, there are three ways to configure Python models: * In a dedicated `.yml` file, within the `models/` directory * Within the model's `.py` file, using the `dbt.config()` method * Calling the `dbt.config()` method will set configurations for your model within your `.py` file, similar to the `{{ config() }} macro` in `.sql` model files: ```python def model(dbt, session): # setting configuration dbt.config(materialized="table") ``` * There's a limit to how complex you can get with the `dbt.config()` method. It accepts only literal values (strings, booleans, and numeric types). Passing another function or a more complex data structure is not possible. The reason is that dbt statically analyzes the arguments to `.config()` while parsing your model without executing your Python code. If you need to set a more complex configuration, we recommend you define it using the config property in a [properties YAML file](https://docs.getdbt.com/reference/resource-properties/config.md). Learn more about configurations [here](https://docs.getdbt.com/reference/model-configs.md). #### Prepare for machine learning: cleaning, encoding, and splits Now that we’ve gained insights and business intelligence about Formula 1 at a descriptive level, we want to extend our capabilities into prediction. We’re going to take the scenario where we censor the data. This means that we will pretend that we will train a model using earlier data and apply it to future data. In practice, this means we’ll take data from 2010-2019 to train our model and then predict 2020 data. In this section, we’ll be preparing our data to predict the final race position of a driver. At a high level we’ll be: * Creating new prediction features and filtering our dataset to active drivers * Encoding our data (algorithms like numbers) and simplifying our target variable called `position` * Splitting our dataset into training, testing, and validation ##### ML data prep 1. To keep our project organized, we’ll need to create two new subfolders in our `ml` directory. Under the `ml` folder, make the subfolders `prep` and `train_predict`. 2. Create a new file under `ml/prep` called `ml_data_prep.py`. Copy the following code into the file and **Save**. ```python import pandas as pd def model(dbt, session): # dbt configuration dbt.config(packages=["pandas"]) # get upstream data fct_results = dbt.ref("fct_results").to_pandas() # provide years so we do not hardcode dates in filter command start_year=2010 end_year=2020 # describe the data for a full decade data = fct_results.loc[fct_results['RACE_YEAR'].between(start_year, end_year)] # convert string to an integer data['POSITION'] = data['POSITION'].astype(float) # we cannot have nulls if we want to use total pit stops data['TOTAL_PIT_STOPS_PER_RACE'] = data['TOTAL_PIT_STOPS_PER_RACE'].fillna(0) # some of the constructors changed their name over the year so replacing old names with current name mapping = {'Force India': 'Racing Point', 'Sauber': 'Alfa Romeo', 'Lotus F1': 'Renault', 'Toro Rosso': 'AlphaTauri'} data['CONSTRUCTOR_NAME'].replace(mapping, inplace=True) # create confidence metrics for drivers and constructors dnf_by_driver = data.groupby('DRIVER').sum(numeric_only=True)['DNF_FLAG'] driver_race_entered = data.groupby('DRIVER').count()['DNF_FLAG'] driver_dnf_ratio = (dnf_by_driver/driver_race_entered) driver_confidence = 1-driver_dnf_ratio driver_confidence_dict = dict(zip(driver_confidence.index,driver_confidence)) dnf_by_constructor = data.groupby('CONSTRUCTOR_NAME').sum(numeric_only=True)['DNF_FLAG'] constructor_race_entered = data.groupby('CONSTRUCTOR_NAME').count()['DNF_FLAG'] constructor_dnf_ratio = (dnf_by_constructor/constructor_race_entered) constructor_relaiblity = 1-constructor_dnf_ratio constructor_relaiblity_dict = dict(zip(constructor_relaiblity.index,constructor_relaiblity)) data['DRIVER_CONFIDENCE'] = data['DRIVER'].apply(lambda x:driver_confidence_dict[x]) data['CONSTRUCTOR_RELAIBLITY'] = data['CONSTRUCTOR_NAME'].apply(lambda x:constructor_relaiblity_dict[x]) #removing retired drivers and constructors active_constructors = ['Renault', 'Williams', 'McLaren', 'Ferrari', 'Mercedes', 'AlphaTauri', 'Racing Point', 'Alfa Romeo', 'Red Bull', 'Haas F1 Team'] active_drivers = ['Daniel Ricciardo', 'Kevin Magnussen', 'Carlos Sainz', 'Valtteri Bottas', 'Lance Stroll', 'George Russell', 'Lando Norris', 'Sebastian Vettel', 'Kimi Räikkönen', 'Charles Leclerc', 'Lewis Hamilton', 'Daniil Kvyat', 'Max Verstappen', 'Pierre Gasly', 'Alexander Albon', 'Sergio Pérez', 'Esteban Ocon', 'Antonio Giovinazzi', 'Romain Grosjean','Nicholas Latifi'] # create flags for active drivers and constructors so we can filter downstream data['ACTIVE_DRIVER'] = data['DRIVER'].apply(lambda x: int(x in active_drivers)) data['ACTIVE_CONSTRUCTOR'] = data['CONSTRUCTOR_NAME'].apply(lambda x: int(x in active_constructors)) return data ``` 3. As usual, let’s break down what we are doing in this Python model: * We’re first referencing our upstream `fct_results` table and casting it to a pandas dataframe. * Filtering on years 2010-2020 since we’ll need to clean all our data we are using for prediction (both training and testing). * Filling in empty data for `total_pit_stops` and making a mapping active constructors and drivers to avoid erroneous predictions * ⚠️ You might be wondering why we didn’t do this upstream in our `fct_results` table! The reason for this is that we want our machine learning cleanup to reflect the year 2020 for our predictions and give us an up-to-date team name. However, for business intelligence purposes we can keep the historical data at that point in time. Instead of thinking of one table as “one source of truth” we are creating different datasets fit for purpose: one for historical descriptions and reporting and another for relevant predictions. * Create new confidence features for drivers and constructors * Generate flags for the constructors and drivers that were active in 2020 4. Execute the following in the command bar: ```bash dbt run --select ml_data_prep ``` 5. There are more aspects we could consider for this project, such as normalizing the driver confidence by the number of races entered. Including this would help account for a driver’s history and consider whether they are a new or long-time driver. We’re going to keep it simple for now, but these are some of the ways we can expand and improve our machine learning dbt projects. Breaking down our machine learning prep model: * Lambda functions — We use some lambda functions to transform our data without having to create a fully-fledged function using the `def` notation. So what exactly are lambda functions? * In Python, a lambda function is a small, anonymous function defined using the keyword "lambda". Lambda functions are used to perform a quick operation, such as a mathematical calculation or a transformation on a list of elements. They are often used in conjunction with higher-order functions, such as `apply`, `map`, `filter`, and `reduce`. * `.apply()` method — We used `.apply()` to pass our functions into our lambda expressions to the columns and perform this multiple times in our code. Let’s explain apply a little more: * The `.apply()` function in the pandas library is used to apply a function to a specified axis of a DataFrame or a Series. In our case the function we used was our lambda function! * The `.apply()` function takes two arguments: the first is the function to be applied, and the second is the axis along which the function should be applied. The axis can be specified as 0 for rows or 1 for columns. We are using the default value of 0 so we aren’t explicitly writing it in the code. This means that the function will be applied to each *row* of the DataFrame or Series. 6. Let’s look at the preview of our clean dataframe after running our `ml_data_prep` model: [![What our clean dataframe fit for machine learning looks like](/img/guides/dbt-ecosystem/dbt-python-snowpark/11-machine-learning-prep/1-completed-ml-data-prep.png?v=2 "What our clean dataframe fit for machine learning looks like")](#)What our clean dataframe fit for machine learning looks like ##### Covariate encoding In this next part, we’ll be performing covariate encoding. Breaking down this phrase a bit, a *covariate* is a variable that is relevant to the outcome of a study or experiment, and *encoding* refers to the process of converting data (such as text or categorical variables) into a numerical format that can be used as input for a model. This is necessary because most machine learning algorithms can only work with numerical data. Algorithms don’t speak languages, have eyes to see images, etc. so we encode our data into numbers so algorithms can perform tasks by using calculations they otherwise couldn’t. 🧠 We’ll think about this as : “algorithms like numbers”. 1. Create a new file under `ml/prep` called `covariate_encoding` copy the code below and save. ```python import pandas as pd import numpy as np from sklearn.preprocessing import StandardScaler,LabelEncoder,OneHotEncoder from sklearn.linear_model import LogisticRegression def model(dbt, session): # dbt configuration dbt.config(packages=["pandas","numpy","scikit-learn"]) # get upstream data data = dbt.ref("ml_data_prep").to_pandas() # list out covariates we want to use in addition to outcome variable we are modeling - position covariates = data[['RACE_YEAR','CIRCUIT_NAME','GRID','CONSTRUCTOR_NAME','DRIVER','DRIVERS_AGE_YEARS','DRIVER_CONFIDENCE','CONSTRUCTOR_RELAIBLITY','TOTAL_PIT_STOPS_PER_RACE','ACTIVE_DRIVER','ACTIVE_CONSTRUCTOR', 'POSITION']] # filter covariates on active drivers and constructors # use fil_cov as short for "filtered_covariates" fil_cov = covariates[(covariates['ACTIVE_DRIVER']==1)&(covariates['ACTIVE_CONSTRUCTOR']==1)] # Encode categorical variables using LabelEncoder # TODO: we'll update this to both ohe in the future for non-ordinal variables! le = LabelEncoder() fil_cov['CIRCUIT_NAME'] = le.fit_transform(fil_cov['CIRCUIT_NAME']) fil_cov['CONSTRUCTOR_NAME'] = le.fit_transform(fil_cov['CONSTRUCTOR_NAME']) fil_cov['DRIVER'] = le.fit_transform(fil_cov['DRIVER']) fil_cov['TOTAL_PIT_STOPS_PER_RACE'] = le.fit_transform(fil_cov['TOTAL_PIT_STOPS_PER_RACE']) # Simply target variable "position" to represent 3 meaningful categories in Formula1 # 1. Podium position 2. Points for team 3. Nothing - no podium or points! def position_index(x): if x<4: return 1 if x>10: return 3 else : return 2 # we are dropping the columns that we filtered on in addition to our training variable encoded_data = fil_cov.drop(['ACTIVE_DRIVER','ACTIVE_CONSTRUCTOR'],axis=1)) encoded_data['POSITION_LABEL']= encoded_data['POSITION'].apply(lambda x: position_index(x)) encoded_data_grouped_target = encoded_data.drop(['POSITION'],axis=1)) return encoded_data_grouped_target ``` 2. Execute the following in the command bar: ```bash dbt run --select covariate_encoding ``` 3. In this code, we are using a ton of functions from libraries! This is really cool, because we can utilize code other people have developed and bring it into our project simply by using the `import` function. [Scikit-learn](https://scikit-learn.org/stable/), “sklearn” for short, is an extremely popular data science library. Sklearn contains a wide range of machine learning techniques, including supervised and unsupervised learning algorithms, feature scaling and imputation, as well as tools model evaluation and selection. We’ll be using Sklearn for both preparing our covariates and creating models (our next section). 4. Our dataset is pretty small data so we are good to use pandas and `sklearn`. If you have larger data for your own project in mind, consider `dask` or `category_encoders`. 5. Breaking it down a bit more: * We’re selecting a subset of variables that will be used as predictors for a driver’s position. * Filter the dataset to only include rows using the active driver and constructor flags we created in the last step. * The next step is to use the `LabelEncoder` from scikit-learn to convert the categorical variables `CIRCUIT_NAME`, `CONSTRUCTOR_NAME`, `DRIVER`, and `TOTAL_PIT_STOPS_PER_RACE` into numerical values. * Create a new variable called `POSITION_LABEL`, which is a derived from our position variable. * 💭 Why are we changing our position variable? There are 20 total positions in Formula 1 and we are grouping them together to simplify the classification and improve performance. We also want to demonstrate you can create a new function within your dbt model! * Our new `position_label` variable has meaning: * In Formula1 if you are in: * Top 3 you get a “podium” position * Top 10 you gain points that add to your overall season total * Below top 10 you get no points! * We are mapping our original variable position to `position_label` to the corresponding places above to 1,2, and 3 respectively. * Drop the active driver and constructor flags since they were filter criteria and additionally drop our original position variable. ##### Splitting into training and testing datasets Now that we’ve cleaned and encoded our data, we are going to further split in by time. In this step, we will create dataframes to use for training and prediction. We’ll be creating two dataframes 1) using data from 2010-2019 for training, and 2) data from 2020 for new prediction inferences. We’ll create variables called `start_year` and `end_year` so we aren’t filtering on hardcasted values (and can more easily swap them out in the future if we want to retrain our model on different timeframes). 1. Create a file called `train_test_dataset.py` copy and save the following code: ```python import pandas as pd def model(dbt, session): # dbt configuration dbt.config(packages=["pandas"], tags="train") # get upstream data encoding = dbt.ref("covariate_encoding").to_pandas() # provide years so we do not hardcode dates in filter command start_year=2010 end_year=2019 # describe the data for a full decade train_test_dataset = encoding.loc[encoding['RACE_YEAR'].between(start_year, end_year)] return train_test_dataset ``` 2. Create a file called `hold_out_dataset_for_prediction.py` copy and save the following code below. Now we’ll have a dataset with only the year 2020 that we’ll keep as a hold out set that we are going to use similar to a deployment use case. ```python import pandas as pd def model(dbt, session): # dbt configuration dbt.config(packages=["pandas"], tags="predict") # get upstream data encoding = dbt.ref("covariate_encoding").to_pandas() # variable for year instead of hardcoding it year=2020 # filter the data based on the specified year hold_out_dataset = encoding.loc[encoding['RACE_YEAR'] == year] return hold_out_dataset ``` 3. Execute the following in the command bar: ```bash dbt run --select train_test_dataset hold_out_dataset_for_prediction ``` To run our temporal data split models, we can use this syntax in the command line to run them both at once. Make sure you use a *space* [syntax](https://docs.getdbt.com/reference/node-selection/syntax.md) between the model names to indicate you want to run both! 4. **Commit and push** our changes to keep saving our work as we go using `ml data prep and splits` before moving on. 👏 Now that we’ve finished our machine learning prep work we can move onto the fun part — training and prediction! #### Training a model to predict in machine learning We’re ready to start training a model to predict the driver’s position. Now is a good time to pause and take a step back and say, usually in ML projects you’ll try multiple algorithms during development and use an evaluation method such as cross validation to determine which algorithm to use. You can definitely do this in your dbt project, but for the content of this lab we’ll have decided on using a logistic regression to predict position (we actually tried some other algorithms using cross validation outside of this lab such as k-nearest neighbors and a support vector classifier but that didn’t perform as well as the logistic regression and a decision tree that overfit). There are 3 areas to break down as we go since we are working at the intersection all within one model file: 1. Machine Learning 2. Snowflake and Snowpark 3. dbt Python models If you haven’t seen code like this before or use joblib files to save machine learning models, we’ll be going over them at a high level and you can explore the links for more technical in-depth along the way! Because Snowflake and dbt have abstracted away a lot of the nitty gritty about serialization and storing our model object to be called again, we won’t go into too much detail here. There’s *a lot* going on here so take it at your pace! ##### Training and saving a machine learning model 1. Project organization remains key, so let’s make a new subfolder called `train_predict` under the `ml` folder. 2. Now create a new file called `train_test_position.py` and copy and save the following code: ```python import snowflake.snowpark.functions as F from sklearn.model_selection import train_test_split import pandas as pd from sklearn.metrics import confusion_matrix, balanced_accuracy_score import io from sklearn.linear_model import LogisticRegression from joblib import dump, load import joblib import logging import sys from joblib import dump, load logger = logging.getLogger("mylog") def save_file(session, model, path, dest_filename): input_stream = io.BytesIO() joblib.dump(model, input_stream) session._conn.upload_stream(input_stream, path, dest_filename) return "successfully created file: " + path def model(dbt, session): dbt.config( packages = ['numpy','scikit-learn','pandas','numpy','joblib','cachetools'], materialized = "table", tags = "train" ) # Create a stage in Snowflake to save our model file session.sql('create or replace stage MODELSTAGE').collect() #session._use_scoped_temp_objects = False version = "1.0" logger.info('Model training version: ' + version) # read in our training and testing upstream dataset test_train_df = dbt.ref("train_test_dataset") # cast snowpark df to pandas df test_train_pd_df = test_train_df.to_pandas() target_col = "POSITION_LABEL" # split out covariate predictors, x, from our target column position_label, y. split_X = test_train_pd_df.drop([target_col], axis=1) split_y = test_train_pd_df[target_col] # Split out our training and test data into proportions X_train, X_test, y_train, y_test = train_test_split(split_X, split_y, train_size=0.7, random_state=42) train = [X_train, y_train] test = [X_test, y_test] # now we are only training our one model to deploy # we are keeping the focus on the workflows and not algorithms for this lab! model = LogisticRegression() # fit the preprocessing pipeline and the model together model.fit(X_train, y_train) y_pred = model.predict_proba(X_test)[:,1] predictions = [round(value) for value in y_pred] balanced_accuracy = balanced_accuracy_score(y_test, predictions) # Save the model to a stage save_file(session, model, "@MODELSTAGE/driver_position_"+version, "driver_position_"+version+".joblib" ) logger.info('Model artifact:' + "@MODELSTAGE/driver_position_"+version+".joblib") # Take our pandas training and testing dataframes and put them back into snowpark dataframes snowpark_train_df = session.write_pandas(pd.concat(train, axis=1, join='inner'), "train_table", auto_create_table=True, create_temp_table=True) snowpark_test_df = session.write_pandas(pd.concat(test, axis=1, join='inner'), "test_table", auto_create_table=True, create_temp_table=True) # Union our training and testing data together and add a column indicating train vs test rows return snowpark_train_df.with_column("DATASET_TYPE", F.lit("train")).union(snowpark_test_df.with_column("DATASET_TYPE", F.lit("test"))) ``` 3. Execute the following in the command bar: ```bash dbt run --select train_test_position ``` 4. Breaking down our Python script here: * We’re importing some helpful libraries. * Defining a function called `save_file()` that takes four parameters: `session`, `model`, `path` and `dest_filename` that will save our logistic regression model file. * `session` — an object representing a connection to Snowflake. * `model` — an object that needs to be saved. In this case, it's a Python object that is a scikit-learn that can be serialized with joblib. * `path` — a string representing the directory or bucket location where the file should be saved. * `dest_filename` — a string representing the desired name of the file. * Creating our dbt model * Within this model we are creating a stage called `MODELSTAGE` to place our logistic regression `joblib` model file. This is really important since we need a place to keep our model to reuse and want to ensure it's there. When using Snowpark commands, it's common to see the `.collect()` method to ensure the action is performed. Think of the session as our “start” and collect as our “end” when [working with Snowpark](https://docs.snowflake.com/en/developer-guide/snowpark/python/working-with-dataframes.html) (you can use other ending methods other than collect). * Using `.ref()` to connect into our `train_test_dataset` model. * Now we see the machine learning part of our analysis: * Create new dataframes for our prediction features from our target variable `position_label`. * Split our dataset into 70% training (and 30% testing), train\_size=0.7 with a `random_state` specified to have repeatable results. * Specify our model is a logistic regression. * Fit our model. In a logistic regression this means finding the coefficients that will give the least classification error. * Round our predictions to the nearest integer since logistic regression creates a probability between for each class and calculate a balanced accuracy to account for imbalances in the target variable. * Right now our model is only in memory, so we need to use our nifty function `save_file` to save our model file to our Snowflake stage. We save our model as a joblib file so Snowpark can easily call this model object back to create predictions. We really don’t need to know much else as a data practitioner unless we want to. It’s worth noting that joblib files aren’t able to be queried directly by SQL. To do this, we would need to transform the joblib file to an SQL querable format such as JSON or CSV (out of scope for this workshop). * Finally we want to return our dataframe, but create a new column indicating what rows were used for training and those for training. 5. Viewing our output of this model: [![Preview which rows of our model were used for training and testing](/img/guides/dbt-ecosystem/dbt-python-snowpark/12-machine-learning-training-prediction/1-preview-train-test-position.png?v=2 "Preview which rows of our model were used for training and testing")](#)Preview which rows of our model were used for training and testing 6. Let’s pop back over to Snowflake and check that our logistic regression model has been stored in our `MODELSTAGE` using the command: ```sql list @modelstage ``` [![List the objects in our Snowflake stage to check for our logistic regression to predict driver position](/img/guides/dbt-ecosystem/dbt-python-snowpark/12-machine-learning-training-prediction/2-list-snowflake-stage.png?v=2 "List the objects in our Snowflake stage to check for our logistic regression to predict driver position")](#)List the objects in our Snowflake stage to check for our logistic regression to predict driver position 7. To investigate the commands run as part of `train_test_position` script, navigate to Snowflake query history to view it **Activity > Query History**. We can view the portions of query that we wrote such as `create or replace stage MODELSTAGE`, but we also see additional queries that Snowflake uses to interpret python code. [![View Snowflake query history to see how python models are run under the hood](/img/guides/dbt-ecosystem/dbt-python-snowpark/12-machine-learning-training-prediction/3-view-snowflake-query-history.png?v=2 "View Snowflake query history to see how python models are run under the hood")](#)View Snowflake query history to see how python models are run under the hood ##### Predicting on new data 1. Create a new file called `predict_position.py` and copy and save the following code: ```python import logging import joblib import pandas as pd import os from snowflake.snowpark import types as T DB_STAGE = 'MODELSTAGE' version = '1.0' # The name of the model file model_file_path = 'driver_position_'+version model_file_packaged = 'driver_position_'+version+'.joblib' # This is a local directory, used for storing the various artifacts locally LOCAL_TEMP_DIR = f'/tmp/driver_position' DOWNLOAD_DIR = os.path.join(LOCAL_TEMP_DIR, 'download') TARGET_MODEL_DIR_PATH = os.path.join(LOCAL_TEMP_DIR, 'ml_model') TARGET_LIB_PATH = os.path.join(LOCAL_TEMP_DIR, 'lib') # The feature columns that were used during model training # and that will be used during prediction FEATURE_COLS = [ "RACE_YEAR" ,"CIRCUIT_NAME" ,"GRID" ,"CONSTRUCTOR_NAME" ,"DRIVER" ,"DRIVERS_AGE_YEARS" ,"DRIVER_CONFIDENCE" ,"CONSTRUCTOR_RELAIBLITY" ,"TOTAL_PIT_STOPS_PER_RACE"] def register_udf_for_prediction(p_predictor ,p_session ,p_dbt): # The prediction udf def predict_position(p_df: T.PandasDataFrame[int, int, int, int, int, int, int, int, int]) -> T.PandasSeries[int]: # Snowpark currently does not set the column name in the input dataframe # The default col names are like 0,1,2,... Hence we need to reset the column # names to the features that we initially used for training. p_df.columns = [*FEATURE_COLS] # Perform prediction. this returns an array object pred_array = p_predictor.predict(p_df) # Convert to series df_predicted = pd.Series(pred_array) return df_predicted # The list of packages that will be used by UDF udf_packages = p_dbt.config.get('packages') predict_position_udf = p_session.udf.register( predict_position ,name=f'predict_position' ,packages = udf_packages ) return predict_position_udf def download_models_and_libs_from_stage(p_session): p_session.file.get(f'@{DB_STAGE}/{model_file_path}/{model_file_packaged}', DOWNLOAD_DIR) def load_model(p_session): # Load the model and initialize the predictor model_fl_path = os.path.join(DOWNLOAD_DIR, model_file_packaged) predictor = joblib.load(model_fl_path) return predictor # ------------------------------- def model(dbt, session): dbt.config( packages = ['snowflake-snowpark-python' ,'scipy','scikit-learn' ,'pandas' ,'numpy'], materialized = "table", tags = "predict" ) session._use_scoped_temp_objects = False download_models_and_libs_from_stage(session) predictor = load_model(session) predict_position_udf = register_udf_for_prediction(predictor, session ,dbt) # Retrieve the data, and perform the prediction hold_out_df = (dbt.ref("hold_out_dataset_for_prediction") .select(*FEATURE_COLS) ) # Perform prediction. new_predictions_df = hold_out_df.withColumn("position_predicted" ,predict_position_udf(*FEATURE_COLS) ) return new_predictions_df ``` 2. Execute the following in the command bar: ```bash dbt run --select predict_position ``` 3. **Commit and push** our changes to keep saving our work as we go using the commit message `logistic regression model training and application` before moving on. 4. At a high level in this script, we are: * Retrieving our staged logistic regression model * Loading the model in * Placing the model within a user defined function (UDF) to call in line predictions on our driver’s position 5. At a more detailed level: * Import our libraries. * Create variables to reference back to the `MODELSTAGE` we just created and stored our model to. * The temporary file paths we created might look intimidating, but all we’re doing here is programmatically using an initial file path and adding to it to create the following directories: * LOCAL\_TEMP\_DIR ➡️ /tmp/driver\_position * DOWNLOAD\_DIR ➡️ /tmp/driver\_position/download * TARGET\_MODEL\_DIR\_PATH ➡️ /tmp/driver\_position/ml\_model * TARGET\_LIB\_PATH ➡️ /tmp/driver\_position/lib * Provide a list of our feature columns that we used for model training and will now be used on new data for prediction. * Next, we are creating our main function `register_udf_for_prediction(p_predictor ,p_session ,p_dbt):`. This function is used to register a user-defined function (UDF) that performs the machine learning prediction. It takes three parameters: `p_predictor` is an instance of the machine learning model, `p_session` is an instance of the Snowflake session, and `p_dbt` is an instance of the dbt library. The function creates a UDF named `predict_churn` which takes a pandas dataframe with the input features and returns a pandas series with the predictions. * ⚠️ Pay close attention to the whitespace here. We are using a function within a function for this script. * We have 2 simple functions that are programmatically retrieving our file paths to first get our stored model out of our `MODELSTAGE` and downloaded into the session `download_models_and_libs_from_stage` and then to load the contents of our model in (parameters) in `load_model` to use for prediction. * Take the model we loaded in and call it `predictor` and wrap it in a UDF. * Return our dataframe with both the features used to predict and the new label. 🧠 Another way to read this script is from the bottom up. This can help us progressively see what is going into our final dbt model and work backwards to see how the other functions are being referenced. 6. Let’s take a look at our predicted position alongside our feature variables. Open a new scratchpad and use the following query. I chose to order by the prediction of who would obtain a podium position: ```sql select * from {{ ref('predict_position') }} order by position_predicted ``` 7. We can see that we created predictions in our final dataset, we are ready to move on to testing! #### Test your data models We have now completed building all the models for today’s lab, but how do we know if they meet our assertions? Put another way, how do we know the quality of our data models are any good? This brings us to testing! We test data models for mainly two reasons: * Ensure that our source data is clean on ingestion before we start data modeling/transformation (aka avoid garbage in, garbage out problem). * Make sure we don’t introduce bugs in the transformation code we wrote (stop ourselves from creating bad joins/fanouts). Testing in dbt comes in two flavors: [generic](https://docs.getdbt.com/docs/build/data-tests.md#generic-data-tests) and [singular](https://docs.getdbt.com/docs/build/data-tests.md#singular-data-tests). You define them in a test block (similar to a macro) and once defined, you can reference them by name in your `.yml` files (applying them to models, columns, sources, snapshots, and seeds). You might be wondering: *what about testing Python models?* Since the output of our Python models are tables, we can test SQL and Python models the same way! We don’t have to worry about any syntax differences when testing SQL versus Python data models. This means we use `.yml` and `.sql` files to test our entities (tables, views, etc.). Under the hood, dbt is running an SQL query on our tables to see if they meet assertions. If no rows are returned, dbt will surface a passed test. Conversely, if a test results in returned rows, it will fail or warn depending on the configuration (more on that later). ##### Generic tests 1. To implement generic out-of-the-box tests dbt comes with, we can use YAML files to specify information about our models. To add generic tests to our aggregates model, create a file called `aggregates.yml`, copy the code block below into the file, and save. [![The aggregates.yml file in our file tree](/img/guides/dbt-ecosystem/dbt-python-snowpark/13-testing/1-generic-testing-file-tree.png?v=2 "The aggregates.yml file in our file tree")](#)The aggregates.yml file in our file tree ```yaml models: - name: fastest_pit_stops_by_constructor description: Use the python .describe() method to retrieve summary statistics table about pit stops by constructor. Sort by average stop time ascending so the first row returns the fastest constructor. columns: - name: constructor_name description: team that makes the car data_tests: - unique - name: lap_times_moving_avg description: Use the python .rolling() method to calculate the 5 year rolling average of pit stop times alongside the average for each year. columns: - name: race_year description: year of the race data_tests: - relationships: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. to: ref('int_lap_times_years') field: race_year ``` 2. Let’s unpack the code we have here. We have both our aggregates models with the model name to know the object we are referencing and the description of the model that we’ll populate in our documentation. At the column level (a level below our model), we are providing the column name followed by our tests. We want to ensure our `constructor_name` is unique since we used a pandas `groupby` on `constructor_name` in the model `fastest_pit_stops_by_constructor`. Next, we want to ensure our `race_year` has referential integrity from the model we selected from `int_lap_times_years` into our subsequent `lap_times_moving_avg` model. 3. Finally, if we want to see how tests were deployed on sources and SQL models, we can look at other files in our project such as the `f1_sources.yml` we created in our Sources and staging section. ##### Using macros for testing 1. Under your `macros` folder, create a new file and name it `test_all_values_gte_zero.sql`. Copy the code block below and save the file. For clarity, “gte” is an abbreviation for greater than or equal to. [![macro file for reusable testing code](/img/guides/dbt-ecosystem/dbt-python-snowpark/13-testing/2-macro-testing.png?v=2 "macro file for reusable testing code")](#)macro file for reusable testing code ```sql {% macro test_all_values_gte_zero(table, column) %} select * from {{ ref(table) }} where {{ column }} < 0 {% endmacro %} ``` 2. Macros in Jinja are pieces of code that can be reused multiple times in our SQL models — they are analogous to "functions" in other programming languages, and are extremely useful if you find yourself repeating code across multiple models. 3. We use the `{% macro %}` to indicate the start of the macro and `{% endmacro %}` for the end. The text after the beginning of the macro block is the name we are giving the macro to later call it. In this case, our macro is called `test_all_values_gte_zero`. Macros take in *arguments* to pass through, in this case the `table` and the `column`. In the body of the macro, we see an SQL statement that is using the `ref` function to dynamically select the table and then the column. You can always view macros without having to run them by using `dbt run-operation`. You can learn more [here](https://docs.getdbt.com/reference/commands/run-operation.md). 4. Great, now we want to reference this macro as a test! Let’s create a new test file called `macro_pit_stops_mean_is_positive.sql` in our `tests` folder. [![creating a test on our pit stops model referencing the macro](/img/guides/dbt-ecosystem/dbt-python-snowpark/13-testing/3-gte-macro-applied-to-pit-stops.png?v=2 "creating a test on our pit stops model referencing the macro")](#)creating a test on our pit stops model referencing the macro 5. Copy the following code into the file and save: ```sql {{ config( enabled=true, severity='warn', tags = ['bi'] ) }} {{ test_all_values_gte_zero('fastest_pit_stops_by_constructor', 'mean') }} ``` 6. In our testing file, we are applying some configurations to the test including `enabled`, which is an optional configuration for disabling models, seeds, snapshots, and tests. Our severity is set to `warn` instead of `error`, which means our pipeline will still continue to run. We have tagged our test with `bi` since we are applying this test to one of our bi models. Then, in our final line, we are calling the `test_all_values_gte_zero` macro that takes in our table and column arguments and inputting our table `'fastest_pit_stops_by_constructor'` and the column `'mean'`. ##### Custom singular tests to validate Python models The simplest way to define a test is by writing the exact SQL that will return failing records. We call these "singular" tests, because they're one-off assertions usable for a single purpose. These tests are defined in `.sql` files, typically in your `tests` directory (as defined by your test-paths config). You can use Jinja in SQL models (including ref and source) in the test definition, just like you can when creating models. Each `.sql` file contains one select statement, and it defines one test. Let’s add a custom test that asserts that the moving average of the lap time over the last 5 years is greater than zero (it’s impossible to have time less than 0!). It is easy to assume if this is not the case the data has been corrupted. 1. Create a file `lap_times_moving_avg_assert_positive_or_null.sql` under the `tests` folder. [![custom singular test for testing lap times are positive values](/img/guides/dbt-ecosystem/dbt-python-snowpark/13-testing/4-custom-singular-test.png?v=2 "custom singular test for testing lap times are positive values")](#)custom singular test for testing lap times are positive values 2. Copy the following code and save the file: ```sql {{ config( enabled=true, severity='error', tags = ['bi'] ) }} with lap_times_moving_avg as ( select * from {{ ref('lap_times_moving_avg') }} ) select * from lap_times_moving_avg where lap_moving_avg_5_years < 0 and lap_moving_avg_5_years is not null ``` ##### Putting all our tests together 1. Time to run our tests! Altogether, we have created 4 tests for our 2 Python models: * `fastest_pit_stops_by_constructor` * Unique `constructor_name` * Lap times are greater than 0 or null (to allow for the first leading values in a rolling calculation) * `lap_times_moving_avg` * Referential test on `race_year` * Mean pit stop times are greater than or equal to 0 (no negative time values) 2. To run the tests on both our models, we can use this syntax in the command line to run them both at once, similar to how we did our data splits earlier. Execute the following in the command bar: ```bash dbt test --select fastest_pit_stops_by_constructor lap_times_moving_avg ``` [![running tests on our python models](/img/guides/dbt-ecosystem/dbt-python-snowpark/13-testing/5-running-tests-on-python-models.png?v=2 "running tests on our python models")](#)running tests on our python models 3. All 4 of our tests passed (yay for clean data)! To understand the SQL being run against each of our tables, we can click into the details of the test. 4. Navigating into the **Details** of the `unique_fastest_pit_stops_by_constructor_name`, we can see that each line `constructor_name` should only have one row. [![view details of testing our python model that used SQL to test data assertions](/img/guides/dbt-ecosystem/dbt-python-snowpark/13-testing/6-testing-output-details.png?v=2 "view details of testing our python model that used SQL to test data assertions")](#)view details of testing our python model that used SQL to test data assertions #### Document your dbt project When it comes to documentation, dbt brings together both column and model level descriptions that you can provide as well as details from your Snowflake information schema in a static site for consumption by other data team members and stakeholders. We are going to revisit 2 areas of our project to understand our documentation: * `intermediate.md` file * `dbt_project.yml` file To start, let’s look back at our `intermediate.md` file. We can see that we provided multi-line descriptions for the models in our intermediate models using [docs blocks](https://docs.getdbt.com/docs/build/documentation.md#using-docs-blocks). Then we reference these docs blocks in our `.yml` file. Building descriptions with doc blocks in Markdown files gives you the ability to format your descriptions with Markdown and are particularly helpful when building long descriptions, either at the column or model level. In our `dbt_project.yml`, we added `node_colors` at folder levels. 1. To see all these pieces come together, execute this in the command bar: ```bash dbt docs generate ``` This will generate the documentation for your project. Click the book button, as shown in the screenshot below to access the docs. [![dbt docs book icon](/img/guides/dbt-ecosystem/dbt-python-snowpark/14-documentation/1-docs-icon.png?v=2 "dbt docs book icon")](#)dbt docs book icon 2. Go to our project area and view `int_results`. View the description that we created in our doc block. [![Docblock description within docs site](/img/guides/dbt-ecosystem/dbt-python-snowpark/14-documentation/2-view-docblock-description.png?v=2 "Docblock description within docs site")](#)Docblock description within docs site 3. View the mini-lineage that looks at the model we are currently selected on (`int_results` in this case). [![Mini lineage view on docs site](/img/guides/dbt-ecosystem/dbt-python-snowpark/14-documentation/3-mini-lineage-docs.png?v=2 "Mini lineage view on docs site")](#)Mini lineage view on docs site 4. In our `dbt_project.yml`, we configured `node_colors` depending on the file directory. By color coding your project, it can help you cluster together similar models or steps and more easily troubleshoot when viewing lineage in your docs. [![Full project DAG on docs site](/img/guides/dbt-ecosystem/dbt-python-snowpark/14-documentation/4-full-dag-docs.png?v=2 "Full project DAG on docs site")](#)Full project DAG on docs site #### Deploy your code Before we jump into deploying our code, let's have a quick primer on environments. Up to this point, all of the work we've done in the Studio IDE has been in our development environment, with code committed to a feature branch and the models we've built created in our development schema in Snowflake as defined in our Development environment connection. Doing this work on a feature branch, allows us to separate our code from what other coworkers are building and code that is already deemed production ready. Building models in a development schema in Snowflake allows us to separate the database objects we might still be modifying and testing from the database objects running production dashboards or other downstream dependencies. Together, the combination of a Git branch and Snowflake database objects form our environment. Now that we've completed testing and documenting our work, we're ready to deploy our code from our development environment to our production environment and this involves two steps: * Promoting code from our feature branch to the production branch in our repository. * Generally, the production branch is going to be named your main branch and there's a review process to go through before merging code to the main branch of a repository. Here we are going to merge without review for ease of this workshop. * Deploying code to our production environment. * Once our code is merged to the main branch, we'll need to run dbt in our production environment to build all of our models and run all of our tests. This will allow us to build production-ready objects into our production environment in Snowflake. Luckily for us, the Partner Connect flow has already created our deployment environment and job to facilitate this step. 1. Before getting started, let's make sure that we've committed all of our work to our feature branch. If you still have work to commit, you'll be able to select the **Commit and push**, provide a message, and then select **Commit** again. 2. Once all of your work is committed, the git workflow button will now appear as **Merge this branch to main**. Click **Merge this branch to main** and the merge process will automatically run in the background. [![Merge this branch to main](/img/guides/dbt-ecosystem/dbt-python-snowpark/15-deployment/1-merge-to-main-branch.png?v=2 "Merge this branch to main")](#)Merge this branch to main 3. When it's completed, you should see the git button read **Create branch** and the branch you're currently looking at will become **main**. 4. Now that all of our development work has been merged to the main branch, we can build our deployment job. Given that our production environment and production job were created automatically for us through Partner Connect, all we need to do here is update some default configurations to meet our needs. 5. In the left-hand menu, go to **Orchestration** > **Environments**. 6. You should see two environments listed and you'll want to select the **Deployment** environment then **Settings** to modify it. 7. Before making any changes, let's touch on what is defined within this environment. The Snowflake connection shows the credentials that dbt is using for this environment and in our case they are the same as what was created for us through Partner Connect. Our deployment job will build in our `PC_DBT_DB` database and use the default Partner Connect role and warehouse to do so. The deployment credentials section also uses the info that was created in our Partner Connect job to create the credential connection. However, it is using the same default schema that we've been using as the schema for our development environment. 8. Let's update the schema to create a new schema specifically for our production environment. Click **Edit** to allow you to modify the existing field values. Navigate to **Deployment Credentials >** **schema.** 9. Update the schema name to **production**. Remember to select **Save** after you've made the change. [![Update the deployment credentials schema to production](/img/guides/dbt-ecosystem/dbt-python-snowpark/15-deployment/3-update-deployment-credentials-production.png?v=2 "Update the deployment credentials schema to production")](#)Update the deployment credentials schema to production 10. By updating the schema for our production environment to **production**, it ensures that our deployment job for this environment will build our dbt models in the **production** schema within the `PC_DBT_DB` database as defined in the Snowflake Connection section. 11. Now let's switch over to our production job. Click on the deploy tab again and then select **Jobs**. You should see an existing and preconfigured **Partner Connect Trial Job**. Similar to the environment, click on the job, then select **Settings** to modify it. Let's take a look at the job to understand it before making changes. * The Environment section is what connects this job with the environment we want it to run in. This job is already defaulted to use the Deployment environment that we just updated and the rest of the settings we can keep as is. * The Execution settings section gives us the option to generate docs, run source freshness, and defer to a previous run state. For the purposes of our lab, we're going to keep these settings as is as well and stick with just generating docs. * The Commands section is where we specify exactly which commands we want to run during this job, and we also want to keep this as is. We want our seed to be uploaded first, then run our models, and finally test them. The order of this is important as well, considering that we need our seed to be created before we can run our incremental model, and we need our models to be created before we can test them. * Finally, we have the Triggers section, where we have a number of different options for scheduling our job. Given that our data isn't updating regularly here and we're running this job manually for now, we're also going to leave this section alone. So, what are we changing then? Just the name! Click **Edit** to allow you to make changes. Then update the name of the job to **Production Job** to denote this as our production deployment job. After that's done, click **Save**. 12. Now let's go to run our job. Clicking on the job name in the path at the top of the screen will take you back to the job run history page where you'll be able to click **Run run** to kick off the job. If you encounter any job failures, try running the job again before further troubleshooting. [![Run production job](/img/guides/dbt-ecosystem/dbt-python-snowpark/15-deployment/4-run-production-job.png?v=2 "Run production job")](#)Run production job [![View production job details](/img/guides/dbt-ecosystem/dbt-python-snowpark/15-deployment/5-job-details.png?v=2 "View production job details")](#)View production job details 13. Let's go over to Snowflake to confirm that everything built as expected in our production schema. Refresh the database objects in your Snowflake account and you should see the production schema now within our default Partner Connect database. If you click into the schema and everything ran successfully, you should be able to see all of the models we developed. [![Check all our models in our pipeline are in Snowflake](/img/guides/dbt-ecosystem/dbt-python-snowpark/15-deployment/6-all-models-generated.png?v=2 "Check all our models in our pipeline are in Snowflake")](#)Check all our models in our pipeline are in Snowflake ##### Conclusion Fantastic! You’ve finished the workshop! We hope you feel empowered in using both SQL and Python in your dbt workflows with Snowflake. Having a reliable pipeline to surface both analytics and machine learning is crucial to creating tangible business value from your data. For more help and information join our [dbt community Slack](https://www.getdbt.com/community/) which contains more than 50,000 data practitioners today. We have a dedicated slack channel #db-snowflake to Snowflake related content. Happy dbt'ing! --- ### Migrate between dbt platform tenancy types [Back to guides](https://docs.getdbt.com/guides.md) Migration Tenancy dbt platform Advanced [Menu ]() Contract change required Migrating between tenancy types requires a contract change. Make sure you contact your account manager before starting this process to confirm eligibility, timeline, and initiate the necessary contract updates. *Do not* begin migration steps until a contract change is confirmed. #### Overview This guide walks you through a self-serve migration between [dbt platform tenancy types](https://docs.getdbt.com/docs/platform/about-platform/tenancy.md) — for example, moving from multi-tenant to single-tenant. It's meant for accounts that have been assessed by the dbt Labs team as low complexity. Your dbt Labs contact will provide a migration window before you begin — do not start until you've received it. Use the following table to confirm this guide is the right path for your account. If your account crosses the high-complexity threshold for even one signal, we recommend a Professional Services-led migration instead. | Signal | Low complexity (self-serve) | High complexity (PS-led) | | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | Active jobs (run in last 30 days) | Fewer than 10 | 10 or more | | Projects | Fewer than 3 | 3 or more | | Environments | Fewer than 5 | 5 or more | | **How to proceed** | If all signals are low complexity, follow this guide or contact [Professional Services migration](https://www.getdbt.com/services) for help | If one or more signals are high complexity, contact your account manager to scope a [Professional Services migration](https://www.getdbt.com/services) | The following factors add coordination overhead regardless of account size — flag these to your account manager if they apply: * CI or webhook-triggered jobs * SSO or SCIM provisioning * Privatelink endpoints * Multiple warehouse connection types ##### Related docs * [Tenancy](https://docs.getdbt.com/docs/platform/about-platform/tenancy.md) * [Access, regions, and IP addresses](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md) #### What doesn't migrate automatically The migration tool automates most configuration, but some things require manual setup after you apply the Terraform configuration. **Credentials and secrets** — these must be recreated manually: * SSH keys and repository credentials * Service account tokens * Environment variables containing secrets * Warehouse connections and environment-level credentials * dbt platform profiles **Project data** — not transferred: * Job run history, artifacts, and logs **User management** — requires manual verification: * Users must already exist in the target account * Permissions must be manually verified after migration * SSO, keypairs, and warehouse authentication credentials must be reconfigured **Unsupported resources:** * PrivateLink endpoints require assistance from the dbt Labs infrastructure team — [open a support ticket](https://www.getdbt.com/support) #### Prerequisites Before starting your migration, confirm the following: * A contract change has been initiated and confirmed with your account manager * You have received a migration window from the dbt Labs Support team * You have admin access to both the source and target dbt platform accounts * The target account is provisioned in the new environment * You have a service token with Account Admin permissions on the source account #### Migration steps This migration uses two open-source tools: * [dbt platform migration tool](https://github.com/dpguthrie/dbt-cloud-migration-tool) — exports your source account configuration and applies it to the target account via Terraform * [dbtcloud-terraforming](https://github.com/dbt-labs/dbtcloud-terraforming) — generates Terraform HCL from an existing dbt platform account info Credentials, secrets, and some other items don't migrate automatically — plan to recreate them manually. Refer to [What doesn't migrate automatically](#what-doesnt-migrate-automatically) for the full list. ##### Mac/Linux ##### Step 1: Install prerequisites You need Python, Terraform, and `dbtcloud-terraforming`. Install them using any method you prefer — we use Homebrew as an example here. Create and activate a Python virtual environment: ```shell python3 -m venv tf_acct_migrator_env source tf_acct_migrator_env/bin/activate ``` Install Terraform: ```shell brew tap hashicorp/tap && brew install hashicorp/tap/terraform ``` Install `dbtcloud-terraforming`: ```shell brew install dbt-labs/dbt-cli/dbtcloud-terraforming ``` ##### Step 2: Set up the migration tool Clone the migration tool repo and navigate into it: ```shell git clone https://github.com/dpguthrie/dbt-cloud-migration-tool cd dbt-cloud-migration-tool ``` Set environment variables for your source dbt platform account: ```shell export DBT_CLOUD_HOST_URL="https://YOUR_SOURCE_REGION.getdbt.com/api" export DBT_CLOUD_TOKEN="your_source_service_token" export DBT_CLOUD_ACCOUNT_ID="your_source_account_id" ``` ##### Step 3: Generate and apply Terraform configuration Run the migration tool to generate a `resources.tf` file with your source account's resources: ```shell sh migrate.sh ``` Copy the example vars file and add your target account's details: ```shell cp terraform.tfvars.example terraform.tfvars # Edit terraform.tfvars with your target account details ``` Apply the configuration to your target account: ```shell terraform init terraform plan terraform apply ``` Terraform will output the newly created resources and any warnings or errors. You may need to re-run `migrate.sh` multiple times to catch all required resource types. ##### Windows ##### Step 1: Install prerequisites You need Python, Terraform, and `dbtcloud-terraforming`. Install them using any method you prefer — we use Chocolatey as an example here. Install Python and Terraform: ```shell choco install python terraform -y ``` After installation, restart your terminal and verify: ```shell python --version terraform --version ``` Install `dbtcloud-terraforming` by adding it to your Terraform installation directory. If you used Chocolatey, the path is typically `C:\ProgramData\chocolatey\bin`. ##### Step 2: Set up the migration tool The following steps should be run in a bash environment (for example, Git Bash). Clone the migration tool repo and navigate into it: ```shell git clone https://github.com/dpguthrie/dbt-cloud-migration-tool.git cd dbt-cloud-migration-tool ``` ##### Step 3: Configure and run the migration Set environment variables for your source dbt platform account (these reset when you close the terminal): ```shell export DBT_CLOUD_HOST_URL="https://YOUR_SOURCE_REGION.getdbt.com/api" export DBT_CLOUD_TOKEN="your_source_service_token" export DBT_CLOUD_ACCOUNT_ID="your_source_account_id" ``` Run the migration script: ```shell sh migrate.sh ``` Copy the example vars file and add your target account's details: ```shell cp terraform.tfvars.example terraform.tfvars # Edit terraform.tfvars with your target account details ``` Apply the configuration to your target account: ```shell terraform init terraform plan terraform apply ``` Terraform will output the newly created resources and any warnings or errors. You may need to re-run `migrate.sh` multiple times to catch all required resource types. #### Post-migration checklist * \[ ] Verify all connections, permissions, and secrets in the target account * \[ ] Run test jobs to confirm end-to-end functionality * \[ ] Confirm IP allowlist and network settings are updated for the new tenant * \[ ] Update internal bookmarks, SSO configurations, and API integrations to the new account URL --- ### Migrate from dbt-spark to dbt-databricks [Back to guides](https://docs.getdbt.com/guides.md) Migration dbt platform Intermediate [Menu ]() #### Introduction You can migrate your projects from using the `dbt-spark` adapter to using the [dbt-databricks adapter](https://github.com/databricks/dbt-databricks). In collaboration with dbt Labs, Databricks built this adapter using dbt-spark as the foundation and added some critical improvements. With it, you get an easier set up — requiring only three inputs for authentication — and more features such as support for [Unity Catalog](https://www.databricks.com/product/unity-catalog). ##### Prerequisite * For dbt, you need administrative (admin) privileges to migrate dbt projects. ##### Simpler authentication Previously, you had to provide a `cluster` or `endpoint` ID which was hard to parse from the `http_path` that you were given. Now, it doesn't matter if you're using a cluster or an SQL endpoint because the [dbt-databricks setup](https://docs.getdbt.com/docs/local/connect-data-platform/databricks-setup.md) requires the *same* inputs for both. All you need to provide is: * hostname of the Databricks workspace * HTTP path of the Databricks SQL warehouse or cluster * appropriate credentials ##### Better defaults The `dbt-databricks` adapter provides better defaults than `dbt-spark` does. The defaults help optimize your workflow so you can get the fast performance and cost-effectiveness of Databricks. They are: * The dbt models use the [Delta](https://docs.databricks.com/delta/index.html) table format. You can remove any declared configurations of `file_format = 'delta'` since they're now redundant. * Accelerate your expensive queries with the [Photon engine](https://docs.databricks.com/runtime/photon.html). * The `incremental_strategy` config is set to `merge`. With dbt-spark, however, the default for `incremental_strategy` is `append`. If you want to continue using `incremental_strategy=append`, you must set this config specifically on your incremental models. If you already specified `incremental_strategy=merge` on your incremental models, you don't need to change anything when moving to dbt-databricks; but, you can keep your models clean (tidy) by removing the config since it's redundant. Read [About incremental\_strategy](https://docs.getdbt.com/docs/build/incremental-strategy.md) to learn more. For more information on defaults, see [Caveats](https://docs.getdbt.com/docs/local/connect-data-platform/databricks-setup.md#caveats). ##### Pure Python If you use dbt v1, you no longer have to download an independent driver to interact with Databricks. The connection information is all embedded in a pure-Python library called `databricks-sql-connector`. #### Migrate your dbt projects in dbt You can migrate your projects to the Databricks-specific adapter from the generic Apache Spark adapter. If you're using dbt v1, then skip to Step 4. The migration to the `dbt-databricks` adapter from `dbt-spark` shouldn't cause any downtime for production jobs. dbt Labs recommends that you schedule the connection change when usage of the IDE is light to avoid disrupting your team. To update your Databricks connection in dbt: 1. Select **Account Settings** in the main navigation bar. 2. On the **Projects** tab, find the project you want to migrate to the dbt-databricks adapter. 3. Click the hyperlinked Connection for the project. 4. Click **Edit** in the top right corner. 5. Select **Databricks** for the warehouse 6. Enter the: 1. `hostname` 2. `http_path` 3. (optional) catalog name 7. Click **Save**. Everyone in your organization who uses dbt must refresh the Studio IDE before starting work again. It should refresh in less than a minute. #### Configure your credentials When you update the Databricks connection in dbt, your team will not lose their credentials. This makes migrating easier since it only requires you to delete the Databricks connection and re-add the cluster or endpoint information. These credentials will not get lost when there's a successful connection to Databricks using the `dbt-spark` ODBC method: * The credentials you supplied to dbt to connect to your Databricks workspace. * The personal access tokens your team added in their dbt profile so they can develop in the Studio IDE for a given project. * The access token you added for each deployment environment so dbt can connect to Databricks during production jobs. #### Migrate dbt projects in dbt To migrate your dbt v1 projects to the `dbt-databricks` adapter from `dbt-spark`, you: 1. Install the [dbt-databricks adapter](https://github.com/databricks/dbt-databricks) in your environment 2. Update your Databricks connection by modifying your `target` in your `~/.dbt/profiles.yml` file Anyone who's using your project must also make these changes in their environment. #### Try these examples You can use the following examples of the `profiles.yml` file to see the authentication setup with `dbt-spark` compared to the simpler setup with `dbt-databricks` when connecting to an SQL endpoint. A cluster example would look similar. An example of what authentication looks like with `dbt-spark`: \~/.dbt/profiles.yml ```yaml your_profile_name: target: dev outputs: dev: type: spark method: odbc driver: '/opt/simba/spark/lib/64/libsparkodbc_sb64.so' schema: my_schema host: dbc-l33t-nwb.cloud.databricks.com endpoint: 8657cad335ae63e3 token: [my_secret_token] ``` An example of how much simpler authentication is with `dbt-databricks`: \~/.dbt/profiles.yml ```yaml your_profile_name: target: dev outputs: dev: type: databricks schema: my_schema host: dbc-l33t-nwb.cloud.databricks.com http_path: /sql/1.0/endpoints/8657cad335ae63e3 token: [my_secret_token] ``` --- ### Migrate from DDL, DML, and stored procedures [Back to guides](https://docs.getdbt.com/guides.md) Migration Beginner [Menu ]() #### Introduction One of the more common situations that new dbt adopters encounter is a historical codebase of transformations written as a hodgepodge of DDL and DML statements, or stored procedures. Going from DML statements to dbt models is often a challenging hump for new users to get over, because the process involves a significant paradigm shift between a procedural flow of building a dataset (e.g. a series of DDL and DML statements) to a declarative approach to defining a dataset (e.g. how dbt uses SELECT statements to express data models). This guide aims to provide tips, tricks, and common patterns for converting DML statements to dbt models. ##### Preparing to migrate Before getting into the meat of conversion, it’s worth noting that DML statements will not always illustrate a comprehensive set of columns and column types that an original table might contain. Without knowing the DDL to create the table, it’s impossible to know precisely if your conversion effort is apples-to-apples, but you can generally get close. If your data warehouse supports `SHOW CREATE TABLE`, that can be a quick way to get a comprehensive set of columns you’ll want to recreate. If you don’t have the DDL, but are working on a substantial stored procedure, one approach that can work is to pull column lists out of any DML statements that modify the table, and build up a full set of the columns that appear. As for ensuring that you have the right column types, since models materialized by dbt generally use `CREATE TABLE AS SELECT` or `CREATE VIEW AS SELECT` as the driver for object creation, tables can end up with unintended column types if the queries aren’t explicit. For example, if you care about `INT` versus `DECIMAL` versus `NUMERIC`, it’s generally going to be best to be explicit. The good news is that this is easy with dbt: you just cast the column to the type you intend. We also generally recommend that column renaming and type casting happen as close to the source tables as possible, typically in a layer of staging transformations, which helps ensure that future dbt modelers will know where to look for those transformations! See [How we structure our dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md) for more guidance on overall project structure. ##### Operations we need to map There are four primary DML statements that you are likely to have to convert to dbt operations while migrating a procedure: * `INSERT` * `UPDATE` * `DELETE` * `MERGE` Each of these can be addressed using various techniques in dbt. Handling `MERGE`s is a bit more involved than the rest, but can be handled effectively via dbt. The first three, however, are fairly simple to convert. #### Map INSERTs An `INSERT` statement is functionally the same as using dbt to `SELECT` from an existing source or other dbt model. If you are faced with an `INSERT`-`SELECT` statement, the easiest way to convert the statement is to just create a new dbt model, and pull the `SELECT` portion of the `INSERT` statement out of the procedure and into the model. That’s basically it! To really break it down, let’s consider a simple example: ```sql INSERT INTO returned_orders (order_id, order_date, total_return) SELECT order_id, order_date, total FROM orders WHERE type = 'return' ``` Converting this with a first pass to a [dbt model](https://docs.getdbt.com/guides/bigquery.md?step=8) (in a file called returned\_orders.sql) might look something like: ```sql SELECT order_id as order_id, order_date as order_date, total as total_return FROM {{ ref('orders') }} WHERE type = 'return' ``` Functionally, this would create a model (which could be materialized as a table or view depending on needs) called `returned_orders` that contains three columns: `order_id`, `order_date`, `total_return`) predicated on the type column. It achieves the same end as the `INSERT`, just in a declarative fashion, using dbt. ##### **A note on `FROM` clauses** In dbt, using a hard-coded table or view name in a `FROM` clause is one of the most serious mistakes new users make. dbt uses the ref and source macros to discover the ordering that transformations need to execute in, and if you don’t use them, you’ll be unable to benefit from dbt’s built-in lineage generation and pipeline execution. In the sample code throughout the remainder of this article, we’ll use ref statements in the dbt-converted versions of SQL statements, but it is an exercise for the reader to ensure that those models exist in their dbt projects. ##### **Sequential `INSERT`s to an existing table can be `UNION ALL`’ed together** Since dbt models effectively perform a single `CREATE TABLE AS SELECT` (or if you break it down into steps, `CREATE`, then an `INSERT`), you may run into complexities if there are multiple `INSERT` statements in your transformation that all insert data into the same table. Fortunately, this is a simple thing to handle in dbt. Effectively, the logic is performing a `UNION ALL` between the `INSERT` queries. If I have a transformation flow that looks something like (ignore the contrived nature of the scenario): ```sql CREATE TABLE all_customers INSERT INTO all_customers SELECT * FROM us_customers INSERT INTO all_customers SELECT * FROM eu_customers ``` The dbt-ified version of this would end up looking something like: ```sql SELECT * FROM {{ ref('us_customers') }} UNION ALL SELECT * FROM {{ ref('eu_customers') }} ``` The logic is functionally equivalent. So if there’s another statement that `INSERT`s into a model that I’ve already created, I can just add that logic into a second `SELECT` statement that is just `UNION ALL`'ed with the first. Easy! #### Map UPDATEs `UPDATE`s start to increase the complexity of your transformations, but fortunately, they’re pretty darn simple to migrate, as well. The thought process that you go through when translating an `UPDATE` is quite similar to how an `INSERT` works, but the logic for the `SELECT` list in the dbt model is primarily sourced from the content in the `SET` section of the `UPDATE` statement. Let’s look at a simple example: ```sql UPDATE orders SET type = 'return' WHERE total < 0 ``` The way to look at this is similar to an `INSERT`-`SELECT` statement. The table being updated is the model you want to modify, and since this is an `UPDATE`, that model has likely already been created, and you can either: * add to it with subsequent transformations * create an intermediate model that builds off of the original model – perhaps naming it something like `int_[entity]_[verb].sql`. The `SELECT` list should contain all of the columns for the table, but for the specific columns being updated by the DML, you’ll use the computation on the right side of the equals sign as the `SELECT`ed value. Then, you can use the target column name on the left of the equals sign as the column alias. If I were building an intermediate transformation from the above query would translate to something along the lines of: ```sql SELECT CASE WHEN total < 0 THEN 'return' ELSE type END AS type, order_id, order_date FROM {{ ref('stg_orders') }} ``` Since the `UPDATE` statement doesn’t modify every value of the type column, we use a `CASE` statement to apply the contents’ `WHERE` clause. We still want to select all of the columns that should end up in the target table. If we left one of the columns out, it wouldn’t be passed through to the target table at all due to dbt’s declarative approach. Sometimes, you may not be sure what all the columns are in a table, or in the situation as above, you’re only modifying a small number of columns relative to the total number of columns in the table. It can be cumbersome to list out every column in the table, but fortunately dbt contains some useful utility macros that can help list out the full column list of a table. Another way I could have written the model a bit more dynamically might be: ```sql SELECT {{ dbt_utils.star(from=ref('stg_orders'), except=['type']) }}, CASE WHEN total < 0 THEN 'return' ELSE type END AS type, FROM {{ ref('stg_orders') }} ``` The `dbt_utils.star()` macro will print out the full list of columns in the table, but skip the ones I’ve listed in the except list, which allows me to perform the same logic while writing fewer lines of code. This is a simple example of using dbt macros to simplify and shorten your code, and dbt can get a lot more sophisticated as you learn more techniques. Read more about the [dbt\_utils package](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/) and the [star macro](https://github.com/dbt-labs/dbt-utils/tree/0.8.6/#star-source). #### Map DELETEs One of the biggest differences between a procedural transformation and how dbt models data is that dbt, in general, will never destroy data. While there are ways to execute hard `DELETE`s in dbt that are outside of the scope of this article, the general best practice for handling deleted data is to just use soft deletes, and filter out soft-deleted data in a final transformation. Let’s consider a simple example query: ```sql DELETE FROM stg_orders WHERE order_status IS NULL ``` In a dbt model, you’ll need to first identify the records that should be deleted and then filter them out. There are really two primary ways you might translate this query: ```sql SELECT * FROM {{ ref('stg_orders') }} WHERE order_status IS NOT NULL ``` This first approach just inverts the logic of the DELETE to describe the set of records that should remain, instead of the set of records that should be removed. This ties back to the way dbt declaratively describes datasets. You reference the data that should be in a dataset, and the table or view gets created with that set of data. Another way you could achieve this is by marking the deleted records, and then filtering them out. For example: ```sql WITH soft_deletes AS ( SELECT *, CASE WHEN order_status IS NULL THEN true ELSE false END AS to_delete FROM {{ ref('stg_orders') }} ) SELECT * FROM soft_deletes WHERE to_delete = false ``` This approach flags all of the deleted records, and the final `SELECT` filters out any deleted data, so the resulting table contains only the remaining records. It’s a lot more verbose than just inverting the `DELETE` logic, but for complex `DELETE` logic, this ends up being a very effective way of performing the `DELETE` that retains historical context. It’s worth calling out that while this doesn’t enable a hard delete, hard deletes can be executed a number of ways, the most common being to execute a dbt [macros](https://docs.getdbt.com/docs/build/jinja-macros.md) via as a [run-operation](https://docs.getdbt.com/reference/commands/run-operation.md), or by using a [post-hook](https://docs.getdbt.com/reference/resource-configs/pre-hook-post-hook.md) to perform a `DELETE` statement after the records to-be-deleted have been marked. These are advanced approaches outside the scope of this guide. #### Map MERGEs dbt has a concept called [materialization](https://docs.getdbt.com/docs/build/materializations.md), which determines how a model is physically or logically represented in the warehouse. `INSERT`s, `UPDATE`s, and `DELETE`s will typically be accomplished using table or view materializations. For incremental workloads accomplished via commands like `MERGE` or `UPSERT`, dbt has a particular materialization called [incremental](https://docs.getdbt.com/docs/build/incremental-models.md). The incremental materialization is specifically used to handle incremental loads and updates to a table without recreating the entire table from scratch on every run. ##### Step 1: Map the MERGE like an INSERT/UPDATE to start Before we get into the exact details of how to implement an incremental materialization, let’s talk about logic conversion. Extracting the logic of the `MERGE` and handling it as you would an `INSERT` or an `UPDATE` is the easiest way to get started migrating a `MERGE` command. . To see how the logic conversion works, we’ll start with an example `MERGE`. In this scenario, imagine a ride sharing app where rides are loaded into a details table daily, and tips may be updated at some later date, and need to be kept up-to-date: ```sql MERGE INTO ride_details USING ( SELECT ride_id, subtotal, tip FROM rides_to_load AS rtl ON ride_details.ride_id = rtl.ride_id WHEN MATCHED THEN UPDATE SET ride_details.tip = rtl.tip WHEN NOT MATCHED THEN INSERT (ride_id, subtotal, tip) VALUES (rtl.ride_id, rtl.subtotal, NVL(rtl.tip, 0, rtl.tip) ); ``` The content of the `USING` clause is a useful piece of code because that can easily be placed in a CTE as a starting point for handling the match statement. I find that the easiest way to break this apart is to treat each match statement as a separate CTE that builds on the previous match statements. We can ignore the `ON` clause for now, as that will only come into play once we get to a point where we’re ready to turn this into an incremental. As with `UPDATE`s and `INSERT`s, you can use the `SELECT` list and aliases to name columns appropriately for the target table, and `UNION` together `INSERT` statements (taking care to use `UNION`, rather than `UNION ALL` to avoid duplicates). The `MERGE` would end up translating to something like this: ```sql WITH using_clause AS ( SELECT ride_id, subtotal, tip FROM {{ ref('rides_to_load') }} ), updates AS ( SELECT ride_id, subtotal, tip FROM using_clause ), inserts AS ( SELECT ride_id, subtotal, NVL(tip, 0, tip) FROM using_clause ) SELECT * FROM updates UNION inserts ``` To be clear, this transformation isn’t complete. The logic here is similar to the `MERGE`, but will not actually do the same thing, since the updates and inserts CTEs are both selecting from the same source query. We’ll need to ensure we grab the separate sets of data as we transition to the incremental materialization. One important caveat is that dbt does not natively support `DELETE` as a `MATCH` action. If you have a line in your `MERGE` statement that uses `WHEN MATCHED THEN DELETE`, you’ll want to treat it like an update and add a soft-delete flag, which is then filtered out in a follow-on transformation. ##### Step 2: Convert to incremental materialization As mentioned above, incremental materializations are a little special in that when the target table does not exist, the materialization functions in nearly the same way as a standard table materialization, and executes a `CREATE TABLE AS SELECT` statement. If the target table does exist, however, the materialization instead executes a `MERGE` statement. Since a `MERGE` requires a `JOIN` condition between the `USING` clause and the target table, we need a way to specify how dbt determines whether or not a record triggers a match or not. That particular piece of information is specified in the dbt model configuration. We can add the following `config()` block to the top of our model to specify how it should build incrementally: ```sql {{ config( materialized='incremental', unique_key='ride_id', incremental_strategy='merge' ) }} ``` The three configuration fields in this example are the most important ones. * Setting `materialized='incremental'` tells dbt to apply UPSERT logic to the target table. * The `unique_key` should be a primary key of the target table. This is used to match records with the existing table. * `incremental_strategy` here is set to MERGE any existing rows in the target table with a value for the `unique_key` which matches the incoming batch of data. There are [various incremental strategies](https://docs.getdbt.com/docs/build/incremental-strategy.md) for different situations and warehouses. The bulk of the work in converting a model to an incremental materialization comes in determining how the logic should change for incremental loads versus full backfills or initial loads. dbt offers a special macro, `is_incremental()`, which evaluates false for initial loads or for backfills (called full refreshes in dbt parlance), but true for incremental loads. This macro can be used to augment the model code to adjust how data is loaded for subsequent loads. How that logic should be added will depend a little bit on how data is received. Some common ways might be: 1. The source table is truncated ahead of incremental loads, and only contains the data to be loaded in that increment. 2. The source table contains all historical data, and there is a load timestamp column that identifies new data to be loaded. In the first case, the work is essentially done already. Since the source table always contains only the new data to be loaded, the query doesn’t have to change for incremental loads. The second case, however, requires the use of the `is_incremental()` macro to correctly handle the logic. Taking the converted `MERGE` statement that we’d put together previously, we’d augment it to add this additional logic: ```sql WITH using_clause AS ( SELECT ride_id, subtotal, tip, max(load_timestamp) as load_timestamp FROM {{ ref('rides_to_load') }} {% if is_incremental() %} WHERE load_timestamp > (SELECT max(load_timestamp) FROM {{ this }}) {% endif %} ), updates AS ( SELECT ride_id, subtotal, tip, load_timestamp FROM using_clause {% if is_incremental() %} WHERE ride_id IN (SELECT ride_id FROM {{ this }}) {% endif %} ), inserts AS ( SELECT ride_id, subtotal, NVL(tip, 0, tip), load_timestamp FROM using_clause WHERE ride_id NOT IN (SELECT ride_id FROM updates) ) SELECT * FROM updates UNION inserts ``` There are a couple important concepts to understand here: 1. The code in the `is_incremental()` conditional block only executes for incremental executions of this model code. If the target table doesn’t exist, or if the `--full-refresh` option is used, that code will not execute. 2. `{{ this }}` is a special keyword in dbt that when used in a Jinja block, self-refers to the model for which the code is executing. So if you have a model in a file called `my_incremental_model.sql`, `{{ this }}` will refer to `my_incremental_model` (fully qualified with database and schema name if necessary). By using that keyword, we can leverage the current state of the target table to inform the source query. #### Migrate Stores procedures The techniques shared above are useful ways to get started converting the individual DML statements that are often found in stored procedures. Using these types of patterns, legacy procedural code can be rapidly transitioned to dbt models that are much more readable, maintainable, and benefit from software engineering best practices like DRY principles. Additionally, once transformations are rewritten as dbt models, it becomes much easier to test the transformations to ensure that the data being used downstream is high-quality and trustworthy. --- ### Migrate off legacy dbt versions [Back to guides](https://docs.getdbt.com/guides.md) Migration dbt v1 dbt platform Upgrade Beginner [Menu ]() Legacy dbt versions v1.3–v1.7 are being deprecated on January 31, 2027. To keep your work running and supported, move your environments and jobs to a [dbt v1 release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) now. A release track keeps you on a supported version automatically and prepares your project for [dbt v2](https://docs.getdbt.com/docs/introduction.md) later. The single-hop path Move to a dbt v1 release track now; move to dbt v2 later. Migrating first to a supported release track lowers your migration risk and gets you on a supported version faster. Your project code, connections, and history stay accessible throughout. #### Identify projects using legacy versions What you migrate is driven by a job's effective version: the version pinned on the job if one is set, otherwise the version its environment inherits. 1. Navigate to **Orchestration** > **Environments** and note any environment whose **dbt version** is v1.3–v1.7. The current version is displayed below the environment name. 2. Open the jobs in your supported environments and check for any job with a **version override** pinned to v1.3–v1.7. #### Select your path Find the row that matches each environment, then follow the linked steps. | Environment version | Job version | What you do | | ------------------------------------------ | -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | | Legacy (v1.3–v1.7) | Inherits from environment, or pinned to a legacy version | [Migrate the environment to a release track](#migrate-an-environment-to-a-release-track) | | Legacy (v1.3–v1.7) | At least one job pinned to a supported version | [Migrate the environment to a release track](#migrate-an-environment-to-a-release-track); the supported jobs are retained | | Supported | One or more jobs pinned to a legacy version | [Update the job's version](#update-a-jobs-version); the environment is untouched | | Legacy and **dormant** (unused 12+ months) | — | [Delete the environment](#delete-a-dormant-environment), or migrate it if you still need it | If you don't migrate a legacy environment or clear a legacy job, it's cleaned up when legacy versions are retired: a legacy environment (and its jobs) is deleted, a legacy environment that already has a job on a supported version is **updated** with only the legacy jobs removed, and a supported environment with legacy-pinned jobs keeps the environment and loses **only those jobs**. #### Migrate an environment to a release track To update your environment to a release track: 1. Navigate to the Settings page of the environment, then click **Edit**. 2. Click the **dbt version** dropdown and select a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) (**v1 Latest** is recommended). 3. Save your changes. As a best practice, test the upgrade in a development environment first. See [Upgrade versions in dbt platform](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) for details, including how to set the version through the [Admin API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md) or Terraform. #### Update a job's version If a job on a supported environment is pinned to a legacy version, clear or change the override: 1. Open the job's settings and find the **dbt version** setting. 2. Either clear the override so the job inherits the environment's version, or set it to a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md). 3. Save your changes. #### Delete a dormant environment If a legacy environment has been dormant for a year or more, and you no longer need it, delete it. (If you still need it, [migrate it to a release track](#migrate-an-environment-to-a-release-track) instead.) Deleting an environment automatically deletes its associated job(s). If you want to keep those jobs, move them to a different environment first. Follow these steps to delete an environment in dbt: 1. Navigate to **Orchestration** > **Environments**. 2. Select the environment you want to delete. 3. Click **Settings** on the top right of the page and then click **Edit**. 4. Scroll to the bottom of the page and click **Delete** to delete the environment. [![Delete an environment](/img/docs/dbt-platform/platform-configuring-dbt-platform/delete-environment.png?v=2 "Delete an environment")](#)Delete an environment 5. Confirm your action in the pop-up by clicking **Confirm delete** in the bottom right to delete the environment immediately. This action cannot be undone. However, you can create a new environment with the same information if the deletion was made in error. 6. Refresh your page and the deleted environment should now be gone. To delete multiple environments, you'll need to perform these steps to delete each one. If you're having any issues, feel free to [contact us](mailto:support@getdbt.com) for additional help. #### Validate your migration Moving from a pinned legacy version to a release track can surface behavior changes, dependency issues, or adapter differences that weren't visible before. To reduce risk: * Test in a development environment before upgrading your production and default development environments. * Review your compile, build, and job run results after the change. * Establish a fallback path in case you need to roll back. #### Get help If you hit a blocker you can't resolve, [contact Support](mailto:support@getdbt.com) with your project ID, environment ID, affected job run IDs and logs, your current version, and your target release track. --- ### Move to the dbt platform: Get started [Back to guides](https://docs.getdbt.com/guides.md) Total estimated time: 3-4 hours Migration dbt platform Intermediate [Menu ]() #### Introduction Moving from self-hosted dbt to the dbt platform streamlines analytics engineering workflows by allowing teams to develop, test, deploy, and explore data products using a single, fully managed software service. The data layer is the foundation for trusted analytics and AI; the dbt platform gives you the governance, shared definitions, and reliability to scale both — without the hidden cost of self-hosting in engineer hours and wasted compute. Explore our 3-part-guide series on moving from self-hosted dbt to the dbt platform. This series is ideal for users aiming for streamlined workflows and enhanced analytics: | Guide | Information | Audience | | ---------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | ------------------------------------------------- | | [Move to the dbt platform: What you need to know](https://docs.getdbt.com/guides/dbt-migration-2.md) | Understand the considerations and methods needed in your move to the dbt platform. | Team leads<br />Admins | | [Move to the dbt platform: Get started](https://docs.getdbt.com/guides/dbt-migration-1.md?step=1) | Learn the steps needed to move to the dbt platform. | Developers<br />Data engineers<br />Data analysts | | [Move to the dbt platform: Optimization tips](https://docs.getdbt.com/guides/dbt-migration-3.md) | Learn how to optimize your dbt platform experience with common scenarios and useful tips. | Everyone | ##### Why move to the dbt platform? If your team is using self-hosted dbt today, you could be reading this guide because: * You've realized the burden of maintaining that deployment. * The person who set it up has since left. * You're interested in what the dbt platform could do to better manage the complexity of your dbt deployment, democratize access to more contributors, or improve security and governance practices. * You need a governed data foundation for AI—shared definitions, lineage, and testing so analytics and AI give answers the business can trust. Self-hosting hides its true cost in engineer hours and wasted compute. dbt platform eliminates that overhead with managed infrastructure and browser-based development so more people can contribute without you being the bottleneck. State-aware orchestration is now dbt State [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) works with all engines and environments: dbt v1, the dbt platform, and dbt v2. If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). The data layer is the AI layer—make sure it's tested, defined, and trusted end to end. Moving to the dbt platform simplifies workflows by providing a fully managed environment that improves collaboration, security, and orchestration. With the dbt platform, you gain access to features like cross-team collaboration ([dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md)), version management, streamlined CI/CD, [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) for comprehensive insights, and more — making it easier to manage complex dbt deployments and scale your data workflows efficiently. It's ideal for teams looking to reduce the burden of maintaining their own infrastructure while enhancing governance and productivity.  What is dbt? * The dbt platform is the fastest and most reliable way to deploy dbt. It enables you to develop, test, deploy, and explore data products using a single, fully managed service. Infrastructure is managed for you — no custom scripts or fragile orchestration. State-aware orchestration only builds what's changed, so you waste less compute and time. Browser-based development and dbt Wizard open up development to analysts, so you're no longer the bottleneck for every change. With end-to-end lineage, shared metric definitions, and CI that catches regressions before production, you spend less time debugging and more time building. The dbt platform also supports: * Development experiences tailored to multiple personas ([Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) or [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md)) * Out-of-the-box [CI/CD workflows](https://docs.getdbt.com/docs/deploy/ci-jobs.md) * The [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) for consistent metrics * Domain ownership of data with multi-project [dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md) setups * [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) for easier data discovery and understanding Learn more about [dbt platform features](https://docs.getdbt.com/docs/platform/about-platform/dbt-platform-features.md). * dbt is an open-source tool that enables data teams to define and execute data transformations in a cloud data warehouse following analytics engineering best practices. While this can work well for 'single players' and small technical teams, all development happens on a command-line interface (CLI), and production deployments must be self-hosted and maintained. You absorb the cost of every upgrade, every broken CI run, and every request that pulls you away from real work: maintaining infrastructure, debugging the CI pipeline, and fielding every change that requires CLI access. Compute runs unchecked, upgrades are risky, and there's no easy way to trace what broke or why. This requires significant, costly work that adds up over time to maintain and scale — and without governance, shared definitions, or reliable testing. #### What you'll learn This guide outlines the steps you need to take to move from self-hosted dbt to the dbt platform and highlights the necessary technical changes: * [Account setup](https://docs.getdbt.com/guides/dbt-migration-1.md?step=4): Learn how to create a dbt platform account, invite team members, and configure it for your team. * [Data platform setup](https://docs.getdbt.com/guides/dbt-migration-1.md?step=5): Find out about connecting your data platform to the dbt platform. * [Git setup](https://docs.getdbt.com/guides/dbt-migration-1.md?step=6): Learn to link your dbt project's Git repository with the dbt platform. * [Developer setup:](https://docs.getdbt.com/guides/dbt-migration-1.md?step=7) Understand the setup needed for developing in the dbt platform. * [Environment variables](https://docs.getdbt.com/guides/dbt-migration-1.md?step=8): Discover how to manage environment variables in the dbt platform, including their priority. * [Orchestration setup](https://docs.getdbt.com/guides/dbt-migration-1.md?step=9): Learn how to prepare your dbt platform environment and jobs for orchestration. * [Models configuration](https://docs.getdbt.com/guides/dbt-migration-1.md?step=10): Get insights on validating and running your models in the dbt platform, using either the Studio IDE or dbt platform CLI. * [What's next?](https://docs.getdbt.com/guides/dbt-migration-1.md?step=11): Summarizes key takeaways and introduces what to expect in the following guides. ##### Related docs * [Learn dbt](https://learn.getdbt.com) on-demand video learning. * Book [expert-led demos](https://www.getdbt.com/resources/dbt-cloud-demos-with-experts) and insights * Work with the [dbt Labs' Professional Services](https://www.getdbt.com/dbt-labs/services) team to support your data organization and migration. #### Prerequisites * You have an existing dbt project connected to a Git repository and data platform supported in the [dbt platform](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md). * You have a dbt platform account. **[Don't have one? Start your free trial today](https://www.getdbt.com/signup)**! #### Account setup This section outlines the steps to set up your dbt platform account and configure it for your team. 1. [Create your dbt platform account](https://www.getdbt.com/signup). 2. Provide user [access](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md) and [invite users](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md) to your dbt platform account and project. 3. Configure [Single Sign-On (SSO)](https://docs.getdbt.com/docs/platform/manage-access/sso-overview.md) or [Role-based access control (RBAC)](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md#role-based-access-control) for easy and secure access. [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") * This removes the need to save passwords and secret environment variables locally. ##### Additional configuration Explore these additional configurations for performance and reliability improvements: 1. In **Account settings**, enable [partial parsing](https://docs.getdbt.com/docs/platform/account-settings.md#partial-parsing) to only reparse changed files, saving time. 2. In **Account settings**, enable [Git repo caching](https://docs.getdbt.com/docs/platform/account-settings.md#git-repository-caching) for job reliability & third-party outage protection. [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") #### Data platform setup This section outlines the considerations and methods to connect your data platform to the dbt platform. 1. In the dbt platform, set up your [data platform connections](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md) and [environment variables](https://docs.getdbt.com/docs/build/environment-variables.md). The dbt platform can connect with a variety of data platform providers including: * [AlloyDB](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-postgresql-alloydb.md) * [Amazon Athena](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-amazon-athena.md) * [Amazon Redshift](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-redshift.md) * [Apache Spark](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-apache-spark.md) * [Azure Synapse Analytics](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-azure-synapse-analytics.md) * [Databricks](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-databricks.md) * [Google BigQuery](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-bigquery.md) * [Microsoft Fabric](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-microsoft-fabric.md) * [PostgreSQL](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-postgresql-alloydb.md) * [Snowflake](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-snowflake.md) * [Starburst or Trino](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-starburst-trino.md) * [Teradata](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-teradata.md) 2. You can verify your data platform connections by clicking the **Test connection** button in your deployment and user credentials settings. ##### Additional configuration Explore these additional configurations to optimize your data platform setup further: 1. Use [OAuth connections](https://docs.getdbt.com/docs/platform/manage-access/set-up-snowflake-oauth.md), which enables secure authentication using your data platform's SSO. [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") #### Git setup Your existing dbt project source code should live in a Git repository. In this section, you will connect your existing dbt project source code from Git to the dbt platform. 1. Ensure your dbt project is in a Git repository. 2. In **Account settings**, select **Integrations** to [connect your Git repository](https://docs.getdbt.com/docs/platform/git/configure-git.md) to the dbt platform: * (**Recommended**) Connect with one of the [native integrations](https://docs.getdbt.com/docs/platform/git/configure-git.md) in the dbt platform (such as GitHub, GitLab, and Azure DevOps). This method is preferred for its simplicity, security features (including secure OAuth logins and automated workflows like CI builds on pull requests), and overall ease of use. * [Import a Git repository](https://docs.getdbt.com/docs/platform/git/import-a-project-by-git-url.md) from any valid Git URL that points to a dbt project. #### Developer setup This section highlights the development configurations you'll need for your dbt project. The following categories are covered in this section: * [dbt platform environments](https://docs.getdbt.com/guides/dbt-migration-1.md?step=7#dbt-cloud-environments) * [Initial setup steps](https://docs.getdbt.com/guides/dbt-migration-1.md?step=7#initial-setup-steps) * [Additional configuration](https://docs.getdbt.com/guides/dbt-migration-1.md?step=7#additional-configuration-2) * [dbt platform commands](https://docs.getdbt.com/guides/dbt-migration-1.md?step=7#dbt-cloud-commands) ##### dbt environments The most common data environments are production, staging, and development. The way self-hosted dbt manages [environments](https://docs.getdbt.com/docs/environments-in-dbt.md) is through `target`, which are different sets of connection details. [dbt platform environments](https://docs.getdbt.com/docs/dbt-platform-environments.md) go further by: * Integrating with features such as job scheduling or version control, making it easier to manage the full lifecycle of your dbt projects within a single platform. * Streamlining the process of switching between development, staging, and production contexts. * Making it easy to configure environments through the dbt platform UI instead of manually editing the `profiles.yml` file. You can also [set up](https://docs.getdbt.com/reference/dbt-jinja-functions/target.md) or [customize](https://docs.getdbt.com/docs/build/custom-target-names.md) target names in the dbt platform. * Adding `profiles.yml` attributes to dbt platform environment settings with [Extended Attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes). * Using [Git repo caching](https://docs.getdbt.com/docs/platform/account-settings.md#git-repository-caching) to protect you from third-party outages, Git auth failures, and more. [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") ##### Initial setup steps 1. **Set up development environment** — Set up your [development](https://docs.getdbt.com/docs/dbt-platform-environments.md#create-a-development-environment) environment and [user credentials](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md#get-started-with-the-studio-ide). You'll need this to access your dbt project and start developing. 2. **dbt version** — In your dbt platform environment, select a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) for ongoing dbt version upgrades. If your team plans to use both self-hosted dbt and the dbt platform for developing or deploying your dbt project, you can run `dbt --version` in the command line to find out which version of dbt you're using. * When using self-hosted dbt, you need to think about which version you're using and manage your own upgrades. When using the dbt platform, leverage [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) so you don't have to. 3. **Connect to your data platform** — When using the dbt platform, you can [connect to your data platform](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md) directly in the UI. * Each environment is roughly equivalent to an entry in your `profiles.yml` file. This means you don't need a `profiles.yml` file in your project. 4. **Development tools** — Set up your development workspace with the [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) (command line interface or code editor) or [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) (browser-based) to build, test, run, and version control your dbt code in your tool of choice. * If you've previously installed dbt, the [dbt platform CLI installation doc](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md?install=pip#install-dbt-cloud-cli) has more information on how to install the dbt platform CLI, create aliases, or uninstall dbt for a smooth transition. ##### Additional configuration Explore these additional configurations to optimize your developer setup further: 1. **Custom target names** — Using [`custom target.names`](https://docs.getdbt.com/docs/build/custom-target-names.md) in your dbt projects helps identify different environments (like development, staging, and production). While you can specify the `custom target.name` values in your development credentials or orchestration setup, we recommend using [environment variables](https://docs.getdbt.com/docs/build/environment-variables.md) as the preferred method. They offer a clearer way to handle different environments and are better supported by dbt's partial parsing feature, unlike using [`{{ target }}` logic](https://docs.getdbt.com/reference/dbt-jinja-functions/target.md) which is meant for defining the data warehouse connection. ##### dbt commands 1. Review the [dbt commands](https://docs.getdbt.com/reference/dbt-commands.md) supported for dbt platform development. For example, `dbt init` isn't needed in the dbt platform as you can create a new project directly in the dbt platform. #### Environment variables This section will help you understand how to set up and manage dbt platform environment variables for your project. The following categories are covered: * [Environment variables in the dbt platform](https://docs.getdbt.com/guides/dbt-migration-1.md?step=7#environment-variables-in-dbt-cloud) * [dbt platform environment variables order of precedence](https://docs.getdbt.com/guides/dbt-migration-1.md?step=7#dbt-cloud-environment-variables-order-of-precedence) * [Set environment variables in the dbt platform](https://docs.getdbt.com/guides/dbt-migration-1.md?step=7#set-environment-variables-in-dbt-cloud) In the dbt platform, you can set [environment variables](https://docs.getdbt.com/docs/build/environment-variables.md) in the dbt platform user interface (UI). Read [Set up environment variables](#set-environment-variables-in-dbt-cloud) for more info. In self-hosted dbt, environment variables, or the [`env_var` function](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var.md), are defined manually by the developer or within the external application running dbt. ##### Environment variables in dbt * dbt platform environment variables must be prefixed with `DBT_` (including `DBT_ENV_CUSTOM_ENV_` or `DBT_ENV_SECRET`). * If your self-hosted dbt environment variables don't follow this naming convention, perform a ["find and replace"](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md#studio-ide-features) in your project to make sure all references to these environment variables contain the proper naming conventions. * The dbt platform secures environment variables that enable more flexible configuration of data warehouse connections or git provider integrations, offering additional measures for sensitive values, such as prefixing keys with `DBT_ENV_SECRET`to obscure them in logs and the UI. [![Setting project level and environment level values](</img/docs/dbt-platform/using-dbt-platform/Environment Variables/project-environment-view.png?v=2> "Setting project level and environment level values")](#)Setting project level and environment level values ##### dbt environment variables order of precedence Environment variables in the dbt platform are managed with a clear [order of precedence](https://docs.getdbt.com/docs/build/environment-variables.md#setting-and-overriding-environment-variables), allowing users to define values at four levels (highest to lowest order of precedence): * The job level (job override) or in the Studio IDE for an individual developer (personal override). *Highest precedence* * The environment level, which can be overridden by the job level or personal override. * A project-wide default value, which can be overridden by the environment level, job level, or personal override. * The optional default argument supplied to the `env_var` Jinja function in the code. *Lowest precedence* [![Environment variables order of precedence](</img/docs/dbt-platform/using-dbt-platform/Environment Variables/env-var-precdence.png?v=2> "Environment variables order of precedence")](#)Environment variables order of precedence ##### Set environment variables in dbt * To set these variables for an entire project or specific environments, navigate to **Deploy** > **Environments** > **Environment variables** tab. * To set these variables at the job level, navigate to **Deploy** > **Jobs** > **Select your job** > **Settings** > **Advanced settings**. * To set these variables at the personal override level, navigate to **Your profile** > **Credentials** > **Select your project** > **Environment variables**. #### Orchestration setup This section outlines the considerations and methods to set up your dbt platform environments and jobs for orchestration. The following categories are covered in this section: * [dbt platform environments](https://docs.getdbt.com/guides/dbt-migration-1.md?step=8#dbt-cloud-environments-1) * [Initial setup steps](https://docs.getdbt.com/guides/dbt-migration-1.md?step=8#initial-setup-steps-1) * [Additional configuration](https://docs.getdbt.com/guides/dbt-migration-1.md?step=8#additional-configuration-3) * [CI/CD setup](https://docs.getdbt.com/guides/dbt-migration-1.md?step=8#cicd-setup) ##### dbt environments To use the [dbt platform's job scheduler](https://docs.getdbt.com/docs/deploy/job-scheduler.md), set up one environment as the production environment. This is the [deployment](https://docs.getdbt.com/docs/deploy/deploy-environments.md) environment. You can set up multiple environments for different stages of your deployment pipeline, such as development, staging/QA, and production. ##### Initial setup steps 1. **dbt version** — In your environment settings, configure the dbt platform with the same dbt version. * Once your full migration is complete, we recommend upgrading your environments to [release tracks](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) to always get the latest features and more. You only need to do this once. 2. **Configure your jobs** — [Create jobs](https://docs.getdbt.com/docs/deploy/deploy-jobs.md#create-and-schedule-jobs) for scheduled or event-driven dbt jobs. You can use cron execution, manual, pull requests, or trigger on the completion of another job. * Note that alongside [jobs in the dbt platform](https://docs.getdbt.com/docs/deploy/jobs.md), discover other ways to schedule and run your dbt jobs with the help of other tools. Refer to [Integrate with other tools](https://docs.getdbt.com/docs/deploy/deployment-tools.md) for more information. ##### Additional configuration Explore these additional configurations to optimize your dbt platform orchestration setup further: 1. **Custom target names** — Use environment variables to set a `custom target.name` for every [corresponding dbt platform job](https://docs.getdbt.com/docs/build/custom-target-names.md) at the environment level. 2. **dbt commands** — Add any relevant [dbt commands](https://docs.getdbt.com/docs/deploy/job-commands.md) to execute your dbt platform jobs runs. 3. **Notifications** — Set up [notifications](https://docs.getdbt.com/docs/deploy/job-notifications.md) by configuring email and Slack alerts to monitor your jobs. 4. **Monitoring tools** — Use [monitoring tools](https://docs.getdbt.com/docs/deploy/monitor-jobs.md) like run history, job retries, job chaining, dashboard status tiles, and more for a seamless orchestration experience. 5. **API access** — Create [API auth tokens](https://docs.getdbt.com/docs/dbt-apis/authentication.md) and access to [dbt platform APIs](https://docs.getdbt.com/docs/dbt-apis/overview.md) as needed. [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") 6. **Catalog** — If you use [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) and run production jobs with an external orchestrator, ensure your production jobs run `dbt run` or `dbt build` to update and view models and their [metadata](https://docs.getdbt.com/docs/explore/explore-projects.md#generate-metadata) in Catalog. Running `dbt compile` alone will not update model metadata. In addition, features like column-level lineage also requires catalog metadata produced through running `dbt docs generate`. [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") ##### CI/CD setup Building a custom solution to efficiently check code upon pull requests is complicated. With the dbt platform, you can enable [continuous integration / continuous deployment (CI/CD)](https://docs.getdbt.com/docs/deploy/continuous-integration.md) and configure the dbt platform to run your dbt projects in a temporary schema when new commits are pushed to open pull requests. [![Workflow of continuous integration in dbt](/img/docs/dbt-platform/using-dbt-platform/ci-workflow.png?v=2 "Workflow of continuous integration in dbt")](#)Workflow of continuous integration in dbt This build-on-PR functionality is a great way to catch bugs before deploying to production, and an essential tool for data practitioners. 1. Set up an integration with a native Git application (such as Azure DevOps, GitHub, GitLab) and a CI environment in the dbt platform. 2. Create [a CI/CD job](https://docs.getdbt.com/docs/deploy/ci-jobs.md) to automate quality checks before code is deployed to production. 3. Run your jobs in a production environment to fully implement CI/CD. Future pull requests will also leverage the last production runs to compare against. #### Model development and discovery In this section, you'll be able to validate whether your models run or compile correctly in your development tool of choice: The [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) or [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md). You'll want to make sure you set up your [development environment and credentials](https://docs.getdbt.com/docs/dbt-platform-environments.md#set-developer-credentials). 1. In your [development tool](https://docs.getdbt.com/docs/platform/about-develop-dbt.md) of choice, you can review your dbt project, ensure it's set up correctly, and run some [dbt commands](https://docs.getdbt.com/reference/dbt-commands.md): * Run `dbt compile` to make sure your project compiles correctly. * Run a few models in the Studio IDE or dbt platform CLI to ensure you're experiencing accurate results in development. 2. Once your first job has successfully run in your production environment, use [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) to view your project's [resources](https://docs.getdbt.com/docs/build/projects.md) (such as models, tests, and metrics) and their data lineage to gain a better understanding of its latest production state. [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") #### What's next? Congratulations on completing the first part of your move to the dbt platform 🎉! You have learned: * How to set up your dbt platform account * How to connect your data platform and Git repository * How to configure your development, orchestration, and CI/CD environments * How to set up environment variables and validate your models For the next steps, you can continue exploring our 3-part-guide series on moving from self-hosted dbt to the dbt platform: | Guide | Information | Audience | | ---------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | ------------------------------------------------- | | [Move to the dbt platform: What you need to know](https://docs.getdbt.com/guides/dbt-migration-2.md) | Understand the considerations and methods needed in your move to the dbt platform. | Team leads<br />Admins | | [Move to the dbt platform: Get started](https://docs.getdbt.com/guides/dbt-migration-1.md?step=1) | Learn the steps needed to move to the dbt platform. | Developers<br />Data engineers<br />Data analysts | | [Move to the dbt platform: Optimization tips](https://docs.getdbt.com/guides/dbt-migration-3.md) | Learn how to optimize your dbt platform experience with common scenarios and useful tips. | Everyone | ##### Why move to the dbt platform? If your team is using self-hosted dbt today, you could be reading this guide because: * You've realized the burden of maintaining that deployment. * The person who set it up has since left. * You're interested in what the dbt platform could do to better manage the complexity of your dbt deployment, democratize access to more contributors, or improve security and governance practices. * You need a governed data foundation for AI—shared definitions, lineage, and testing so analytics and AI give answers the business can trust. Self-hosting hides its true cost in engineer hours and wasted compute. dbt platform eliminates that overhead with managed infrastructure and browser-based development so more people can contribute without you being the bottleneck. State-aware orchestration is now dbt State [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) works with all engines and environments: dbt v1, the dbt platform, and dbt v2. If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). The data layer is the AI layer—make sure it's tested, defined, and trusted end to end. Moving to the dbt platform simplifies workflows by providing a fully managed environment that improves collaboration, security, and orchestration. With the dbt platform, you gain access to features like cross-team collaboration ([dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md)), version management, streamlined CI/CD, [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) for comprehensive insights, and more — making it easier to manage complex dbt deployments and scale your data workflows efficiently. It's ideal for teams looking to reduce the burden of maintaining their own infrastructure while enhancing governance and productivity. ##### Related docs * [Learn dbt](https://learn.getdbt.com) video courses for on-demand learning. * Book [expert-led demos](https://www.getdbt.com/resources/dbt-cloud-demos-with-experts) and insights. * Work with the [dbt Labs' Professional Services](https://www.getdbt.com/dbt-labs/services) team to support your data organization and migration. * [How the dbt platform compares with self-hosted dbt](https://www.getdbt.com/product/dbt-core-vs-dbt-cloud) for a detailed comparison of self-hosted dbt and the dbt platform. * Subscribe to the [dbt platform RSS alerts](https://status.getdbt.com/) --- ### Move to the dbt platform: Optimization tips [Back to guides](https://docs.getdbt.com/guides.md) Migration dbt platform Intermediate [Menu ]() #### Introduction Moving from self-hosted dbt to the dbt platform streamlines analytics engineering workflows by allowing teams to develop, test, deploy, and explore data products using a single, fully managed software service. It's not just better tooling — it's about lowering total cost of ownership, powering AI with trusted data, and scaling with governed self-service. Explore our 3-part-guide series on moving from self-hosted dbt to the dbt platform. The series is ideal for users aiming for streamlined workflows and enhanced analytics: | Guide | Information | Audience | | ---------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | ------------------------------------------------- | | [Move to the dbt platform: What you need to know](https://docs.getdbt.com/guides/dbt-migration-2.md) | Understand the considerations and methods needed in your move to the dbt platform. | Team leads<br />Admins | | [Move to the dbt platform: Get started](https://docs.getdbt.com/guides/dbt-migration-1.md?step=1) | Learn the steps needed to move to the dbt platform. | Developers<br />Data engineers<br />Data analysts | | [Move to the dbt platform: Optimization tips](https://docs.getdbt.com/guides/dbt-migration-3.md) | Learn how to optimize your dbt platform experience with common scenarios and useful tips. | Everyone | ##### Why move to the dbt platform? If your team is using self-hosted dbt today, you could be reading this guide because: * You've realized the burden of maintaining that deployment. * The person who set it up has since left. * You're interested in what the dbt platform could do to better manage the complexity of your dbt deployment, democratize access to more contributors, or improve security and governance practices. * You need a governed data foundation for AI—shared definitions, lineage, and testing so analytics and AI give answers the business can trust. Self-hosting hides its true cost in engineer hours and wasted compute. dbt platform eliminates that overhead with managed infrastructure and browser-based development so more people can contribute without you being the bottleneck. State-aware orchestration is now dbt State [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) works with all engines and environments: dbt v1, the dbt platform, and dbt v2. If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). The data layer is the AI layer—make sure it's tested, defined, and trusted end to end. Moving to the dbt platform simplifies workflows by providing a fully managed environment that improves collaboration, security, and orchestration. With the dbt platform, you gain access to features like cross-team collaboration ([dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md)), version management, streamlined CI/CD, [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) for comprehensive insights, and more — making it easier to manage complex dbt deployments and scale your data workflows efficiently. It's ideal for teams looking to reduce the burden of maintaining their own infrastructure while enhancing governance and productivity. #### What you'll learn You may have already started your move to the dbt platform and are looking for tips to help you optimize your dbt platform experience. This guide includes tips and caveats for the following areas: * [Adapters and connections](https://docs.getdbt.com/guides/dbt-migration-3.md?step=3) * [Development tools](https://docs.getdbt.com/guides/dbt-migration-3.md?step=4) * [Orchestration](https://docs.getdbt.com/guides/dbt-migration-3.md?step=5) * [Mesh](https://docs.getdbt.com/guides/dbt-migration-3.md?step=6) * [Semantic Layer](https://docs.getdbt.com/guides/dbt-migration-3.md?step=7) * [Catalog](https://docs.getdbt.com/guides/dbt-migration-3.md?step=8) #### Adapters and connections In the dbt platform, you can natively connect to your data platform and test its [connection](https://docs.getdbt.com/docs/connect-adapters.md) with a click of a button. This is especially useful for users who are new to the dbt platform or are looking to streamline their connection setup. Here are some tips and caveats to consider: ##### Tips * Manage [dbt versions](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-platform-version.md) and ensure team collaboration with dbt platform's one-click feature, eliminating the need for manual updates and version discrepancies. Select a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) for ongoing updates, to always stay up to date with fixes and (optionally) get early access to new functionality for your dbt project. * The dbt platform supports a whole host of [cloud providers](https://docs.getdbt.com/docs/platform/connect-data-platform/about-connections.md), including Snowflake, Databricks, BigQuery, Fabric, and Redshift (to name a few). * Use [Extended Attributes](https://docs.getdbt.com/docs/deploy/deploy-environments.md#extended-attributes) to set a flexible [profiles.yml](https://docs.getdbt.com/docs/local/profiles.yml.md) snippet in your dbt platform environment settings. It gives you more control over environments (both deployment and development) and extends how the dbt platform connects to the data platform within a given environment. * For example, if you have a field in your `profiles.yml` that you’d like to add to the dbt platform adapter user interface, you can use Extended Attributes to set it. ##### Caveats * Not all parameters are available for adapters. * A project can only use one warehouse type. #### Development tools The dbt platform empowers data practitioners to develop in the tool of their choice. It ships with a [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) (local) or [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) (browser-based) to build, test, run, and version control your dbt projects. Both development tools are tailored to suit different audiences and preferences within your team. To streamline your team's workflow, it's important to know who will prefer the Studio IDE and who might lean towards the dbt platform CLI. This section aims to clarify these preferences. ##### Studio IDE A web-based interface for building, testing, running, and version-controlling dbt projects. It compiles dbt code into SQL and executes it directly on your database. The Studio IDE makes developing fast and easy for new and seasoned data practitioners to build and test changes. **Who might prefer the Studio IDE?** * New dbt users or those transitioning from other tools who appreciate a more guided experience through a browser-based interface. * Team members focused on speed and convenience for getting started with a new or existing project. * Individuals who prioritize direct feedback from the Studio IDE, such as seeing unsaved changes. **Key features** * The Studio IDE has simplified Git functionality: * Create feature branches from the branch configured in the development environment. * View saved but not-committed code changes directly in the Studio IDE. * [Format or lint](https://docs.getdbt.com/docs/platform/studio-ide/lint-format.md) your code with `sqlfluff` or `sqlfmt`. This includes support for adding your custom linting rules. * Allows users to natively [defer to production](https://docs.getdbt.com/docs/platform/about-defer.md#defer-in-dbt-cli) metadata directly in their development workflows, reducing the number of objects. * Support running multiple dbt commands at the same time through [safe parallel execution](https://docs.getdbt.com/reference/dbt-commands.md#parallel-execution), a [feature](https://docs.getdbt.com/docs/platform/about-platform/dbt-platform-features.md) available in dbt platform's infrastructure. In contrast, `dbt-core` *doesn't support* safe parallel execution for multiple invocations in the same process. The Studio IDE provides a simplified interface that's accessible to all users, regardless of their technical background. However, there are some capabilities that are intentionally not available in the Studio IDE due to its focus on simplicity and ease of use: * Pre-commit for automated checks before *committing* code is not available (yet). * Mass-generating files / interacting with the file system are not available. * Combining/piping commands, such as `dbt run -s (bash command)`, is not available. ##### dbt platform CLI The dbt platform CLI allows you to run dbt [commands](https://docs.getdbt.com/reference/dbt-commands.md#available-commands) against your dbt platform development environment from your local command line. For users who seek full control over their development environment and ideal for those comfortable with the command line. When moving from self-hosted dbt to the dbt platform, make sure you check the `.gitignore` file contains the [necessary folders](https://docs.getdbt.com/docs/platform/git/version-control-basics.md#the-gitignore-file). dbt doesn't interact with git so the dbt platform doesn't automatically add or verify entries in the `.gitignore` file. Additionally, if the repository already contains dbt code and doesn't require initialization, the dbt platform won't add any missing entries to the `.gitignore file`. **Who might prefer the dbt platform CLI?** * Data practitioners accustomed to working with a specific set of development tooling. * Users looking for granular control over their Git workflows (such as pre-commits for automated checks before committing code). * Data practitioners who need to perform complex operations, like mass file generation or specific command combinations. **Key features** * Allows users to run dbt commands against their dbt platform development environment from their local command line with minimal configuration. * Allows users to natively [defer to production](https://docs.getdbt.com/docs/platform/about-defer.md#defer-in-dbt-cli) metadata directly in their development workflows, reducing the number of objects. * Support running multiple dbt commands at the same time through [safe parallel execution](https://docs.getdbt.com/reference/dbt-commands.md#parallel-execution), a [feature](https://docs.getdbt.com/docs/platform/about-platform/dbt-platform-features.md) available in dbt platform's infrastructure. In contrast, `dbt-core` *doesn't support* safe parallel execution for multiple invocations in the same process. * Able to use Visual Studio (VS) Code extensions #### Orchestration The dbt platform provides robust orchestration that enables you to schedule, run, and monitor dbt jobs with ease. Here are some tips and caveats to consider when using dbt platform's orchestration features: ##### Tips * Enable [partial parsing](https://docs.getdbt.com/docs/platform/account-settings.md#partial-parsing) between jobs in the dbt platform to significantly speed up project parsing by only processing changed files, optimizing performance for large projects. * [Run multiple CI/CD](https://docs.getdbt.com/docs/deploy/continuous-integration.md) jobs at the same time which will not block production runs. The Job scheduler automatically cancels stale runs when a newer commit is pushed. This is because each PR will run in its own schema. * The dbt platform automatically [cancels](https://docs.getdbt.com/docs/deploy/job-scheduler.md#run-cancellation-for-over-scheduled-jobs) a scheduled run if the existing run is still executing. This prevents unnecessary, duplicative executions. * Protect you and your data freshness from third-party outages by enabling dbt platform's [Git repository caching](https://docs.getdbt.com/docs/platform/account-settings.md#git-repository-caching), which keeps a cache of the project's Git repository. [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") * [Link deploy jobs](https://docs.getdbt.com/docs/deploy/deploy-jobs.md#trigger-on-job-completion) across dbt platform projects by configuring your job or using the [Create Job API](https://docs.getdbt.com/dbt-cloud/api-v2#/operations/Create%20Job) to do this. [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") * [Rerun your jobs](https://docs.getdbt.com/docs/deploy/retry-jobs.md) from the start or the point of failure if your dbt job run completed with a status of **`Error.`** ##### Caveats * To automate the setup and configuration of your dbt platform, you can store your job configurations as code within a repository: * Check out our [Terraform provider.](https://registry.terraform.io/providers/dbt-labs/dbtcloud/latest/docs/resources/job) * Alternatively, check out our [jobs-as-code](https://github.com/dbt-labs/dbt-jobs-as-code) repository, which is a tool built to handle dbt platform jobs as a well-defined YAML file. * dbt platform users and external emails can receive notifications if a job fails, succeeds, or is cancelled. To get notifications for warnings, you can create a [webhook subscription](https://docs.getdbt.com/guides/zapier-slack.md) and post to Slack. #### dbt Mesh [Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md) helps organizations with mature, complex transformation workflows in dbt increase the flexibility and performance of their dbt projects. It allows you to make use of multiple interconnected dbt projects instead of a single large, monolithic project. It enables you to interface and navigate between different projects and models with [cross-project dependencies](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#how-to-write-cross-project-ref), enhancing collaboration and data governance. Here are some tips and caveats to consider when using Mesh: ##### Tips * To dynamically resolve [cross-project references](https://docs.getdbt.com/docs/mesh/govern/project-dependencies.md#how-to-write-cross-project-ref), all developers need to develop with the dbt platform (either with the dbt platform CLI or Studio IDE). Cross-project references aren't natively supported in self-hosted dbt, except by installing the source code from upstream projects [as packages](https://docs.getdbt.com/docs/build/packages.md#how-do-i-add-a-package-to-my-project) * Link models across projects for a modular and scalable approach for your project and teams. * Manage access to your dbt models both within and across projects using: * **[Groups](https://docs.getdbt.com/docs/mesh/govern/model-access.md#groups)** — Organize nodes in your dbt DAG that share a logical connection and assign an owner to the entire group. * **[Model access](https://docs.getdbt.com/docs/mesh/govern/model-access.md#access-modifiers)** — Control which other models or projects can reference this model. * **[Model versions](https://docs.getdbt.com/docs/mesh/govern/model-versions.md)** — Enable adoption and deprecation of models as they evolve. * **[Model contracts](https://docs.getdbt.com/docs/mesh/govern/model-contracts.md)** — Set clear expectations on the shape of the data to ensure data changes upstream of dbt or within a project's logic don't break downstream consumers' data products. ##### Caveats * To use cross-project references in dbt, each dbt project must correspond to just one dbt platform project. We strongly discourage defining multiple projects for the same codebase, even if you're trying to manage access permissions, connect to different data warehouses, or separate production and non-production data. While this was required historically, features like [Staging environments](https://docs.getdbt.com/docs/dbt-platform-environments.md#types-of-environments), Environment-level RBAC (*coming soon*), and [Extended attributes](https://docs.getdbt.com/docs/dbt-platform-environments.md#extended-attributes) will make it unnecessary. * Project dependencies are uni-directional, meaning they go in one direction. This means dbt checks for cycles across projects (circular dependencies) and raise errors if any are detected. However, we are considering support to allow projects to depend on each other in both directions in the future, with dbt still checking for node-level cycles while allowing cycles at the project level. * Everyone in the account can view public model metadata, which helps users find data products more easily. This is separate from who can access the actual data, which is controlled by permissions in the data warehouse. For use cases where even metadata about a reusable data asset is sensitive, we are [considering](https://github.com/dbt-labs/dbt/issues/9340) an optional extension of protected models. Refer to the [Mesh FAQs](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-5-faqs.md) for more questions. #### dbt Semantic Layer Leverage the [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md), powered by MetricFlow, to create a unified view of your business metrics, ensuring consistency across all analytics tools. The data layer is the foundation for AI as well as analytics — shared definitions and lineage give AI and BI the same trusted context so answers are consistent and actionable. Here are some tips and caveats to consider when using Semantic Layer: ##### Tips * Define semantic models and metrics once in the dbt platform with the [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) (powered by MetricFlow). Reuse them across various analytics platforms, reducing redundancy and errors. * Use the [Semantic Layer APIs](https://docs.getdbt.com/docs/dbt-apis/sl-api-overview.md) to query metrics in downstream tools for consistent, reliable data metrics. * Connect to several data applications, from business intelligence tools to notebooks, spreadsheets, data catalogs, and more, to query your metrics. [Available integrations](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md) include Tableau, Google Sheets, Hex, and more. * Use [exports](https://docs.getdbt.com/docs/use-dbt-semantic-layer/exports.md) to write commonly used queries directly within your data platform, on a schedule. ##### Caveats * Semantic Layer currently supports the Deployment environment for querying. Development querying experience coming soon. * Run queries/semantic layer commands in the dbt platform CLI, however running queries/semantic layer commands in the Studio IDE isn't supported *yet.* * Semantic Layer doesn't support using [Single sign-on (SSO)](https://docs.getdbt.com/docs/platform/manage-access/sso-overview.md) for Semantic Layer [production credentials](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md#permissions-for-service-account-tokens), however, SSO is supported for development user accounts. Refer to the [Semantic Layer FAQs](https://docs.getdbt.com/docs/use-dbt-semantic-layer/sl-faqs.md) for more information. #### dbt Catalog [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) enhances your ability to discover, understand, and troubleshoot your data assets through rich metadata and lineage visualization. Lineage and discovery are essential for governance and for feeding reliable context to AI workflows. Here are some tips and caveats to consider when using Catalog: ##### Tips * Use the search and filter capabilities in Catalog to quickly locate models, sources, and tests, streamlining your workflow. * View all the [different projects](https://docs.getdbt.com/docs/explore/explore-multiple-projects.md) and public models in the account, where the public models are defined, and how they are used to gain a better understanding of your cross-project resources. * Use the [Lenses](https://docs.getdbt.com/docs/explore/explore-projects.md#lenses) feature, which are map-like layers for your DAG, available from your project's lineage graph. Lenses help you further understand your project's contextual metadata at scale, especially to distinguish a particular model or a subset of models. * Access column-level lineage (CLL) for the resources in your dbt project. [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") ##### Caveats * There must be at least one successful job run in the production deployment environment for Catalog to populate information. Familiarize yourself with Catalog's features to fully leverage its capabilities to avoid missed opportunities for efficiency gains. Refer to the [Catalog FAQs](https://docs.getdbt.com/docs/explore/dbt-explorer-faqs.md) for more information. #### What's next? Congratulations on making it through the guide 🎉! We hope you're equipped with useful insights and tips to help you with your move. Something to note is that moving from self-hosted dbt to the dbt platform isn't just about evolving your data projects, it's about exploring new levels of collaboration, governance, efficiency, and innovation within your team, and building a data layer that's ready for AI. For the next steps, continue exploring our 3-part-guide series on moving from self-hosted dbt to the dbt platform: | Guide | Information | Audience | | ---------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | ------------------------------------------------- | | [Move to the dbt platform: What you need to know](https://docs.getdbt.com/guides/dbt-migration-2.md) | Understand the considerations and methods needed in your move to the dbt platform. | Team leads<br />Admins | | [Move to the dbt platform: Get started](https://docs.getdbt.com/guides/dbt-migration-1.md?step=1) | Learn the steps needed to move to the dbt platform. | Developers<br />Data engineers<br />Data analysts | | [Move to the dbt platform: Optimization tips](https://docs.getdbt.com/guides/dbt-migration-3.md) | Learn how to optimize your dbt platform experience with common scenarios and useful tips. | Everyone | ##### Why move to the dbt platform? If your team is using self-hosted dbt today, you could be reading this guide because: * You've realized the burden of maintaining that deployment. * The person who set it up has since left. * You're interested in what the dbt platform could do to better manage the complexity of your dbt deployment, democratize access to more contributors, or improve security and governance practices. * You need a governed data foundation for AI—shared definitions, lineage, and testing so analytics and AI give answers the business can trust. Self-hosting hides its true cost in engineer hours and wasted compute. dbt platform eliminates that overhead with managed infrastructure and browser-based development so more people can contribute without you being the bottleneck. State-aware orchestration is now dbt State [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) works with all engines and environments: dbt v1, the dbt platform, and dbt v2. If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). The data layer is the AI layer—make sure it's tested, defined, and trusted end to end. Moving to the dbt platform simplifies workflows by providing a fully managed environment that improves collaboration, security, and orchestration. With the dbt platform, you gain access to features like cross-team collaboration ([dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md)), version management, streamlined CI/CD, [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) for comprehensive insights, and more — making it easier to manage complex dbt deployments and scale your data workflows efficiently. It's ideal for teams looking to reduce the burden of maintaining their own infrastructure while enhancing governance and productivity. ##### Resources If you need any additional help or have some questions, use the following resources: * [dbt Learn courses](https://learn.getdbt.com) for on-demand video learning. * Our [Support team](https://docs.getdbt.com/docs/dbt-support.md) is always available to help you troubleshoot your dbt platform issues. * Join the [dbt Community](https://community.getdbt.com/) to connect with other dbt users, ask questions, and share best practices. * Subscribe to the [dbt platform RSS alerts](https://status.getdbt.com/) * Enterprise accounts have an account management team available to help troubleshoot solutions and account management assistance. [Book a demo](https://www.getdbt.com/contact) to learn more. * [How the dbt platform compares with self-hosted dbt](https://www.getdbt.com/product/dbt-core-vs-dbt-cloud) for a detailed comparison of self-hosted dbt and the dbt platform. For tailored assistance, you can use the following resources: * Book [expert-led demos](https://www.getdbt.com/resources/dbt-cloud-demos-with-experts) and insights * Work with the [dbt Labs' Professional Services](https://www.getdbt.com/dbt-labs/services) team to support your data organization and move. --- ### Move to the dbt platform: What you need to know [Back to guides](https://docs.getdbt.com/guides.md) Migration dbt platform Intermediate [Menu ]() #### Introduction Moving from self-hosted dbt to the dbt platform streamlines analytics engineering workflows by allowing teams to develop, test, deploy, and explore data products using a single, fully managed software service. It's not just better tooling — it's about lowering total cost of ownership, powering AI with trusted data, and scaling with governed self-service. Explore our 3-part-guide series on moving from self-hosted dbt to the dbt platform. The series is ideal for users aiming for streamlined workflows and enhanced analytics: | Guide | Information | Audience | | ---------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | ------------------------------------------------- | | [Move to the dbt platform: What you need to know](https://docs.getdbt.com/guides/dbt-migration-2.md) | Understand the considerations and methods needed in your move to the dbt platform. | Team leads<br />Admins | | [Move to the dbt platform: Get started](https://docs.getdbt.com/guides/dbt-migration-1.md?step=1) | Learn the steps needed to move to the dbt platform. | Developers<br />Data engineers<br />Data analysts | | [Move to the dbt platform: Optimization tips](https://docs.getdbt.com/guides/dbt-migration-3.md) | Learn how to optimize your dbt platform experience with common scenarios and useful tips. | Everyone | ##### Why move to the dbt platform? If your team is using self-hosted dbt today, you could be reading this guide because: * You've realized the burden of maintaining that deployment. * The person who set it up has since left. * You're interested in what the dbt platform could do to better manage the complexity of your dbt deployment, democratize access to more contributors, or improve security and governance practices. * You need a governed data foundation for AI—shared definitions, lineage, and testing so analytics and AI give answers the business can trust. Self-hosting hides its true cost in engineer hours and wasted compute. dbt platform eliminates that overhead with managed infrastructure and browser-based development so more people can contribute without you being the bottleneck. State-aware orchestration is now dbt State [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) works with all engines and environments: dbt v1, the dbt platform, and dbt v2. If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). The data layer is the AI layer—make sure it's tested, defined, and trusted end to end. Moving to the dbt platform simplifies workflows by providing a fully managed environment that improves collaboration, security, and orchestration. With the dbt platform, you gain access to features like cross-team collaboration ([dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md)), version management, streamlined CI/CD, [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) for comprehensive insights, and more — making it easier to manage complex dbt deployments and scale your data workflows efficiently. It's ideal for teams looking to reduce the burden of maintaining their own infrastructure while enhancing governance and productivity.  What are dbt and dbt v1? * The dbt platform is the fastest and most reliable way to deploy dbt. It enables you to develop, test, deploy, and explore data products using a single, fully managed service. It also supports: * Development experiences tailored to multiple personas ([Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) or [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md)) * Out-of-the-box [CI/CD workflows](https://docs.getdbt.com/docs/deploy/ci-jobs.md) * The [Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) for consistent metrics * Domain ownership of data with multi-project [Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md) setups * [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) for easier data discovery and understanding Learn more about [dbt platform features](https://docs.getdbt.com/docs/platform/about-platform/dbt-platform-features.md). * dbt is an open-source tool that enables data teams to define and execute data transformations in a cloud data warehouse following analytics engineering best practices. While this can work well for 'single players' and small technical teams, all development happens on a command-line interface, and production deployments must be self-hosted and maintained. This requires significant, costly work that adds up over time to maintain and scale. #### What you'll learn Today thousands of companies, with data teams ranging in size from 2 to 2,000, rely on the dbt platform to accelerate data work, increase collaboration, and win the trust of the business. Understanding what you'll need to do in order to move between the dbt platform and your current self-hosted dbt deployment will help you strategize and plan for your move. The guide outlines the following steps: * [Considerations](https://docs.getdbt.com/guides/dbt-migration-2.md?step=3): Learn about the most important things you need to think about when moving from self-hosted dbt to the dbt platform. * [Plan your move](https://docs.getdbt.com/guides/dbt-migration-2.md?step=4): Considerations you need to make, such as user roles and permissions, onboarding order, current workflows, and more. * [Move to the dbt platform](https://docs.getdbt.com/guides/dbt-migration-2.md?step=5): Review the steps to move your dbt project to the dbt platform, including setting up your account, data platform, and Git repository. * [Test and validate](https://docs.getdbt.com/guides/dbt-migration-2.md?step=6): Discover how to ensure model accuracy and performance post-move. * [Transition and training](https://docs.getdbt.com/guides/dbt-migration-2.md?step=7): Learn how to fully transition to the dbt platform and what training and support you may need. * [Summary](https://docs.getdbt.com/guides/dbt-migration-2.md?step=8): Summarizes key takeaways and what you've learned in this guide. * [What's next?](https://docs.getdbt.com/guides/dbt-migration-2.md?step=9): Introduces what to expect in the following guides. #### Considerations If your team is using self-hosted dbt today, you could be reading this guide because: * You've realized the burden of maintaining that deployment. * The person who set it up has since left. * You're interested in what the dbt platform could do to better manage the complexity of your dbt deployment, democratize access to more contributors, or improve security and governance practices. This guide shares the technical adjustments and team collaboration strategies you'll need to know to move your project from self-hosted dbt to the dbt platform. Each "build your own" deployment of dbt will look a little different, but after seeing hundreds of teams make the migration, there are many things in common. The most important things you need to think about when moving from self-hosted dbt to the dbt platform: * How is your team structured? Are there natural divisions of domain? * Should you have one project or multiple? Which dbt resources do you want to standardize & keep central? * Who should have permission to view, develop, and administer? * How are you scheduling your dbt models to run in production? * How are you currently managing Continuous integration/Continuous deployment (CI/CD) of logical changes (if at all)? * How do your data developers prefer to work? * How do you manage different data environments and the different behaviors in those environments? The dbt platform provides standard mechanisms for tackling these considerations, all of which deliver long-term benefits to your organization: * Cross-team collaboration * Access control * Orchestration * Isolated data environments If you have rolled out your own self-hosted dbt deployment, you have probably come up with different answers. #### Plan your move As you plan your move, consider your workflow and team layout to ensure a smooth transition. Here are some key considerations to keep in mind:  Start small to minimize risk and maximize learning You don't need to move every team and every developer's workflow all at once. Many customers with large dbt deployments start by moving one team and one project. Once the benefits of a consolidated platform are clear, move the rest of your teams and workflows. While long-term 'hybrid' deployments can be challenging, it may make sense as a temporary on-ramp.  User roles and responsibilities Assess the users or personas involved in the pre-move, during the move, and post-move. * **Administrators**: Plan for new [access controls](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md) in the dbt platform, such as deciding what teams can manage themselves and what should be standardized. Determine who will be responsible for setting up and maintaining projects, data platform connections, and environments. * **Data developers** (data analysts, data engineers, analytics engineers, business analysts): Determine onboarding order, workflow adaptation in the dbt platform, training on [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) or [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) usage, and role changes. * **Data consumers:** Discover data insights by using [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) to view your project's resources (such as models, tests, and metrics) and their lineage to gain a better understanding of its latest production state. [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")  Onboarding order If you have multiple teams of dbt developers, think about how to start your onboarding sequence for the dbt platform: * Start with downstream (like business-embedded teams) who may benefit from the Studio IDE as dev experience (less technical users) and sharing features (like auto-deferral and Catalog) to share with their stakeholders, moving to more technical teams later. * Consider setting up a [CI job](https://docs.getdbt.com/docs/deploy/ci-jobs.md) in the dbt platform (even before development or production jobs) to streamline development workflows. This is especially beneficial if there's no existing CI process.  Analyze current workflows, review processes, and team structures Discover how the dbt platform can help simplify development, orchestration, and testing: * **Development**: Develop dbt models, allowing you to build, test, run, and version control your dbt projects using the dbt platform CLI (command line interface or code editor) or Studio IDE (browser-based). * **Orchestration**: Create custom schedules to run your production jobs. Schedule jobs by day of the week, time of day, or a recurring interval. * Set up [a CI job](https://docs.getdbt.com/docs/deploy/ci-jobs.md) to ensure developer effectiveness, and CD jobs to deploy changes as soon as they're merged. * Link deploy jobs together by [triggering a job](https://docs.getdbt.com/docs/deploy/deploy-jobs.md#trigger-on-job-completion) when another one is completed. * For the most flexibility, use the [dbt platform API](https://docs.getdbt.com/dbt-cloud/api-v2#/) to trigger jobs. This makes sense when you want to integrate dbt execution with other data workflows. * **Continuous integration (CI)**: Use [CI jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md) to run your dbt projects in a temporary schema when new commits are pushed to open pull requests. This build-on-PR functionality is a great way to catch bugs before deploying to production. * For many teams, dbt platform CI represents a major improvement compared to their previous development workflows. * **How are you defining tests today?**: While testing production data is important, it's not the most efficient way to catch logical errors introduced by developers You can use [unit testing](https://docs.getdbt.com/docs/build/unit-tests.md) to allow you to validate your SQL modeling logic on a small set of static inputs *before* you materialize your full model in production.  Understand access control Transition to the dbt platform's [access control](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md) mechanisms to ensure security and proper access management. dbt platform administrators can use the dbt platform's permission model to control user-level access in a dbt platform account: * **License-based access controls:** Users are configured with account-wide license types. These licenses control the things a user can do within the application: view project metadata, develop changes within those projects, or administer access to those projects. * **Role-based Access Control (RBAC):** Users are assigned to *groups* with specific permissions on specific projects or all projects in the account. A user may be a member of multiple groups, and those groups may have permissions on multiple projects. [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")  Manage environments If you require isolation between production and pre-production data environments due to sensitive data, the dbt platform can support Development, Staging, and Production data [environments](https://docs.getdbt.com/docs/dbt-platform-environments.md). This provides developers with the benefits of an enhanced workflow while ensuring isolation between Staging and Production data, and locking down permissions on Prod. #### Move to dbt This guide is your roadmap to help you think about migration strategies and what moving from self-hosted dbt to the dbt platform could look like. After reviewing the considerations and planning your move, you may want to start moving your dbt project to the dbt platform: * Check out the detailed [Move to the dbt platform: Get started](https://docs.getdbt.com/guides/dbt-migration-1.md?step=1) guide for useful tasks and insights for a smooth transition from self-hosted dbt to the dbt platform. For a more detailed comparison of self-hosted dbt and the dbt platform, check out [How the dbt platform compares with self-hosted dbt](https://www.getdbt.com/product/dbt-core-vs-dbt-cloud). #### Test and validate After [setting the foundations of the dbt platform](https://docs.getdbt.com/guides/dbt-migration-1.md?step=1), it's important to validate your migration to ensure seamless functionality and data integrity: * **Review your dbt project:** Ensure your project compiles correctly and that you can run commands. Make sure your models are accurate and monitor performance post-move. * **Start cutover:** You can start the cutover to the dbt platform by creating a dbt platform job with commands that only run a small subset of the DAG. Validate the tables are being populated in the proper database/schemas as expected. Then continue to expand the scope of the job to include more sections of the DAG as you gain confidence in the results. * **Precision testing:** Use [unit testing](https://docs.getdbt.com/docs/build/unit-tests.md) to allow you to validate your SQL modeling logic on a small set of static inputs *before* you materialize your full model in production. * **Access and permissions**: Review and adjust [access controls and permissions](https://docs.getdbt.com/docs/platform/manage-access/about-user-access.md) within the dbt platform to maintain security protocols and safeguard your data. #### Transition and training Once you've confirmed that dbt platform orchestration and CI/CD are working as expected, you should pause your current orchestration tool and stop or update your current CI/CD process. This is not relevant if you're still using an external orchestrator (such as Airflow), and you've swapped out `dbt-core` execution for dbt platform execution (through the [API](https://docs.getdbt.com/docs/dbt-apis/overview.md)). Familiarize your team with the dbt platform's [features](https://docs.getdbt.com/docs/platform/about-platform/dbt-platform-features.md) and optimize development and deployment processes. Some key features to consider include: * **Release tracks:** Choose a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) for automatic dbt version upgrades, at the cadence appropriate for your team — removing the hassle of manual updates and the risk of version discrepancies. You can also get early access to new functionality, ahead of self-hosted dbt. * **Development tools**: Use the [dbt platform CLI](https://docs.getdbt.com/docs/platform/dbt-cli-installation.md) or [Studio IDE](https://docs.getdbt.com/docs/platform/studio-ide/develop-in-studio.md) to build, test, run, and version control your dbt projects. * **Documentation and Source freshness:** Automate storage of [documentation](https://docs.getdbt.com/docs/build/documentation.md) and track [source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) in the dbt platform, which streamlines project maintenance. * **Notifications and logs:** Receive immediate [notifications](https://docs.getdbt.com/docs/deploy/monitor-jobs.md) for job failures, with direct links to the job details. Access comprehensive logs for all job runs to help with troubleshooting. * **CI/CD:** Use the dbt platform's [CI/CD](https://docs.getdbt.com/docs/deploy/ci-jobs.md) feature to run your dbt projects in a temporary schema whenever new commits are pushed to open pull requests. This helps with catching bugs before deploying to production. ##### Beyond your move Now that you've chosen the dbt platform as your platform, you've unlocked the power of streamlining collaboration, enhancing workflow efficiency, and leveraging powerful [features](https://docs.getdbt.com/docs/platform/about-platform/dbt-platform-features.md) for analytics engineering teams. Here are some additional features you can use to unlock the full potential of the dbt platform: * **Audit logs:** Use [audit logs](https://docs.getdbt.com/docs/platform/manage-access/audit-log.md) to review actions performed by people in your organization. Audit logs contain audited user and system events in real time. You can even [export](https://docs.getdbt.com/docs/platform/manage-access/audit-log.md#exporting-logs) *all* the activity (beyond the 90 days you can view in the dbt platform). [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") * **dbt platform APIs:** Use the dbt platform's robust [APIs](https://docs.getdbt.com/docs/dbt-apis/overview.md) to create, read, update, and delete (CRUD) projects/jobs/environments project. The [dbt platform Administrative API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md) and [Terraform provider](https://registry.terraform.io/providers/dbt-labs/dbtcloud/latest/docs/resources/job) facilitate programmatic access and configuration storage. While the [Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md) offers extensive metadata querying capabilities, such as job data, model configurations, usage, and overall project health. [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") * **Catalog**: Use [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) to view your project's [resources](https://docs.getdbt.com/docs/build/projects.md) (such as models, tests, and metrics) and their [lineage](https://docs.getdbt.com/terms/data-lineage) to gain a better understanding of its latest production state. (Once you have a successful job in a Production environment). [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") * **dbt Semantic Layer:** The [dbt Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl.md) allows you to define universal metrics on top of your models that can then be queried in your [business intelligence (BI) tool](https://docs.getdbt.com/docs/platform-integrations/avail-sl-integrations.md). This means no more inconsistent metrics — there's now a centralized way to define these metrics and create visibility in every component of the data flow. [Starter](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") * **dbt Mesh:** Use [dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md) to share data models across organizations, enabling data teams to collaborate on shared data models and leverage the work of other teams. [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") ##### Additional help * **dbt Learn courses**: Access our free [Learn dbt](https://learn.getdbt.com) video courses for on-demand training. * **dbt Community:** Join the [dbt Community](https://community.getdbt.com/) to connect with other dbt users, ask questions, and share best practices. * **dbt Support team:** Our [dbt Support team](https://docs.getdbt.com/docs/dbt-support.md) is always available to help you troubleshoot your dbt issues. Create a support ticket in the dbt platform and we'll be happy to help! * **Account management** Enterprise accounts have an account management team available to help troubleshoot solutions and account management assistance. [Book a demo](https://www.getdbt.com/contact) to learn more. [Enterprise](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing")[Enterprise +](https://www.getdbt.com/pricing "Go to https://www.getdbt.com/pricing") #### Summary This guide should now have given you some insight and equipped you with a framework for moving from self-hosted dbt to the dbt platform. This guide has covered the following key areas: * **Considerations:** Understanding the foundational steps required for a successful migration, including evaluating your current setup and identifying key considerations unique to your team's structure and workflow needs. * **Plan you move**: Highlighting the importance of workflow redesign, role-specific responsibilities, and the adoption of new processes to harness the dbt platform's collaborative and efficient environment. * **Move to the dbt platform**: Linking to [the guide](https://docs.getdbt.com/guides/dbt-migration-1.md?step=1) that outlines technical steps required to transition your dbt project to the dbt platform, including setting up your account, data platform, and Git repository. * **Test and validate**: Emphasizing technical transitions, including testing and validating your dbt projects within the dbt platform ecosystem to ensure data integrity and performance. * **Transition and training**: Share useful transition, training, and onboarding information for your team. Fully leverage the dbt platform's capabilities, from development tools (dbt platform CLI and Studio IDE) to advanced features such as Catalog, the Semantic Layer, and Mesh. #### What's next? Congratulations on finishing this guide, we hope it's given you insight into the considerations you need to take to best plan your move to the dbt platform. For the next steps, you can continue exploring our 3-part-guide series on moving from self-hosted dbt to the dbt platform: | Guide | Information | Audience | | ---------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | ------------------------------------------------- | | [Move to the dbt platform: What you need to know](https://docs.getdbt.com/guides/dbt-migration-2.md) | Understand the considerations and methods needed in your move to the dbt platform. | Team leads<br />Admins | | [Move to the dbt platform: Get started](https://docs.getdbt.com/guides/dbt-migration-1.md?step=1) | Learn the steps needed to move to the dbt platform. | Developers<br />Data engineers<br />Data analysts | | [Move to the dbt platform: Optimization tips](https://docs.getdbt.com/guides/dbt-migration-3.md) | Learn how to optimize your dbt platform experience with common scenarios and useful tips. | Everyone | ##### Why move to the dbt platform? If your team is using self-hosted dbt today, you could be reading this guide because: * You've realized the burden of maintaining that deployment. * The person who set it up has since left. * You're interested in what the dbt platform could do to better manage the complexity of your dbt deployment, democratize access to more contributors, or improve security and governance practices. * You need a governed data foundation for AI—shared definitions, lineage, and testing so analytics and AI give answers the business can trust. Self-hosting hides its true cost in engineer hours and wasted compute. dbt platform eliminates that overhead with managed infrastructure and browser-based development so more people can contribute without you being the bottleneck. State-aware orchestration is now dbt State [dbt State](https://docs.getdbt.com/docs/deploy/dbt-state-about.md) works with all engines and environments: dbt v1, the dbt platform, and dbt v2. If you were using state-aware orchestration prior to June 1, 2026, you can continue using it. Once you start your free dbt State trial, it will be extended beyond the standard 30-day period. If the extension isn't applied to your account, contact your account team. To get started, refer to [Migrate from state-aware orchestration](https://docs.getdbt.com/docs/deploy/dbt-state-migration.md). The data layer is the AI layer—make sure it's tested, defined, and trusted end to end. Moving to the dbt platform simplifies workflows by providing a fully managed environment that improves collaboration, security, and orchestration. With the dbt platform, you gain access to features like cross-team collaboration ([dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro.md)), version management, streamlined CI/CD, [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) for comprehensive insights, and more — making it easier to manage complex dbt deployments and scale your data workflows efficiently. It's ideal for teams looking to reduce the burden of maintaining their own infrastructure while enhancing governance and productivity. ##### Related content * [Learn dbt](https://learn.getdbt.com) courses * Book [expert-led demos](https://www.getdbt.com/resources/dbt-cloud-demos-with-experts) and insights * Work with the [dbt Labs' Professional Services](https://www.getdbt.com/dbt-labs/services) team to support your data organization and migration. * [How the dbt platform compares with self-hosted dbt](https://www.getdbt.com/product/dbt-core-vs-dbt-cloud) for a detailed comparison of self-hosted dbt and the dbt platform. * Subscribe to the [dbt platform RSS alerts](https://status.getdbt.com/) --- ### Optimize and troubleshoot dbt models on Databricks [Back to guides](https://docs.getdbt.com/guides.md) Databricks dbt platform Intermediate [Menu ]() #### Introduction Building on the [Set up your dbt project with Databricks](https://docs.getdbt.com/guides/set-up-your-databricks-dbt-project.md) guide, we'd like to discuss performance optimization. In this follow-up post, we outline simple strategies to optimize for cost, performance, and simplicity when you architect data pipelines. We’ve encapsulated these strategies in this acronym-framework: * Platform Components * Patterns & Best Practices * Performance Troubleshooting #### Platform Components As you start to develop your dbt projects, one of the first decisions you will make is what kind of backend infrastructure to run your models against. Databricks offers SQL warehouses, All-Purpose Compute, and Jobs Compute, each optimized to workloads they are catered to. Our recommendation is to use Databricks SQL warehouses for all your SQL workloads. SQL warehouses are optimized for SQL workloads when compared to other compute options, additionally, they can scale both vertically to support larger workloads and horizontally to support concurrency. Also, SQL warehouses are easier to manage and provide out-of-the-box features such as query history to help audit and optimize your SQL workloads. Between Serverless, Pro, and Classic SQL Warehouse types that Databricks offers, our standard recommendation for you is to leverage Databricks serverless warehouses. You can explore features of these warehouse types in the [Compare features section](https://www.databricks.com/product/pricing/databricks-sql?_gl=1*2rsmlo*_ga*ZmExYzgzZDAtMWU0Ny00N2YyLWFhYzEtM2RhZTQzNTAyZjZi*_ga_PQSEQ3RZQC*MTY3OTYwMDg0Ni4zNTAuMS4xNjc5NjAyMDMzLjUzLjAuMA..&_ga=2.104593536.1471430337.1679342371-fa1c83d0-1e47-47f2-aac1-3dae43502f6b) on the Databricks pricing page. With serverless warehouses, you greatly decrease spin-up time waiting for the cluster to warm up and scale time when your cluster needs to horizontally scale. This mitigates the need to keep clusters idle as serverless warehouses will spin up quickly when the workload begins and then spin down when the workload is complete. Plus, serverless warehouses leverage our Photon engine out of the box for optimal performance in both ELT and serving workloads. The next step would be to decide how big to make your serverless SQL warehouse. This is not an exact science but these subsections provide you with some quick tips that will drive huge improvements in performance. ##### Sizing your SQL warehouses To select the appropriate size of your SQL warehouse, consider the use case and workload you are running and its corresponding latency requirements. You can select a T-shirt size based on the amount of data and auto-scaling based on concurrency needs. A good rule of thumb to follow is to start with a Medium warehouse and work from there. For large and complex workloads, bigger warehouses are the way to go and that won’t necessarily mean higher costs. This is because larger warehouses take a shorter time to complete a unit of work. For example, if a Small warehouse takes an hour to complete a pipeline, it will only take half an hour with a Medium. This linear trend continues as long as there’s enough work for the warehouse to perform. ##### Provision warehouses by workload Another technique worth implementing is to provision separate SQL warehouses for building dbt pipelines instead of ad hoc, interactive SQL analysis. This is because the query design patterns and compute usage are different for these two types of workloads. Choose T-shirt sizes based on data volumes and SLAs (scale-up principle), and choose auto-scaling based on concurrency requirements (scale-out principle). For larger deployments, this approach could be expanded to map different workload sizes to multiple “pipeline” warehouses, if needed. On the dbt side, take into account the [number of threads you have](https://docs.getdbt.com/docs/local/profiles.yml.md#understanding-threads), meaning how many dbt models you can run in parallel. The higher the thread count, the more compute you will require. ##### Configure auto-stop Because of the ability of serverless warehouses to spin up in a matter of seconds, setting your auto-stop configuration to a lower threshold will not impact SLAs and end-user experience. From the SQL Workspace UI, the default value is 10 minutes and  you can set it to 5 minutes for a lower threshold with the UI. If you would like more custom settings, you can set the threshold to as low as 1 minute with the [API](https://docs.databricks.com/sql/api/sql-endpoints.html#). #### Patterns & Best Practices Now that we have a solid sense of the infrastructure components, we can shift our focus to best practices and design patterns on pipeline development.  We recommend the staging/intermediate/mart approach which is analogous to the medallion architecture bronze/silver/gold approach that’s recommended by Databricks. Let’s dissect each stage further. dbt has guidelines on how you can [structure your dbt project](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md) which you can learn more about. ##### Bronze / Staging Layer: There are a few different options for materializing bronze delta tables on Databricks. In the recommended dbt workflow, you should load your flat files into a table first before using dbt to transform on it. To do so, you can use an EL tool to handle this ingestion. However, we know this isn't always possible so for data sets in cloud storage, we recommend that you either leverage our `COPY INTO` functionality or stage the external table. In terms of the `COPY INTO` approach, you would have a few different options. The first option would be to run the `COPY INTO` logic as a pre-hook before building your silver/intermediate models. The second option would be to invoke the databricks `COPY INTO` macro with `dbt run-operation` and then subsequently execute your model runs. You can see an example implementation of the [COPY INTO macro](https://github.com/databricks/dbt-databricks/blob/main/docs/databricks-copy-into-macro-aws.md) in the dbt-databricks docs. The main benefit of leveraging `COPY INTO` is that it's an incremental operation and it ensures that data is written in Delta format (when we refer to Delta, we are simply referring to the open Parquet tables with a transaction log). If you instead opt to stage an external table, the bronze table retains its raw structure (whether it is CSV, Parquet, JSON, etc.). This would prevent the ability to leverage the performance, reliability, and governance advantages inherent in Delta. Further, external Parquet tables require additional manual work such as running repair operations to ensure new partition metadata is accounted for. Nevertheless, staging external tables could be a feasible option if you are migrating to Databricks from another cloud warehouse system, where you heavily leveraged this functionality. ##### Silver / Intermediate Layer Now that we have our bronze table taken care of, we can proceed with the silver layer. For cost and performance reasons, many customers opt to implement an incremental pipeline approach. The main benefit with this approach is that you process a lot less data when you insert new records into the silver layer, rather than re-create the table each time with all the data from the bronze layer. However it should be noted that by default, [dbt recommends using views and tables](https://docs.getdbt.com/best-practices/materializations/1-guide-overview.md) to start out with and then moving to incremental as you require more performance optimization. dbt has an [incremental model materialization](https://docs.getdbt.com/reference/resource-configs/spark-configs.md#the-merge-strategy) to facilitate this framework. How this works at a high level is that Databricks will create a temp view with a snapshot of data and then merge that snapshot into the silver table. You can customize the time range of the snapshot to suit your specific use case by configuring the `where` conditional in your `is_incremental` logic. The most straightforward implementation is to merge data using a timestamp that’s later than the current max timestamp in the silver table, but there are certainly valid use cases for increasing the temporal range of the source snapshot. While merge should be fairly performant out of the box but if you have particularly tight SLAs, there are some more advanced tuning techniques that you can incorporate into your logic. Let us discuss several examples in further detail. ##### File Compaction Most compute engines work best when file sizes are between 32 MB and 256 MB. In Databricks, we take care of optimal file sizing under the hood with our [auto optimize](https://docs.databricks.com/optimizations/auto-optimize.html) features. Auto optimize consists of two distinct features: auto compaction and optimized writes. In Databricks SQL warehouses, optimized writes are enabled by default. We recommend that you [opt in to auto compaction](https://docs.databricks.com/optimizations/auto-optimize.html#when-to-opt-in-to-auto-compaction). ##### Data skipping Under the hood, Databricks will naturally [cluster data based on when it was ingested](https://www.databricks.com/blog/2022/11/18/introducing-ingestion-time-clustering-dbr-112.html). Since many queries include timestamps in `where` conditionals, this will naturally lead to a large amount of file skipping for enhanced performance. Nevertheless, if you have other high cardinality columns (basically columns with a large amount of distinct values such as id columns) that are frequently used in `join` keys or `where` conditionals, performance can typically be augmented further by leveraging Z-order. The SQL syntax for the Z-Order command is `OPTIMIZE table_name ZORDER BY (col1,col2,col3,etc)`. One caveat to be aware of is that you will rarely want to Z-Order by more than three columns. You will likely want to either run Z-order on run end after your model builds or run Z-Order as a separate scheduled job on a consistent cadence, whether it is daily, weekly, or monthly. ```sql config( materialized='incremental', zorder="column_A" | ["column_A", "column_B"] ) ``` ##### Analyze Table The `ANALYZE TABLE` command ensures that our system has the most up-to-date statistics to select the optimal join plan. You will likely want to either run analyze table posthook after your model builds or run analyze table as a separate scheduled dbt job on a consistent cadence, whether it is daily, weekly, or monthly.  The SQL syntax for this is: ```sql ANALYZE TABLE mytable COMPUTE STATISTICS FOR COLUMNS col1, col2, col3 ``` An important item to clarify is that you will want to prioritize statistics for columns that are frequently used in joins. ##### Vacuum When you delete a record from a Delta table, it is a soft delete. What this means is that the record is deleted from the transaction log and is not included in subsequent queries, but the underlying file still remains in cloud storage. If you want to delete the underlying files as well (whether for reducing storage cost or augmenting performance on merges), you can run a vacuum command. The factor you will want to be very cognizant of is restoring older versions of the table. Let’s say  you vacuum a table to delete all unused files that’s older than 7 days. You won’t be  able to restore versions of the table from over 7 days ago that rely on those deleted  files, so use with caution. If/when you choose to leverage vacuum, you will likely want to run vacuum using the dbt functionality [on-run-end](https://docs.getdbt.com/reference/project-configs/on-run-start-on-run-end.md) after your model builds or run vacuum as a separate scheduled dbt job on a consistent cadence (whether it is daily, weekly, or monthly) using the dbt [run-operation](https://docs.getdbt.com/reference/commands/run-operation.md) command (with the vaccum statement in a macro). ##### Gold / Marts Layer Now onto the most final layer — the gold marts that business stakeholders typically interact with from their preferred BI tool. The considerations here will be fairly similar to the silver layer except that these marts are more likely to handling aggregations. Further, you will likely want to be even more intentional about Z-Ordering these tables as SLAs tend to be lower with these direct stakeholder facing tables. In addition, these tables are well suited for defining [metrics](https://docs.getdbt.com/docs/build/build-metrics-intro.md) on to ensure simplicity and consistency across your key business KPIs! Using the [MetricFlow](https://github.com/dbt-labs/metricflow), you can query the metrics inside of your own dbt project even. With the upcoming Semantic Layer Integration, you can also then query the metrics in any of the partner integrated tools. ##### Filter rows in target and/or source It can be done using `incremental_predicates` like in this example: ```sql {{ config( materialized='incremental', incremental_strategy = 'merge', unique_key = 'id', incremental_predicates = [ "dbt_internal_target.create_at >= '2023-01-01'", "dbt_internal_source.create_at >= '2023-01-01'"], ) }} ``` #### Performance Troubleshooting Performance troubleshooting refers to the process of identifying and resolving issues that impact the performance of your dbt models and overall data pipelines. By improving the speed and performance of your Lakehouse platform, you will be able to process data faster, process large and complex queries more effectively, and provide faster time to market.  Let’s go into detail the three effective strategies that you can implement. ##### SQL warehouse query profile The SQL warehouse query profile is an effective tool found inside the Databricks SQL workspace. It’s used to troubleshoot slow-running queries, optimize query execution plans, and analyze granular metrics to see where compute resources are being spent. The query profile includes these high level capability areas: * Detailed information about the three main components of query execution, which are time spent in tasks, number of rows processed, and memory consumption. * Two types of graphical representations. A tree view to easily spot slow operations at a glance, and a graph view that breaks down how data is transformed across tasks. * Ability to understand mistakes and performance bottlenecks in queries. The three common examples of performance bottlenecks that can be surfaced by the query profile are: ##### Inefficient file pruning By default, Databricks Delta tables collect statistics on the *first 32 columns* defined in your table schema. When transforming data from the Bronze/staging layer to the Silver/intermediate layer, it is advised to reorder your columns to account for these file-level stats and improve overall performance. Move numerical keys and high cardinality query predicates to the left of the 32nd ordinal position, and move strings and complex data types after the 32nd ordinal position of the table. It is worth mentioning that while you can change the default table property to collect statistics on more columns, it will add more overhead as you write files. You may change this default value by using the [table property](https://docs.databricks.com/delta/table-properties.html), `delta.dataSkippingNumIndexedCols`. ##### Full Table Scans The Query Profile provides metrics that allow you to identify the presence of full table scans. Full table scans is a query operation that involves scanning the entire table to retrieve records. It can be a performance issue especially for large tables with billions or trillions of rows. This is because scanning an entire table can be time-consuming and resource-intensive, leading to high memory and CPU usage and slower response times. Table layout techniques such as file compaction and Z-Ordering described in the earlier section of this article will help alleviate this problem. ##### Exploding Joins The concept of *exploding joins* refers to a `join` operation that produces a much larger table result set than either of the input tables used, resulting in a Cartesian product. This performance issue can be determined by enabling the verbose mode setting in the Query Profile, by looking at the number of records produced by a join operator. There are several steps you can take to prevent exploding joins. As a first step, make the join conditions more specific to reduce the number of rows that are being matched. Another step is to utilize data preprocessing techniques such as aggregating, filtering, and performing data sampling before the join operation. These techniques can reduce the size of the input tables and help prevent exploding joins. ##### Materialization Best Practices   Remember that data is stored as files, so the unit of I/O work is a file, not a row. That’s a lot of work if we’re dealing with TBs of data. Therefore we recommend relying on merge strategy as the recommended strategy for the majority of incremental models. Databricks is committed to continuously improving its performance. For example, in Delta and DBSQL, we’ve greatly improved performance of MERGE operations recently with [low-shuffle merge and Photon](https://www.databricks.com/blog/2022/10/17/faster-merge-performance-low-shuffle-merge-and-photon.html). With many future implementations in the pipeline such as deletion vectors for efficient deletes & upserts.Here’s the basic strategies to speed it up: 1. Only read partitions that are important by pushing down filters to scan source and target using filters in *model* and *incremental\_predicates* 2. Only update important rows 3. Improve key lookup by defining only *one* materialized key 4. Only update important columns ##### dbt Discovery API Now you might be wondering, how do you identify opportunities for performance improvement inside of dbt? Well, with each job run, dbt generates metadata on the timing, configuration, and freshness of models in your dbt project. The [dbt Discovery API](https://docs.getdbt.com/docs/dbt-apis/discovery-api.md) is a GraphQL service that supports queries on this metadata, using  the [graphical explorer](https://metadata.cloud.getdbt.com/graphiql) or the endpoint itself. Teams can pipe this data into their data warehouse and analyze it like any other data source in a business intelligence platform. dbt users can also use the data from the [Model Timing tab](https://docs.getdbt.com/docs/deploy/run-visibility.md#model-timing) to visually identify models that take the most time and may require refactoring. ##### dbt Admin API With the [dbt Admin API](https://docs.getdbt.com/docs/dbt-apis/admin-api.md), you can  pull the dbt artifacts from your dbt run,  put the generated `manifest.json` into an S3 bucket, stage it, and model the data using the [dbt artifacts package](https://hub.getdbt.com/brooklyn-data/dbt_artifacts/latest/). That package can help you identify inefficiencies in your dbt models and pinpoint where opportunities for improvement are. ##### Conclusion This builds on the content in [Set up your dbt project with Databricks](https://docs.getdbt.com/guides/set-up-your-databricks-dbt-project.md). We welcome you to try these strategies on our example open source TPC-H implementation and to provide us with thoughts/feedback as you start to incorporate these features into production. Looking forward to your feedback on [#db-databricks-and-spark](https://getdbt.slack.com/archives/CNGCW8HKL) Slack channel! --- ### Post to Microsoft Teams when a job finishes [Back to guides](https://docs.getdbt.com/guides.md) Webhooks Advanced [Menu ]() #### Introduction This guide will show you how to set up an integration between dbt jobs and Microsoft Teams using [dbt Webhooks](https://docs.getdbt.com/docs/deploy/webhooks.md) and Zapier, similar to the [native Slack integration](https://docs.getdbt.com/docs/deploy/job-notifications.md#slack-notifications). Want Microsoft Teams notifications without Zapier? If you only need job status alerts in Microsoft Teams (for example, “job succeeded/failed”) and *don’t* need to process webhook payloads, you can use **Job notifications** instead by sending notifications to a Teams channel email address (External Email). See **Job notifications**: [Job notifications](https://docs.getdbt.com/docs/deploy/job-notifications) When a dbt job finishes running, the integration will: * Receive a webhook notification in Zapier, * Extract the results from the dbt admin API, and * Post a summary to a Microsoft Teams channel. ![Screenshot of a message in MS Teams showing a summary of a run which failed](/assets/images/ms-teams-ui-ab48d824ddaa34c88daeeddbf0291616.png) ##### Prerequisites In order to set up the integration, you should have familiarity with: * [dbt Webhooks](https://docs.getdbt.com/docs/deploy/webhooks.md) * Zapier #### Set up the connection between Zapier and Microsoft Teams * Install the [Zapier app in Microsoft Teams](https://appsource.microsoft.com/en-us/product/office/WA200002044) and [grant Zapier access to your account](https://zapier.com/blog/how-to-automate-microsoft-teams/). **Note**: To receive the message, add the Zapier app to the team's channel during installation. #### Create a new Zap in Zapier Use **Webhooks by Zapier** as the Trigger, and **Catch Raw Hook** as the Event. If you don't intend to [validate the authenticity of your webhook](https://docs.getdbt.com/docs/deploy/webhooks.md#validate-a-webhook) (not recommended!) then you can choose **Catch Hook** instead. Press **Continue**, then copy the webhook URL. ![Screenshot of the Zapier UI, showing the webhook URL ready to be copied](/assets/images/catch-raw-hook-16dd72d8a6bc26284c5fad897f3da646.png) ##### 3. Configure a new webhook in dbt See [Create a webhook subscription](https://docs.getdbt.com/docs/deploy/webhooks.md#create-a-webhook-subscription) for full instructions. Choose either **Run completed** or **Run errored**, but not both, or you'll get double messages when a run fails. Make note of the Webhook Secret Key for later. Once you've tested the endpoint in dbt, go back to Zapier and click **Test Trigger**, which will create a sample webhook body based on the test event dbt sent. The sample body's values are hard-coded and not reflective of your project, but they give Zapier a correctly-shaped object during development. #### Store secrets In the next step, you will need the Webhook Secret Key from the prior step, and a dbt [personal access token](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) or [service account token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md). Zapier allows you to [store secrets](https://help.zapier.com/hc/en-us/articles/8496293271053-Save-and-retrieve-data-from-Zaps), which prevents your keys from being displayed in plaintext in the Zap code. You will be able to access them via the [StoreClient utility](https://help.zapier.com/hc/en-us/articles/8496293969549-Store-data-from-code-steps-with-StoreClient). This guide assumes the names for the secret keys are: `DBT_CLOUD_SERVICE_TOKEN` and `DBT_WEBHOOK_KEY`. If you're using different names, make sure you update all references to it in the sample code. This guide uses a short-lived code action to store the secrets, but you can also use a tool like Postman to interact with the [REST API](https://store.zapier.com/) or create a separate Zap and call the [Set Value Action](https://help.zapier.com/hc/en-us/articles/8496293271053-Save-and-retrieve-data-from-Zaps#3-set-a-value-in-your-store-0-3). ###### a. Create a Storage by Zapier connection If you haven't already got one, go to <https://zapier.com/app/connections/storage> and create a new connection. Remember the UUID secret you generate for later. ###### b. Add a temporary code step Choose **Run Python** as the Event. Run the following code: ```python store = StoreClient('abc123') #replace with your UUID secret store.set('DBT_WEBHOOK_KEY', 'abc123') #replace with webhook secret store.set('DBT_CLOUD_SERVICE_TOKEN', 'abc123') #replace with your dbt API token ``` Test the step. You can delete this Action when the test succeeds. The key will remain stored as long as it is accessed at least once every three months. #### Add a code action Select **Code by Zapier** as the App, and **Run Python** as the Event. In the **Set up action** area, add two items to **Input Data**: `raw_body` and `auth_header`. Map those to the `1. Raw Body` and `1. Headers Http Authorization` fields from the **Catch Raw Hook** step above. ![Screenshot of the Zapier UI, showing the mappings of raw\_body and auth\_header](/assets/images/run-python-40333883c6a20727c02d25224d0e40a4.png) In the **Code** field, paste the following code, replacing `YOUR_SECRET_HERE` with the secret you created when setting up the Storage by Zapier integration. Remember that this is not your dbt secret. The code below will validate the authenticity of the request, extract the run logs for the completed job from the Admin API, and then build a summary message that pulls out any error messages from the end-of-invocation logs created by dbt. ````python import hashlib import hmac import json import re auth_header = input_data['auth_header'] raw_body = input_data['raw_body'] # Access secret credentials secret_store = StoreClient('YOUR_SECRET_HERE') hook_secret = secret_store.get('DBT_WEBHOOK_KEY') api_token = secret_store.get('DBT_CLOUD_SERVICE_TOKEN') # Validate the webhook came from dbt signature = hmac.new(hook_secret.encode('utf-8'), raw_body.encode('utf-8'), hashlib.sha256).hexdigest() if signature != auth_header: raise Exception("Calculated signature doesn't match contents of the Authorization header. This webhook may not have been sent from dbt.") full_body = json.loads(raw_body) hook_data = full_body['data'] # Steps derived from these commands won't have their error details shown inline, as they're messy commands_to_skip_logs = ['dbt source', 'dbt docs'] # When testing, you will want to hardcode run_id and account_id to IDs that exist; the sample webhook won't work. run_id = hook_data['runId'] account_id = full_body['accountId'] # Fetch run info from the dbt Admin API url = f'https://YOUR_ACCESS_URL/api/v2/accounts/{account_id}/runs/{run_id}/?include_related=["run_steps"]' headers = {'Authorization': f'Token {api_token}'} run_data_response = requests.get(url, headers=headers) run_data_response.raise_for_status() run_data_results = run_data_response.json()['data'] # Overall run summary outcome_message = f""" **[{hook_data['runStatus']} for Run #{run_id} on Job \"{hook_data['jobName']}\"]({run_data_results['href']})** **Environment:** {hook_data['environmentName']} | **Trigger:** {hook_data['runReason']} | **Duration:** {run_data_results['duration_humanized']} """ # Step-specific summaries for step in run_data_results['run_steps']: if step['status_humanized'] == 'Success': outcome_message += f""" ✅ {step['name']} ({step['status_humanized']} in {step['duration_humanized']}) """ else: outcome_message += f""" ❌ {step['name']} ({step['status_humanized']} in {step['duration_humanized']}) """ show_logs = not any(cmd in step['name'] for cmd in commands_to_skip_logs) if show_logs: full_log = step['logs'] # Remove timestamp and any colour tags full_log = re.sub('\x1b?\[[0-9]+m[0-9:]*', '', full_log) summary_start = re.search('(?:Completed with \d+ error.* and \d+ warnings?:|Database Error|Compilation Error|Runtime Error)', full_log) line_items = re.findall('(^.*(?:Failure|Error) in .*\n.*\n.*)', full_log, re.MULTILINE) if len(line_items) == 0: relevant_log = f'```{full_log[summary_start.start() if summary_start else 0:]}```' else: relevant_log = summary_start[0] for item in line_items: relevant_log += f'\n```\n{item.strip()}\n```\n' outcome_message += f""" {relevant_log} """ # Zapier looks for the `output` dictionary for use in subsequent steps output = {'outcome_message': outcome_message} ```` #### Add the Microsoft Teams action Select **Microsoft Teams** as the App, and **Send Channel Message** as the Action. In the **Set up action** area, choose the team and channel. Set the **Message Text Format** to **markdown**, then put **2. Outcome Message** from the Run Python in Code by Zapier output into the **Message Text** field. ![Screenshot of the Zapier UI, showing the mappings of prior steps to an MS Teams message](/assets/images/ms-teams-zap-config-998b96ebd7b3535473f5641dac7b4243.png) #### Test and deploy As you have gone through each step, you should have tested the outputs, so you can now try posting a message into your Teams channel. When you're happy with it, remember to ensure that your `run_id` and `account_id` are no longer hardcoded, then publish your Zap. ##### Other notes * If you post to a chat instead of a team channel, you don't need to add the Zapier app to Microsoft Teams. * If you post to a chat instead of a team channel, note that markdown is not supported and you will need to remove the markdown formatting. * If you chose the **Catch Hook** trigger instead of **Catch Raw Hook**, you will need to pass each required property from the webhook as an input instead of running `json.loads()` against the raw body. You will also need to remove the validation code. --- ### Post to Slack with error context when a job fails [Back to guides](https://docs.getdbt.com/guides.md) Webhooks Advanced [Menu ]() #### Introduction This guide will show you how to set up an integration between dbt jobs and Slack using [dbt webhooks](https://docs.getdbt.com/docs/deploy/webhooks.md) and Zapier. It builds on the native [native Slack integration](https://docs.getdbt.com/docs/deploy/job-notifications.md#slack-notifications) by attaching error message details of models and tests in a thread. Note: Because there is not a webhook for Run Cancelled, you may want to keep the standard Slack integration installed to receive those notifications. You could also use the [alternative integration](#alternate-approach) that augments the native integration without replacing it. When a dbt job finishes running, the integration will: * Receive a webhook notification in Zapier * Extract the results from the dbt admin API * Post a brief summary of the run to a Slack channel * Create a threaded message attached to that post which contains any reasons that the job failed ![Screenshot of a message in Slack showing a summary of a run which failed](/assets/images/slack-thread-example-f5cdad43397338d5be7a32eb4117228a.png) ##### Prerequisites In order to set up the integration, you should have familiarity with: * [dbt webhooks](https://docs.getdbt.com/docs/deploy/webhooks.md) * Zapier #### Create a new Zap in Zapier 1. Use **Webhooks by Zapier** as the Trigger, and **Catch Raw Hook** as the Event. If you don't intend to [validate the authenticity of your webhook](https://docs.getdbt.com/docs/deploy/webhooks.md#validate-a-webhook) (not recommended!) then you can choose **Catch Hook** instead. 2. Click **Continue**, then copy the webhook URL. ![Screenshot of the Zapier UI, showing the webhook URL ready to be copied](/assets/images/catch-raw-hook-16dd72d8a6bc26284c5fad897f3da646.png) #### Configure a new webhook in dbt See [Create a webhook subscription](https://docs.getdbt.com/docs/deploy/webhooks.md#create-a-webhook-subscription) for full instructions. Choose **Run completed** as the Event. You can alternatively choose **Run errored**, but you will need to account for the fact that the necessary metadata [might not be available immediately](https://docs.getdbt.com/docs/deploy/webhooks.md#completed-errored-event-difference). Remember the Webhook Secret Key for later. Once you've tested the endpoint in dbt, go back to Zapier and click **Test Trigger**. This creates a sample webhook body based on the test event dbt sent. The sample body's values are hardcoded and not reflective of your project, but they give Zapier a correctly-shaped object during development. #### Store secrets In the next step, you will need the Webhook Secret Key from the prior step, and a dbt [personal access token](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) or [service account token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md). Zapier allows you to [store secrets](https://help.zapier.com/hc/en-us/articles/8496293271053-Save-and-retrieve-data-from-Zaps). This prevents your keys from being displayed as plaintext in the Zap code. You can access them with the [StoreClient utility](https://help.zapier.com/hc/en-us/articles/8496293969549-Store-data-from-code-steps-with-StoreClient). This guide assumes the names for the secret keys are: `DBT_CLOUD_SERVICE_TOKEN` and `DBT_WEBHOOK_KEY`. If you're using different names, make sure you update all references to it in the sample code. This guide uses a short-lived code action to store the secrets, but you can also use a tool like Postman to interact with the [REST API](https://store.zapier.com/) or create a separate Zap and call the [Set Value Action](https://help.zapier.com/hc/en-us/articles/8496293271053-Save-and-retrieve-data-from-Zaps#3-set-a-value-in-your-store-0-3). ###### a. Create a Storage by Zapier connection If you haven't already got one, go to <https://zapier.com/app/connections/storage> and create a new connection. Remember the UUID secret you generate for later. ###### b. Add a temporary code step Choose **Run Python** as the Event. Run the following code: ```python store = StoreClient('abc123') #replace with your UUID secret store.set('DBT_WEBHOOK_KEY', 'abc123') #replace with webhook secret store.set('DBT_CLOUD_SERVICE_TOKEN', 'abc123') #replace with your dbt API token ``` Test the step. You can delete this Action when the test succeeds. The key will remain stored as long as it is accessed at least once every three months. #### Add a code action Select **Code by Zapier** as the App, and **Run Python** as the Event. In the **Set up action** section, add two items to **Input Data**: `raw_body` and `auth_header`. Map those to the `1. Raw Body` and `1. Headers Http Authorization` fields from the previous **Catch Raw Hook** step. ![Screenshot of the Zapier UI, showing the mappings of raw\_body and auth\_header](/assets/images/run-python-40333883c6a20727c02d25224d0e40a4.png) In the **Code** field, paste the following code, replacing `YOUR_SECRET_HERE` with the secret you created when setting up the Storage by Zapier integration. Remember that this is not your dbt secret. This example code validates the authenticity of the request, extracts the run logs for the completed job from the Admin API, and then builds two messages: a summary message containing the outcome of each step and its duration, and a message for inclusion in a thread displaying any error messages extracted from the end-of-invocation logs created by dbt. ````python import hashlib import hmac import json import re auth_header = input_data['auth_header'] raw_body = input_data['raw_body'] # Access secret credentials secret_store = StoreClient('YOUR_SECRET_HERE') hook_secret = secret_store.get('DBT_WEBHOOK_KEY') api_token = secret_store.get('DBT_CLOUD_SERVICE_TOKEN') # Validate the webhook came from dbt signature = hmac.new(hook_secret.encode('utf-8'), raw_body.encode('utf-8'), hashlib.sha256).hexdigest() if signature != auth_header: raise Exception("Calculated signature doesn't match contents of the Authorization header. This webhook may not have been sent from dbt.") full_body = json.loads(raw_body) hook_data = full_body['data'] # Steps derived from these commands won't have their error details shown inline, as they're messy commands_to_skip_logs = ['dbt source', 'dbt docs'] # When testing, you will want to hardcode run_id and account_id to IDs that exist; the sample webhook won't work. run_id = hook_data['runId'] account_id = full_body['accountId'] # Fetch run info from the dbt Admin API url = f'https://YOUR_ACCESS_URL/api/v2/accounts/{account_id}/runs/{run_id}/?include_related=["run_steps"]' headers = {'Authorization': f'Token {api_token}'} run_data_response = requests.get(url, headers=headers) run_data_response.raise_for_status() run_data_results = run_data_response.json()['data'] # Overall run summary step_summary_post = f""" *<{run_data_results['href']}|{hook_data['runStatus']} for Run #{run_id} on Job \"{hook_data['jobName']}\">* *Environment:* {hook_data['environmentName']} | *Trigger:* {hook_data['runReason']} | *Duration:* {run_data_results['duration_humanized']} """ threaded_errors_post = "" # Step-specific summaries for step in run_data_results['run_steps']: if step['status_humanized'] == 'Success': step_summary_post += f""" ✅ {step['name']} ({step['status_humanized']} in {step['duration_humanized']}) """ else: step_summary_post += f""" ❌ {step['name']} ({step['status_humanized']} in {step['duration_humanized']}) """ # Don't try to extract info from steps that don't have well-formed logs show_logs = not any(cmd in step['name'] for cmd in commands_to_skip_logs) if show_logs: full_log = step['logs'] # Remove timestamp and any colour tags full_log = re.sub('\x1b?\[[0-9]+m[0-9:]*', '', full_log) summary_start = re.search('(?:Completed with \d+ error.* and \d+ warnings?:|Database Error|Compilation Error|Runtime Error)', full_log) line_items = re.findall('(^.*(?:Failure|Error) in .*\n.*\n.*)', full_log, re.MULTILINE) if not summary_start: continue threaded_errors_post += f""" *{step['name']}* """ # If there are no line items, the failure wasn't related to dbt nodes, and we want the whole rest of the message. # If there are, then we just want the summary line and then to log out each individual node's error. if len(line_items) == 0: relevant_log = f'```{full_log[summary_start.start():]}```' else: relevant_log = summary_start[0] for item in line_items: relevant_log += f'\n```\n{item.strip()}\n```\n' threaded_errors_post += f""" {relevant_log} """ send_error_thread = len(threaded_errors_post) > 0 # Zapier looks for the `output` dictionary for use in subsequent steps output = {'step_summary_post': step_summary_post, 'send_error_thread': send_error_thread, 'threaded_errors_post': threaded_errors_post} ```` #### Add Slack actions in Zapier Select **Slack** as the App, and **Send Channel Message** as the Action. In the **Action** section, choose which **Channel** to post to. Set the **Message Text** field to **2. Step Summary Post** from the Run Python in Code by Zapier output. Configure the other options as you prefer (for example, **Bot Name** and **Bot Icon**). ![Screenshot of the Zapier UI, showing the mappings of prior steps to a Slack message](/assets/images/parent-slack-config-39e85487efcfb04136c351992ed08cb9.png) Add another step, **Filter**. In the **Filter setup and testing** section, set the **Field** to **2. Send Error Thread** and the **condition** to **(Boolean) Is true**. This prevents the Zap from failing if the job succeeded and you try to send an empty Slack message in the next step. ![Screenshot of the Zapier UI, showing the correctly configured Filter step](/assets/images/filter-config-5a7f7eca78c49d24fd5b8674f23337e3.png) Add another **Send Channel Message in Slack** action. In the **Action** section, choose the same channel as last time, but set the **Message Text** to **2. Threaded Errors Post** from the same Run Python step. Set the **Thread** value to **3. Message Ts**, which is the timestamp of the post created by the first Slack action. This tells Zapier to add this post as a threaded reply to the main message, which prevents the full (potentially long) output from cluttering your channel. ![Screenshot of the Zapier UI, showing the mappings of prior steps to a Slack message](/assets/images/thread-slack-config-9ebe2df87964d97e82c18d80d9ff9ac2.png) #### Test and deploy When you're done testing your Zap, make sure that your `run_id` and `account_id` are no longer hardcoded in the Code step, then publish your Zap. #### Alternately, use a dbt app Slack message to trigger Zapier Instead of using a webhook as your trigger, you can keep the existing dbt app installed in your Slack workspace and use its messages being posted to your channel as the trigger. In this case, you can skip validating the webhook and only need to load the context from the thread. ##### 1. Create a new Zap in Zapier Use **Slack** as the initiating app, and **New Message Posted to Channel** as the Trigger. In the **Trigger** section, select the channel where your Slack alerts are being posted, and set **Trigger for Bot Messages?** to **Yes**. ![Screenshot of the Zapier UI, showing the correctly configured Message trigger step](/assets/images/message-trigger-config-432c82983008423e7914d0c59eab38cd.png) Test your Zap to find an example record. You might need to load additional samples until you get one that relates to a failed job, depending on whether you post all job events to Slack or not. ##### 2. Add a Filter step Add a **Filter** step with the following conditions: * **1. Text contains failed on Job** * **1. User Is Bot Is true** * **1. User Name Exactly matches dbt** ![Screenshot of the Zapier UI, showing the correctly configured Filter step](/assets/images/message-trigger-filter-57c4f8c530e21a72704481619b040a51.png) ##### 3. Extract the run ID Add a **Format** step with the **Event** of **Text**, and the Action **Extract Number**. For the **Input**, select **1. Text**. ![Screenshot of the Zapier UI, showing the Transform step configured to extract a number from the Slack message\'s Text property](/assets/images/extract-number-e9674c26f01614ccfd93b7fdefaab3ed.png) Test your step and validate that the run ID has been correctly extracted. ##### 4. Add a Delay Sometimes dbt posts the message about the run failing before the run's artifacts are available through the API. For this reason, it's recommended to add a brief delay to increase the likelihood that the data is available. On certain plans, Zapier will automatically retry a job that fails from to a 404 error, but its standdown period is longer than is normally necessary so the context will be missing from your thread for longer. A one-minute delay is generally sufficient. ##### 5. Store secrets In the next step, you will need either a dbt [personal access token](https://docs.getdbt.com/docs/dbt-apis/user-tokens.md) or [service account token](https://docs.getdbt.com/docs/dbt-apis/service-tokens.md). Zapier allows you to [store secrets](https://help.zapier.com/hc/en-us/articles/8496293271053-Save-and-retrieve-data-from-Zaps). This prevents your keys from being displayed as plaintext in the Zap code. You can access them with the [StoreClient utility](https://help.zapier.com/hc/en-us/articles/8496293969549-Store-data-from-code-steps-with-StoreClient). This guide assumes the name for the secret key is `DBT_CLOUD_SERVICE_TOKEN`. If you're using a different name, make sure you update all references to it in the sample code. This guide uses a short-lived code action to store the secrets, but you can also use a tool like Postman to interact with the [REST API](https://store.zapier.com/) or create a separate Zap and call the [Set Value Action](https://help.zapier.com/hc/en-us/articles/8496293271053-Save-and-retrieve-data-from-Zaps#3-set-a-value-in-your-store-0-3). ###### a. Create a Storage by Zapier connection If you haven't already got one, go to <https://zapier.com/app/connections/storage> and create a new connection. Remember the UUID secret you generate for later. ###### b. Add a temporary code step Choose **Run Python** as the Event. Run the following code: ```python store = StoreClient('abc123') #replace with your UUID secret store.set('DBT_CLOUD_SERVICE_TOKEN', 'abc123') #replace with your dbt API token ``` Test the step. You can delete this Action when the test succeeds. The key will remain stored as long as it is accessed at least once every three months. ##### 6. Add a Code action Select **Code by Zapier** as the App, and **Run Python** as the Event. This step is very similar to the one described in the main example, but you can skip a lot of the initial validation work. In the **Action** section, add two items to **Input Data**: `run_id` and `account_id`. Map those to the `3. Output` property and your hardcoded dbt Account ID, respectively. ![Screenshot of the Zapier UI, showing the mappings of raw\_body and auth\_header](/assets/images/code-example-alternate-bbb2b5028df008f1b6832e8453215ff4.png) In the **Code** field, paste the following code, replacing `YOUR_SECRET_HERE` with the secret you created when setting up the Storage by Zapier integration. Remember that this is not your dbt secret. This example code extracts the run logs for the completed job from the Admin API, and then builds a message displaying any error messages extracted from the end-of-invocation logs created by dbt (which will be posted in a thread). ````python import re # Access secret credentials secret_store = StoreClient('YOUR_SECRET_HERE') api_token = secret_store.get('DBT_CLOUD_SERVICE_TOKEN') # Steps derived from these commands won't have their error details shown inline, as they're messy commands_to_skip_logs = ['dbt source', 'dbt docs'] run_id = input_data['run_id'] account_id = input_data['account_id'] url = f'https://YOUR_ACCESS_URL/api/v2/accounts/{account_id}/runs/{run_id}/?include_related=["run_steps"]' headers = {'Authorization': f'Token {api_token}'} response = requests.get(url, headers=headers) response.raise_for_status() results = response.json()['data'] threaded_errors_post = "" for step in results['run_steps']: show_logs = not any(cmd in step['name'] for cmd in commands_to_skip_logs) if not show_logs: continue if step['status_humanized'] != 'Success': full_log = step['logs'] # Remove timestamp and any colour tags full_log = re.sub('\x1b?\[[0-9]+m[0-9:]*', '', full_log) summary_start = re.search('(?:Completed with \d+ error.* and \d+ warnings?:|Database Error|Compilation Error|Runtime Error)', full_log) line_items = re.findall('(^.*(?:Failure|Error) in .*\n.*\n.*)', full_log, re.MULTILINE) if not summary_start: continue threaded_errors_post += f""" *{step['name']}* """ # If there are no line items, the failure wasn't related to dbt nodes, and we want the whole rest of the message. # If there are, then we just want the summary line and then to log out each individual node's error. if len(line_items) == 0: relevant_log = f'```{full_log[summary_start.start():]}```' else: relevant_log = summary_start[0] for item in line_items: relevant_log += f'\n```\n{item.strip()}\n```\n' threaded_errors_post += f""" {relevant_log} """ output = {'threaded_errors_post': threaded_errors_post} ```` ##### 7. Add Slack action in Zapier Add a **Send Channel Message in Slack** action. In the **Action** section, set the channel to **1. Channel Id**, which is the channel that the triggering message was posted in. Set the **Message Text** to **5. Threaded Errors Post** from the Run Python step. Set the **Thread** value to **1. Ts**, which is the timestamp of the triggering Slack post. This tells Zapier to add this post as a threaded reply to the main message, which prevents the full (potentially long) output from cluttering your channel. ![Screenshot of the Zapier UI, showing the mappings of prior steps to a Slack message](/assets/images/thread-slack-config-alternate-36df7dedc6e8e5688edd5bfe1439ef2c.png) ##### 8. Test and deploy When you're done testing your Zap, publish it. --- ### Productionize your dbt Databricks project [Back to guides](https://docs.getdbt.com/guides.md) Databricks dbt platform Intermediate [Menu ]() #### Introduction Welcome to the third installment of our comprehensive series on optimizing and deploying your data pipelines using Databricks and dbt. In this guide, we'll dive into delivering these models to end users while incorporating best practices to ensure that your production data remains reliable and timely. ##### Prerequisites If you don't have any of the following requirements, refer to the instructions in the [Set up your dbt project with Databricks](https://docs.getdbt.com/guides/set-up-your-databricks-dbt-project.md) for help meeting these requirements: * You have [Set up your dbt project with Databricks](https://docs.getdbt.com/guides/set-up-your-databricks-dbt-project.md). * You have [optimized your dbt models for peak performance](https://docs.getdbt.com/guides/optimize-dbt-models-on-databricks.md). * You have created two catalogs in Databricks: *dev* and *prod*. * You have created Databricks Service Principal to run your production jobs. * You have at least one [deployment environment](https://docs.getdbt.com/docs/deploy/deploy-environments.md) in dbt. To get started, let's revisit the deployment environment created for your production data. ##### Deployment environments In software engineering, environments play a crucial role in allowing engineers to develop and test code without affecting the end users of their software. Similarly, you can design [data lakehouses](https://www.databricks.com/product/data-lakehouse) with separate environments. The *production* environment includes the relations (schemas, tables, and views) that end users query or use, typically in a BI tool or ML model. In dbt, [environments](https://docs.getdbt.com/docs/dbt-platform-environments.md) come in two flavors: * Deployment — Defines the settings used for executing jobs created within that environment. * Development — Determine the settings used in the Studio IDE for a particular dbt project. Each dbt project can have multiple deployment environments, but only one development environment per user. #### Create and schedule a production job With your deployment environment set up, it's time to create a production job to run in your *prod* environment. To deploy our data transformation workflows, we will utilize [dbt’s built-in job scheduler](https://docs.getdbt.com/docs/deploy/deploy-jobs.md). The job scheduler is designed specifically to streamline your dbt project deployments and runs, ensuring that your data pipelines are easy to create, monitor, and modify efficiently. Leveraging dbt's job scheduler allows data teams to own the entire transformation workflow. You don't need to learn and maintain additional tools for orchestration or rely on another team to schedule code written by your team. This end-to-end ownership simplifies the deployment process and accelerates the delivery of new data products. Let’s [create a job](https://docs.getdbt.com/docs/deploy/deploy-jobs.md#create-and-schedule-jobs) in dbt that will transform data in our Databricks *prod* catalog. 1. Create a new job by clicking **Deploy** in the header, click **Jobs** and then **Create job**. 2. **Name** the job “Daily refresh”. 3. Set the **Environment** to your *production* environment. * This will allow the job to inherit the catalog, schema, credentials, and environment variables defined in [Set up your dbt project with Databricks](https://docs.getdbt.com/guides/set-up-your-databricks-dbt-project.md). 4. Under **Execution Settings** * Check the **Generate docs on run** checkbox to configure the job to automatically generate project docs each time this job runs. This will ensure your documentation stays evergreen as models are added and modified. * Select the **Run on source freshness** checkbox to configure dbt [source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) as the first step of this job. Your sources will need to be configured to [snapshot freshness information](https://docs.getdbt.com/docs/build/sources.md#source-data-freshness) for this to drive meaningful insights. Add the following three **Commands:** * `dbt source freshness` * This will check if any sources are stale. We don’t want to recompute models with data that hasn’t changed since our last run. * `dbt test --models source:*` * This will test the data quality our source data, such as checking making sure ID fields are unique and not null. We don’t want bad data getting into production models. * `dbt build --exclude source:* --fail-fast` * dbt build is more efficient than issuing separate commands for dbt run and dbt test separately because it will run then test each model before continuing. * We are excluding source data because we already tested it in step 2. * The fail-fast flag will make dbt exit immediately if a single resource fails to build. If other models are in-progress when the first model fails, then dbt will terminate the connections for these still-running models. 5. Under **Triggers**, use the toggle to configure your job to [run on a schedule](https://docs.getdbt.com/docs/deploy/deploy-jobs.md#schedule-days). You can enter specific days and timing or create a custom cron schedule. * If you want your dbt job scheduled by another orchestrator, like Databricks Workflows, see the [Advanced Considerations](#advanced-considerations) section below. This is just one example of an all-or-nothing command list designed to minimize wasted computing. The [job command list](https://docs.getdbt.com/docs/deploy/job-commands.md) and [selectors](https://docs.getdbt.com/reference/node-selection/syntax.md) provide a lot of flexibility on how your DAG will execute. You may want to design yours to continue running certain models if others fail. You may want to set up multiple jobs to refresh models at different frequencies. See our [Job Creation Best Practices discourse](https://discourse.getdbt.com/t/job-creation-best-practices-in-dbt-cloud-feat-my-moms-lasagna/2980) for more job design suggestions. After your job is set up and runs successfully, configure your **[project artifacts](https://docs.getdbt.com/docs/deploy/artifacts.md)** to make this job inform your production docs site and data sources dashboard that can be reached from the UI. This will be our main production job to refresh data that will be used by end users. Another job everyone should include in their dbt project is a continuous integration job. #### Add a CI job CI/CD, or Continuous Integration and Continuous Deployment/Delivery, has become a standard practice in software development for rapidly delivering new features and bug fixes while maintaining high quality and stability. dbt enables you to apply these practices to your data transformations. The steps below show how to create a CI test for your dbt project. CD in dbt requires no additional steps, as your jobs will automatically pick up the latest changes from the branch assigned to the environment your job is running in. You may choose to add steps depending on your deployment strategy. If you want to dive deeper into CD options, check out [this blog on adopting CI/CD with dbt](https://www.getdbt.com/blog/adopting-ci-cd-with-dbt-cloud/). dbt allows you to write [data tests](https://docs.getdbt.com/docs/build/data-tests.md) for your data pipeline, which can be run at every step of the process to ensure the stability and correctness of your data transformations. The main places you’ll use your dbt tests are: 1. **Daily runs:** Regularly running tests on your data pipeline helps catch issues caused by bad source data, ensuring the quality of data that reaches your users. 2. **Development**: Running tests during development ensures that your code changes do not break existing assumptions, enabling developers to iterate faster by catching problems immediately after writing code. 3. **CI checks**: Automated CI jobs run and test your pipeline end-to end when a pull request is created, providing confidence to developers, code reviewers, and end users that the proposed changes are reliable and will not cause disruptions or data quality issues Your CI job will ensure that the models build properly and pass any tests applied to them. We recommend creating a separate *test* environment and having a dedicated service principal. This will ensure the temporary schemas created during CI tests are in their own catalog and cannot unintentionally expose data to other users. Repeat the steps in [Set up your dbt project with Databricks](https://docs.getdbt.com/guides/set-up-your-databricks-dbt-project.md) to create your *prod* environment to create a *test* environment. After setup, you should have: * A catalog called *test* * A service principal called *dbt\_test\_sp* * A new dbt environment called *test* that defaults to the *test* catalog and uses the *dbt\_test\_sp* token in the deployment credentials We recommend setting up a dbt CI job. This will decrease the job’s runtime by running and testing only modified models, which also reduces compute spend on the lakehouse. To create a CI job, refer to [Set up CI jobs](https://docs.getdbt.com/docs/deploy/ci-jobs.md) for details. With dbt tests and SlimCI, you can feel confident that your production data will be timely and accurate even while delivering at high velocity. #### Monitor your jobs Keeping a close eye on your dbt jobs is crucial for maintaining a robust and efficient data pipeline. By monitoring job performance and quickly identifying potential issues, you can ensure that your data transformations run smoothly. dbt provides three entry points to monitor the health of your project: run history, deployment monitor, and status tiles. The [run history](https://docs.getdbt.com/docs/deploy/run-visibility.md#run-history) dashboard in dbt provides a detailed view of all your project's job runs, offering various filters to help you focus on specific aspects. This is an excellent tool for developers who want to check recent runs, verify overnight results, or track the progress of running jobs. To access it, select **Run History** from the **Deploy** menu. The deployment monitor in dbt offers a higher-level view of your run history, enabling you to gauge the health of your data pipeline over an extended period of time. This feature includes information on run durations and success rates, allowing you to identify trends in job performance, such as increasing run times or more frequent failures. The deployment monitor also highlights jobs in progress, queued, and recent failures. To access the deployment monitor click on the dbt logo in the top left corner of the dbt UI. [![The Deployment Monitor Shows Job Status Over Time Across Environments](/img/guides/databricks-guides/deployment_monitor_dbx.png?v=2 "The Deployment Monitor Shows Job Status Over Time Across Environments")](#)The Deployment Monitor Shows Job Status Over Time Across Environments By adding [data health tiles](https://docs.getdbt.com/docs/explore/data-tile.md) to your BI dashboards, you can give stakeholders visibility into the health of your data pipeline without leaving their preferred interface. Data tiles instill confidence in your data and help prevent unnecessary inquiries or context switching. To implement dashboard status tiles, you'll need to have dbt docs with [exposures](https://docs.getdbt.com/docs/build/exposures.md) defined. #### Set up notifications Setting up [notifications](https://docs.getdbt.com/docs/deploy/job-notifications.md) in dbt allows you to receive alerts via email or a Slack channel whenever a run ends. This ensures that the appropriate teams are notified and can take action promptly when jobs fail or are canceled. To set up notifications: 1. Navigate to your dbt project settings. 2. Select the **Notifications** tab. 3. Choose the desired notification type (Email or Slack) and configure the relevant settings. If you require notifications through other means than email or Slack, you can use dbt's outbound [webhooks](https://docs.getdbt.com/docs/deploy/webhooks.md) feature to relay job events to other tools. Webhooks enable you to integrate dbt with a wide range of SaaS applications, extending your pipeline’s automation into other systems. #### Troubleshooting When a disruption occurs in your production pipeline, it's essential to know how to troubleshoot issues effectively to minimize downtime and maintain a high degree of trust with your stakeholders. The five key steps for troubleshooting dbt issues are: 1. Read the error message: dbt error messages usually indicate the error type and the file where the issue occurred. 2. Inspect the problematic file and look for an immediate fix. 3. Isolate the problem by running one model at a time in the Studio IDE or undoing the code that caused the issue. 4. Check for problems in compiled files and logs. Consult the [Debugging errors documentation](https://docs.getdbt.com/guides/debug-errors.md) for a comprehensive list of error types and diagnostic methods. To troubleshoot issues with a dbt job, navigate to the "Deploy > Run History" tab in your dbt project and select the failed run. Then, expand the run steps to view [console and debug logs](https://docs.getdbt.com/docs/deploy/run-visibility.md#access-logs) to review the detailed log messages. To obtain additional information, open the Artifacts tab and download the compiled files associated with the run. If your jobs are taking longer than expected, use the [model timing](https://docs.getdbt.com/docs/deploy/run-visibility.md#model-timing) dashboard to identify bottlenecks in your pipeline. Analyzing the time taken for each model execution helps you pinpoint the slowest components and optimize them for better performance. The Databricks [Query History](https://docs.databricks.com/sql/admin/query-history.html) lets you inspect granular details such as time spent in each task, rows returned, I/O performance, and execution plan. For more on performance tuning, see our guide on [How to Optimize and Troubleshoot dbt Models on Databricks](https://docs.getdbt.com/guides/optimize-dbt-models-on-databricks.md). #### Advanced considerations As you become more experienced with dbt and Databricks, you might want to explore advanced techniques to further enhance your data pipeline and improve the way you manage your data transformations. The topics in this section are not requirements but will help you harden your production environment for greater security, efficiency, and accessibility. ##### Refreshing your data with Databricks Workflows The dbt job scheduler offers several ways to trigger your jobs. If your dbt transformations are just one step of a larger orchestration workflow, use the dbt API to trigger your job from Databricks Workflows. This is a common pattern for analytics use cases that want to minimize latency between ingesting bronze data into the lakehouse with a notebook, transforming that data into gold tables with dbt, and refreshing a dashboard. It is also useful for data science teams who use dbt for feature extraction before using the updated feature store to train and register machine learning models with MLflow. The API enables integration between your dbt jobs and the Databricks workflow, ensuring that your data transformations are effectively managed within the broader context of your data processing pipeline. Inserting dbt jobs into a Databricks Workflows allows you to chain together external tasks while still leveraging these benefits of dbt: * UI Context: The dbt UI enables you to define the job within the context of your dbt environments, making it easier to create and manage relevant configs. * Logs and Run History: Accessing logs and run history becomes more convenient when using dbt. * Monitoring and Notification Features: dbt comes equipped with monitoring and notification features like the ones described above that can help you stay informed about the status and performance of your jobs. To trigger your dbt job from Databricks, follow the instructions in our [Databricks Workflows to run dbt jobs guide](https://docs.getdbt.com/guides/databricks-workflows.md). #### Data masking Our [Best Practices for dbt and Unity Catalog](https://docs.getdbt.com/best-practices/dbt-unity-catalog-best-practices.md) guide recommends using separate catalogs *dev* and *prod* for development and deployment environments, with Unity Catalog and dbt handling configurations and permissions for environment isolation. Ensuring security while maintaining efficiency in your development and deployment environments is crucial. Additional security measures may be necessary to protect sensitive data, such as personally identifiable information (PII). Databricks leverages [Dynamic Views](https://docs.databricks.com/data-governance/unity-catalog/create-views.html#create-a-dynamic-view) to enable data masking based on group membership. Because views in Unity Catalog use Spark SQL, you can implement advanced data masking by using more complex SQL expressions and regular expressions. You can now also apply fine grained access controls like row filters in preview and column masks in preview on tables in Databricks Unity Catalog, which will be the recommended approach to protect sensitive data once this goes GA. Additionally, in the near term, Databricks Unity Catalog will also enable Attribute Based Access Control natively, which will make protecting sensitive data at scale simpler. To implement data masking in a dbt model, ensure the model materialization configuration is set to view. Next, add a case statement using the is\_account\_group\_member function to identify groups permitted to view plain text values. Then, use regex to mask data for all other users. For example: ```sql CASE WHEN is_account_group_member('auditors') THEN email ELSE regexp_extract(email, '^.*@(.*)$', 1) END ``` It is recommended not to grant users the ability to read tables and views referenced in the dynamic view. Instead, assign your dbt sources to dynamic views rather than raw data, allowing developers to run end-to-end builds and source freshness commands securely. Using the same sources for development and deployment environments enables testing with the same volumes and frequency you will see in production. However, this may cause development runs to take longer than necessary. To address this issue, consider using the Jinja variable target.name to [limit data when working in the development environment](https://docs.getdbt.com/reference/dbt-jinja-functions/target.md#use-targetname-to-limit-data-in-dev). #### Pairing dbt Docs and Unity Catalog Though there are similarities between dbt docs and Databricks Unity Catalog, they are ultimately used for different purposes and complement each other well. By combining their strengths, you can provide your organization with a robust and user-friendly data management ecosystem. dbt docs is a documentation site generated from your dbt project that provides an interface for developers and non-technical stakeholders to understand the data lineage and business logic applied to transformations without requiring full access to dbt or Databricks. It gives you additional options on how you can organize and search for your data. You can automatically [build and view your dbt docs using dbt](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) to keep the documentation evergreen. Unity Catalog is a unified governance solution for your lakehouse. It provides a data explorer that can be used for discovery of datasets that have not been defined in dbt. The data explorer also captures [column-level lineage](https://docs.databricks.com/data-governance/unity-catalog/data-lineage.html#capture-and-explore-lineage),  when you need to trace the lineage of a specific column. To get the most out of both tools, you can use the [persist docs config](https://docs.getdbt.com/reference/resource-configs/persist_docs.md) to push table and column descriptions written in dbt into Unity Catalog, making the information easily accessible to both tools' users. Keeping the descriptions in dbt ensures they are version controlled and can be reproduced after a table is dropped. ##### Related docs * [Advanced Deployment course](https://learn.getdbt.com/courses/advanced-deployment) if you want a deeper dive into these topics * [Autoscaling CI: The intelligent Slim CI](https://docs.getdbt.com/docs/deploy/continuous-integration.md) * [Trigger a dbt Job in your automated workflow with Python](https://discourse.getdbt.com/t/triggering-a-dbt-cloud-job-in-your-automated-workflow-with-python/2573) * [Databricks + dbt Quickstart Guide](https://docs.getdbt.com/guides/databricks.md) * Reach out to your Databricks account team to get access to preview features on Databricks. --- ### Quickstart for dbt and Amazon Athena [Back to guides](https://docs.getdbt.com/guides.md) Amazon Athena dbt platform Quickstart Beginner [Menu ]() #### Introduction In this quickstart guide, you'll learn how to use dbt with Amazon Athena. It will show you how to: * Create an S3 bucket for Athena query results. * Create an Athena database. * Access sample data in a public dataset. * Connect dbt to Amazon Athena. * Take a sample query and turn it into a model in your dbt project. A model in dbt is a select statement. * Add tests to your models. * Document your models. * Schedule a job to run. Videos for you You can check out [dbt Fundamentals](https://learn.getdbt.com/courses/dbt-fundamentals) for free if you're interested in course learning with videos. ##### Prerequisites​ * You have a [dbt account](https://www.getdbt.com/signup/). * You have an [AWS account](https://aws.amazon.com/). * You have set up [Amazon Athena](https://docs.aws.amazon.com/athena/latest/ug/getting-started.html). ##### Related content * Learn more with [dbt Learn courses](https://learn.getdbt.com) * [CI jobs](https://docs.getdbt.com/docs/deploy/continuous-integration.md) * [Deploy jobs](https://docs.getdbt.com/docs/deploy/deploy-jobs.md) * [Job notifications](https://docs.getdbt.com/docs/deploy/job-notifications.md) * [Source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) #### Getting started For the following guide you can use an existing S3 bucket or [create a new one](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html). Download the following CSV files (the Jaffle Shop sample data) and upload them to your S3 bucket: * [jaffle\_shop\_customers.csv](https://dbt-tutorial-public.s3-us-west-2.amazonaws.com/jaffle_shop_customers.csv) * [jaffle\_shop\_orders.csv](https://dbt-tutorial-public.s3-us-west-2.amazonaws.com/jaffle_shop_orders.csv) * [stripe\_payments.csv](https://dbt-tutorial-public.s3-us-west-2.amazonaws.com/stripe_payments.csv) #### Configure Amazon Athena 1. Log into your AWS account and navigate to the **Athena console**. * If this is your first time in the Athena console (in your current AWS Region), click **Explore the query editor** to open the query editor. Otherwise, Athena opens automatically in the query editor. 2. Open **Settings** and find the **Location of query result box** field. 1. Enter the path of the S3 bucket (prefix it with `s3://`). 2. Navigate to **Browse S3**, select the S3 bucket you created, and click **Choose**. 3. **Save** these settings. 4. In the **query editor**, create a database by running `create database YOUR_DATABASE_NAME`. 5. To make the database you created the one you `write` into, select it from the **Database** list on the left side menu. 6. Access the Jaffle Shop data in the S3 bucket using one of these options: 1. Manually create the tables. 2. Create a glue crawler to recreate the data as external tables (recommended). 7. Once the tables have been created, you will able to `SELECT` from them. #### Set up security access to Athena To setup the security access for Athena, determine which access method you want to use: * Obtain `aws_access_key_id` and `aws_secret_access_key` (recommended) * Obtain an **AWS credentials** file. ##### AWS access key (recommended) To obtain your `aws_access_key_id` and `aws_secret_access_key`: 1. Open the **AWS Console**. 2. Click on your **username** near the top right and click **Security Credentials**. 3. Click on **Users** in the sidebar. 4. Click on your **username** (or the name of the user for whom to create the key). 5. Click on the **Security Credentials** tab. 6. Click **Create Access Key**. 7. Click **Show User Security Credentials** and Save the `aws_access_key_id` and `aws_secret_access_key` for a future step. ##### AWS credentials file To obtain your AWS credentials file: 1. Follow the instructions for [configuring the credentials file](https://docs.aws.amazon.com/cli/v1/userguide/cli-configure-files.html) usin the AWS CLI 2. Locate the `~/.aws/credentials` file on your computer 1. Windows: `%USERPROFILE%\.aws\credentials` 2. Mac/Linux: `~/.aws/credentials` Retrieve the `aws_access_key_id` and `aws_secret_access_key` from the `~/.aws/credentials` file for a future step. #### Configure the connection in dbt To configure the Athena connection in dbt: 1. Click your **account name** on the left-side menu and click **Account settings**. 2. Click **Connections** and click **New connection**. 3. Click **Athena** and fill out the required fields (and any optional fields). 1. **AWS region name** — The AWS region of your environment. 2. **Database (catalog)** — Enter the database name created in earlier steps (lowercase only). 3. **AWS S3 staging directory** — Enter the S3 bucket created in earlier steps. 4. Click **Save** ##### Configure your environment To configure the Athena credentials in your environment: 1. Click **Deploy** on the left-side menu and click **Environments**. 2. Click **Create environment** and fill out the **General settings**. * Your **dbt version** must be on a [release track](https://docs.getdbt.com/docs/dbt-versions/dbt-release-tracks.md) to use the Athena connection. 3. Select the Athena connection from the **Connection** dropdown. 4. Fill out the `aws_access_key` and `aws_access_id` recorded in previous steps, as well as the `Schema` to write to. 5. Click **Test connection** and once it succeeds, **Save** the environment. Repeat the process to create a [development environment](https://docs.getdbt.com/docs/dbt-platform-environments.md#types-of-environments). #### Set up a dbt managed repository When you develop in dbt, you can leverage [Git](https://docs.getdbt.com/docs/platform/git/git-version-control.md) to version control your code. To connect to a repository, you can either set up a dbt-hosted [managed repository](https://docs.getdbt.com/docs/platform/git/managed-repository.md) or directly connect to a [supported git provider](https://docs.getdbt.com/docs/platform/git/connect-github.md). Managed repositories are a great way to trial dbt without needing to create a new repository. In the long run, it's better to connect to a supported git provider to use features like automation and [continuous integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md). To set up a managed repository: 1. Under "Setup a repository", select **Managed**. 2. Type a name for your repo such as `bbaggins-dbt-quickstart` 3. Click **Create**. It will take a few seconds for your repository to be created and imported. 4. Once you see the "Successfully imported repository," click **Continue**. #### Initialize your dbt project​ and start developing Now that you have a repository configured, you can initialize your project and start development in dbt: 1. Click **Start developing in the Studio IDE**. It might take a few minutes for your project to spin up for the first time as it establishes your git connection, clones your repo, and tests the connection to the warehouse. 2. Above the file tree to the left, click **Initialize dbt project**. This builds out your folder structure with example models. 3. Make your initial commit by clicking **Commit and sync**. Use the commit message `initial commit` and click **Commit**. This creates the first commit to your managed repo and allows you to open a branch where you can add new dbt code. 4. You can now directly query data from your warehouse and execute `dbt run`. You can try this out now: * Click **+ Create new file**, add this query to the new file, and click **Save as** to save the new file: ```sql select * from jaffle_shop.customers ``` * In the command line bar at the bottom, enter `dbt run` and click **Enter**. You should see a `dbt run succeeded` message. #### Build your first model You have two options for working with files in the Studio IDE: * Create a new branch (recommended) — Create a new branch to edit and commit your changes. Navigate to **Version Control** on the left sidebar and click **Create branch**. * Edit in the protected primary branch — If you prefer to edit, format, or lint files and execute dbt commands directly in your primary git branch. The Studio IDE prevents commits to the protected branch, so you will be prompted to commit your changes to a new branch. Name the new branch `add-customers-model`. 1. Click the **...** next to the `models` directory, then select **Create file**. 2. Name the file `customers.sql`, then click **Create**. 3. Copy the following query into the file and click **Save**. ```sql with customers as ( select id as customer_id, first_name, last_name from jaffle_shop.customers ), orders as ( select id as order_id, user_id as customer_id, order_date, status from jaffle_shop.orders ), customer_orders as ( select customer_id, min(order_date) as first_order_date, max(order_date) as most_recent_order_date, count(order_id) as number_of_orders from orders group by 1 ), final as ( select customers.customer_id, customers.first_name, customers.last_name, customer_orders.first_order_date, customer_orders.most_recent_order_date, coalesce(customer_orders.number_of_orders, 0) as number_of_orders from customers left join customer_orders using (customer_id) ) select * from final ``` 4. Enter `dbt run` in the command prompt at the bottom of the screen. You should get a successful run and see the three models. Later, you can connect your business intelligence (BI) tools to these views and tables so they only read cleaned up data rather than raw data in your BI tool. ###### FAQs How can I see the SQL that dbt is running? To check out the SQL that dbt is running, you can look in: * dbt: * Within the run output, click on a model name, and then select "Details" * dbt v1: * The `target/compiled/` directory for compiled `select` statements * The `target/run/` directory for compiled `create` statements * The `logs/dbt.log` file for verbose logging. How did dbt choose which schema to build my models in? By default, dbt builds models in your target schema. To change your target schema: * If you're developing in **dbt**, these are set for each user when you first use a development environment. * If you're developing with **dbt v1**, this is the `schema:` parameter in your `profiles.yml` file. If you wish to split your models across multiple schemas, check out the docs on [using custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md). Note: on BigQuery, `dataset` is used interchangeably with `schema`. Do I need to create my target schema before running dbt? Nope! dbt will check if the schema exists when it runs. If the schema does not exist, dbt will create it for you. If I rerun dbt, will there be any downtime as models are rebuilt? Nope! The SQL that dbt generates behind the scenes ensures that any relations are replaced atomically (i.e. your business users won't experience any downtime). The implementation of this varies on each warehouse, check out the [logs](https://docs.getdbt.com/faqs/Runs/checking-logs.md) to see the SQL dbt is executing. What happens if the SQL in my query is bad or I get a database error? If there's a mistake in your SQL, dbt will return the error that your database returns. ```shell $ dbt run --select customers Running with dbt=1.9.0 Found 3 models, 9 tests, 0 snapshots, 0 analyses, 133 macros, 0 operations, 0 seed files, 0 sources 14:04:12 | Concurrency: 1 threads (target='dev') 14:04:12 | 14:04:12 | 1 of 1 START view model dbt_alice.customers.......................... [RUN] 14:04:13 | 1 of 1 ERROR creating view model dbt_alice.customers................. [ERROR in 0.81s] 14:04:13 | 14:04:13 | Finished running 1 view model in 1.68s. Completed with 1 error and 0 warnings: Database Error in model customers (models/customers.sql) Syntax error: Expected ")" but got identifier `your-info-12345` at [13:15] compiled SQL at target/run/jaffle_shop/customers.sql Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1 ``` Any models downstream of this model will also be skipped. Use the error message and the [compiled SQL](https://docs.getdbt.com/faqs/Runs/checking-logs.md) to debug any errors. #### Change the way your model is materialized One of the most powerful features of dbt is that you can change the way a model is materialized in your warehouse, simply by changing a configuration value. You can change things between tables and views by changing a keyword rather than writing the data definition language (DDL) to do this behind the scenes. By default, everything gets created as a view. You can override that at the directory level so everything in that directory will materialize to a different materialization. 1. Edit your `dbt_project.yml` file. * Update your project `name` to: dbt\_project.yml ```yaml name: 'jaffle_shop' ``` * Configure `jaffle_shop` so everything in it will be materialized as a table; and configure `example` so everything in it will be materialized as a view. Update your `models` config in the project YAML file to: dbt\_project.yml ```yaml models: jaffle_shop: +materialized: table example: +materialized: view ``` * Click **Save**. 2. Enter the `dbt run` command. Your `customers` model should now be built as a table! info To do this, dbt had to first run a `drop view` statement (or API call on BigQuery), then a `create table as` statement. 3. Edit `models/customers.sql` to override the `dbt_project.yml` for the `customers` model only by adding the following snippet to the top, and click **Save**: models/customers.sql ```sql {{ config( materialized='view' ) }} with customers as ( select id as customer_id ... ) ``` 4. Enter the `dbt run` command. Your model, `customers`, should now build as a view. * BigQuery users need to run `dbt run --full-refresh` instead of `dbt run` to full apply materialization changes. 5. Enter the `dbt run --full-refresh` command for this to take effect in your warehouse. ##### FAQs What materializations are available in dbt? dbt ships with five built-in materializations: `view`, `table`, `incremental`, `ephemeral`, and `materialized_view`. Check out the documentation on [materializations](https://docs.getdbt.com/docs/build/materializations.md) for more information on each of these options. You can also create your own [custom materializations](https://docs.getdbt.com/guides/create-new-materializations.md). This is an advanced feature of dbt. Which materialization should I use for my model? Start out with views, and then change models to tables when required for performance reasons (i.e. downstream queries have slowed). Check out the [docs on materializations](https://docs.getdbt.com/docs/build/materializations.md) for advice on when to use each materialization. What model configurations exist? You can also configure: * [tags](https://docs.getdbt.com/reference/resource-configs/tags.md) to support easy categorization and graph selection * [custom schemas](https://docs.getdbt.com/reference/resource-properties/schema.md) to split your models across multiple schemas * [aliases](https://docs.getdbt.com/reference/resource-configs/alias.md) if your view/table name should differ from the filename * Snippets of SQL to run at the start or end of a model, known as [hooks](https://docs.getdbt.com/docs/build/hooks-operations.md) * Warehouse-specific configurations for performance (e.g. `sort` and `dist` keys on Redshift, `partitions` on BigQuery) Check out the docs on [model configurations](https://docs.getdbt.com/reference/model-configs.md) to learn more. #### Delete the example models You can now delete the files that dbt created when you initialized the project: 1. Delete the `models/example/` directory. 2. Delete the `example:` key from your `dbt_project.yml` file, and any configurations that are listed under it. dbt\_project.yml ```yaml # before models: jaffle_shop: +materialized: table example: +materialized: view ``` dbt\_project.yml ```yaml # after models: jaffle_shop: +materialized: table ``` 3. Save your changes. ###### FAQs How do I remove deleted models from my data warehouse? If you delete a model from your dbt project, dbt does not automatically drop the relation from your schema. This means that you can end up with extra objects in schemas that dbt creates, which can be confusing to other users. (This can also happen when you switch a model from being a view or table, to ephemeral) When you remove models from your dbt project, you should manually drop the related relations from your schema. I got an "unused model configurations" error message, what does this mean? You might have forgotten to nest your configurations under your project name, or you might be trying to apply configurations to a directory that doesn't exist. Check out this [article](https://discourse.getdbt.com/t/faq-i-got-an-unused-model-configurations-error-message-what-does-this-mean/112) to understand more. #### Build models on top of other models As a best practice in SQL, you should separate logic that cleans up your data from logic that transforms your data. You have already started doing this in the existing query by using common table expressions (CTEs). Now you can experiment by separating the logic out into separate models and using the [ref](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) function to build models on top of other models: [![The DAG we want for our dbt project](/img/dbt-dag.png?v=2 "The DAG we want for our dbt project")](#)The DAG we want for our dbt project 1. Create a new SQL file, `models/stg_customers.sql`, with the SQL from the `customers` CTE in our original query. 2. Create a second new SQL file, `models/stg_orders.sql`, with the SQL from the `orders` CTE in our original query. models/stg\_customers.sql ```sql select id as customer_id, first_name, last_name from jaffle_shop.customers ``` models/stg\_orders.sql ```sql select id as order_id, user_id as customer_id, order_date, status from jaffle_shop.orders ``` 3. Edit the SQL in your `models/customers.sql` file as follows: models/customers.sql ```sql with customers as ( select * from {{ ref('stg_customers') }} ), orders as ( select * from {{ ref('stg_orders') }} ), customer_orders as ( select customer_id, min(order_date) as first_order_date, max(order_date) as most_recent_order_date, count(order_id) as number_of_orders from orders group by 1 ), final as ( select customers.customer_id, customers.first_name, customers.last_name, customer_orders.first_order_date, customer_orders.most_recent_order_date, coalesce(customer_orders.number_of_orders, 0) as number_of_orders from customers left join customer_orders using (customer_id) ) select * from final ``` 4. Execute `dbt run`. This time, when you performed a `dbt run`, separate views/tables were created for `stg_customers`, `stg_orders` and `customers`. dbt inferred the order to run these models. Because `customers` depends on `stg_customers` and `stg_orders`, dbt builds `customers` last. You do not need to explicitly define these dependencies. ###### FAQs How do I run one model at a time? To run one model, use the `--select` flag (or `-s` flag), followed by the name of the model: ```shell $ dbt run --select customers ``` Check out the [model selection syntax documentation](https://docs.getdbt.com/reference/node-selection/syntax.md) for more operators and examples. Do ref-able resource names need to be unique? Within one project: yes! To build dependencies between resources (such as models, seeds, and snapshots), you need to use the `ref` function, and pass in the resource name as an argument. dbt uses that resource name to uniquely resolve the `ref` to a specific resource. As a result, these resource names need to be unique, *even if they are in distinct folders*. A resource in one project can have the same name as a resource in another project (installed as a dependency). dbt uses the project name to uniquely identify each resource. We call this "namespacing." If you `ref` a resource with a duplicated name, it will resolve to the resource within the same namespace (package or project), or raise an error because of an ambiguous reference. Use [two-argument `ref`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models) to disambiguate references by specifying the namespace. Those resource will still need to land in distinct locations in the data warehouse. Read the docs on [custom aliases](https://docs.getdbt.com/docs/build/custom-aliases.md) and [custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md) for details on how to achieve this. As I create more models, how should I keep my project organized? What should I name my models? There's no one best way to structure a project! Every organization is unique. If you're just getting started, check out how we (dbt Labs) [structure our dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md). #### Add tests to your models Adding [data tests](https://docs.getdbt.com/docs/build/data-tests.md) to a project helps validate that your models are working correctly. To add data tests to your project: 1. Create a new YAML file in the `models` directory, named `models/schema.yml` 2. Add the following contents to the file: models/schema.yml ```yaml version: 2 models: - name: customers columns: - name: customer_id data_tests: - unique - not_null - name: stg_customers columns: - name: customer_id data_tests: - unique - not_null - name: stg_orders columns: - name: order_id data_tests: - unique - not_null - name: status data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['placed', 'shipped', 'completed', 'return_pending', 'returned'] - name: customer_id data_tests: - not_null - relationships: arguments: to: ref('stg_customers') field: customer_id ``` 3. Run `dbt test`, and confirm that all your tests passed. When you run `dbt test`, dbt iterates through your YAML files, and constructs a query for each test. Each query will return the number of records that fail the test. If this number is 0, then the test is successful. ###### FAQs What tests are available for me to use in dbt? Can I add my own custom tests? Out of the box, dbt ships with the following data tests: * `unique` * `not_null` * `accepted_values` * `relationships` (for example, referential integrity) You can also write your own [custom generic tests](https://docs.getdbt.com/docs/build/data-tests.md#generic-data-tests). Some additional generic tests have been open-sourced in the [dbt-utils package](https://github.com/dbt-labs/dbt-utils#generic-tests). Check out the docs on [packages](https://docs.getdbt.com/docs/build/packages.md) to learn how to make these tests available in your project. How do I test one model at a time? Running tests on one model looks very similar to running a model: use the `--select` flag (or `-s` flag), followed by the name of the model: ```shell dbt test --select customers ``` Check out the [model selection syntax documentation](https://docs.getdbt.com/reference/node-selection/syntax.md) for full syntax, and [test selection examples](https://docs.getdbt.com/reference/node-selection/test-selection-examples.md) in particular. One of my tests failed, how can I debug it? To debug a failing test, find the SQL that dbt ran by: * dbt: * Within the test output, click on the failed test, and then select "Details". * dbt v1: * Open the file path returned as part of the error message. * Navigate to the `target/compiled/schema_tests` directory for all compiled test queries. Copy the SQL into a query editor (in dbt, you can paste it into a new `Statement`), and run the query to find the records that failed. Does my test file need to be named \`schema.yml\`? No! You can name this file whatever you want (including `whatever_you_want.yml`), so long as: * The file is in your `models/` directory¹ * The file has `.yml` extension Check out the [docs](https://docs.getdbt.com/reference/configs-and-properties.md) for more information. ¹If you're declaring properties for seeds, snapshots, or macros, you can also place this file in the related directory — `seeds/`, `snapshots/` and `macros/` respectively. Why do model and source YAML files always start with \`version: 2\`? Once upon a time, the structure of these `.yml` files was very different (s/o to anyone who was using dbt back then!). Adding `version: 2` allowed us to make this structure more extensible. From [dbt v1.5](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.5.md#quick-hits>), the top-level `version:` key is optional in all resource YAML files. If present, only `version: 2` is supported. Also starting in v1.5, both the [`config-version: 2`](https://docs.getdbt.com/reference/project-configs/config-version.md) and the top-level `version:` key in the `dbt_project.yml` are optional. Resource YAML files do not currently require this config. We only support `version: 2` if it's specified. Although we do not expect to update YAML files to `version: 3` soon, having this config will make it easier for us to introduce new structures in the future What data tests should I add to my project? We recommend that every model has a data test on a primary key, that is, a column that is `unique` and `not_null`. We also recommend that you test any assumptions on your source data. For example, if you believe that your payments can only be one of three payment methods, you should test that assumption regularly — a new payment method may introduce logic errors in your SQL. In advanced dbt projects, we recommend using [sources](https://docs.getdbt.com/docs/build/sources.md) and running these source data-integrity tests against the sources rather than models. When should I run my data tests? You should run your data tests whenever you are writing new code (to ensure you haven't broken any existing models by changing SQL), and whenever you run your transformations in production (to ensure that your assumptions about your source data are still valid). #### Document your models Adding [documentation](https://docs.getdbt.com/docs/build/documentation.md) to your project allows you to describe your models in rich detail, and share that information with your team. Here, we're going to add some basic documentation to our project. Update your `models/schema.yml` file to include some descriptions, such as those below. models/schema.yml ```yaml version: 2 models: - name: customers description: One record per customer columns: - name: customer_id description: Primary key data_tests: - unique - not_null - name: first_order_date description: NULL when a customer has not yet placed an order. - name: stg_customers description: This model cleans up customer data columns: - name: customer_id description: Primary key data_tests: - unique - not_null - name: stg_orders description: This model cleans up order data columns: - name: order_id description: Primary key data_tests: - unique - not_null - name: status data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['placed', 'shipped', 'completed', 'return_pending', 'returned'] - name: customer_id data_tests: - not_null - relationships: arguments: to: ref('stg_customers') field: customer_id ``` ##### View in Catalog [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) provides powerful tools to interact with your dbt projects, including documentation: 1. Run one of the following commands: * `dbt docs generate` if you're on dbt v1 * `dbt build` if you're on dbt v2 2. Click **Catalog** in the navigation menu to launch Catalog. 3. Catalog reflects **Production** by default. If your account has additional environments (for example, **Staging**), you can select them from the environment dropdown. [![Select an environment in Catalog.](/img/docs/collaborate/dbt-explorer/catalog-nav-dropdown.png?v=2 "Select an environment in Catalog.")](#)Select an environment in Catalog. 4. Select your project from the file tree. 5. Use the search bar or browse the resource list to find the `customers` model. 6. Click the model to view its details, including the descriptions you added. [![View your model's documentation and lineage in Catalog.](/img/docs/collaborate/dbt-explorer/example-model-details.png?v=2 "View your model's documentation and lineage in Catalog.")](#)View your model's documentation and lineage in Catalog. Catalog displays your model's description, column documentation, data tests, and lineage graph. You can also see which columns are missing documentation and track test coverage across your project. ##### View in Studio IDE You can view docs directly from the IDE if you're on **v1 Latest** or another version of dbt v1. Keep in mind that this is a legacy view and doesn't offer the same level of interactivity as Catalog. 1. In the IDE, run `dbt docs generate`. 2. From the navigation bar, click the **View docs** icon located to the right of the **branch name**. [![The View docs icon in the Studio IDE.](/img/docs/collaborate/dbt-explorer/docs-icon.png?v=2 "The View docs icon in the Studio IDE.")](#)The View docs icon in the Studio IDE. 3. From **Projects**, select your project name and expand the folders. 4. Click **models** > **marts** > **customers**. [![View your model's documentation in the legacy docs view.](/img/docs/collaborate/dbt-explorer/legacy-docs-view.png?v=2 "View your model's documentation in the legacy docs view.")](#)View your model's documentation in the legacy docs view. ###### FAQs How do I write long-form explanations in my descriptions? If you need more than a sentence to explain a model, you can: 1. Split your description over multiple lines using `>`. Interior line breaks are removed and Markdown can be used. This method is recommended for simple, single-paragraph descriptions: ```yml models: - name: customers description: > Lorem ipsum **dolor** sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. ``` 2. Split your description over multiple lines using `|`. Interior line breaks are maintained and Markdown can be used. This method is recommended for more complex descriptions: ```yml models: - name: customers description: | ### Lorem ipsum * dolor sit amet, consectetur adipisicing elit, sed do eiusmod * tempor incididunt ut labore et dolore magna aliqua. ``` 3. Use a [docs block](https://docs.getdbt.com/docs/build/documentation.md#using-docs-blocks) to write the description in a separate Markdown file. How do I access documentation in dbt Catalog? If you're using dbt to deploy your project and have a [Starter, Enterprise, or Enterprise+ plan](https://www.getdbt.com/pricing/), you can use Catalog to view your project's [resources](https://docs.getdbt.com/docs/build/projects.md) (such as models, tests, and metrics) and their lineage to gain a better understanding of its latest production state. Access Catalog in dbt by clicking the **Catalog** link in the navigation. Read-only users can access the documentation for your project — for license availability by plan, refer to [dbt pricing](https://www.getdbt.com/pricing). dbt developer plan and dbt v1 users can use [dbt Docs](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md#dbt-docs), which generates basic documentation but it doesn't offer the same speed, metadata, or visibility as Catalog. #### Commit your changes Now that you've built your customer model, you need to commit the changes you made to the project so that the repository has your latest code. **If you edited directly in the protected primary branch:**<br /> 1. Click the **Commit and sync git** button. This action prepares your changes for commit. 2. A modal titled **Commit to a new branch** will appear. 3. In the modal window, name your new branch `add-customers-model`. This branches off from your primary branch with your new changes. 4. Add a commit message, such as "Add customers model, tests, docs" and and commit your changes. 5. Click **Merge this branch to main** to add these changes to the main branch on your repo. **If you created a new branch before editing:**<br /> 1. Since you already branched out of the primary protected branch, go to **Version Control** on the left. 2. Click **Commit and sync** to add a message. 3. Add a commit message, such as "Add customers model, tests, docs." 4. Click **Merge this branch to main** to add these changes to the main branch on your repo. #### Deploy dbt Use dbt's Scheduler to deploy your production jobs confidently and build observability into your processes. You'll learn to create a deployment environment and run a job in the following steps. ##### Create a deployment environment 1. From the main menu, go to **Orchestration** > **Environments**. 2. Click **Create environment**. 3. In the **Name** field, write the name of your deployment environment. For example, "Production." 4. The **dbt version** will default to the latest available. We recommend all new projects run on the latest version of dbt. 5. Under **Deployment connection**, enter the name of the dataset you want to use as the target, such as "Analytics". This will allow dbt to build and work with that dataset. For some data warehouses, the target dataset may be referred to as a "schema". 6. Click **Save**. ##### Create and run a job Jobs are a set of dbt commands that you want to run on a schedule. For example, `dbt build`. As the `jaffle_shop` business gains more customers, and those customers create more orders, you will see more records added to your source data. Because you materialized the `customers` model as a table, you'll need to periodically rebuild your table to ensure that the data stays up-to-date. This update will happen when you run a job. 1. After creating your deployment environment, you should be directed to the page for a new environment. If not, select **Orchestration** from the main menu, then click **Jobs**. 2. Click **Create job** > **Deploy job**. 3. Provide a job name (for example, "Production run") and select the environment you just created. 4. Scroll down to the **Execution settings** section. 5. Under **Commands**, add this command as part of your job if you don't see it: * `dbt build` 6. Select the **Generate docs on run** option to automatically [generate updated project docs](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) each time your job runs. 7. For this exercise, do *not* set a schedule for your project to run — while your organization's project should run regularly, there's no need to run this example project on a schedule. Scheduling a job is sometimes referred to as *deploying a project*. 8. Click **Save**, then click **Run now** to run your job. 9. Click the run and watch its progress under **Run summary**. 10. Once the run is complete, click **View Documentation** to see the docs for your project. Congratulations 🎉! You've just deployed your first dbt project! ###### FAQs What happens if one of my runs fails? If you're using dbt, we recommend setting up email and Slack notifications (`Account Settings > Notifications`) for any failed runs. Then, debug these runs the same way you would debug any runs in development. --- ### Quickstart for dbt and Azure Synapse Analytics [Back to guides](https://docs.getdbt.com/guides.md) dbt platform Quickstart Beginner [Menu ]() #### Introduction In this quickstart guide, you'll learn how to use dbt with [Azure Synapse Analytics](https://azure.microsoft.com/en-us/products/synapse-analytics/). It will show you how to: * Load the Jaffle Shop sample data (provided by dbt Labs) into your Azure Synapse Analytics warehouse. * Connect dbt to Azure Synapse Analytics. * Turn a sample query into a model in your dbt project. A model in dbt is a SELECT statement. * Add tests to your models. * Document your models. * Schedule a job to run. ##### Prerequisites * You have a [dbt](https://www.getdbt.com/signup/) account. * You have an Azure Synapse Analytics account. For a free trial, refer to [Synapse Analytics](https://azure.microsoft.com/en-us/free/synapse-analytics/) in the Microsoft docs. * As a Microsoft admin, you’ve enabled service principal authentication. You must add the service principal to the Synapse workspace with either a Member (recommended) or Admin permission set. For details, refer to [Create a service principal using the Azure portal](https://learn.microsoft.com/en-us/entra/identity-platform/howto-create-service-principal-portal) in the Microsoft docs. dbt needs these authentication credentials to connect to Azure Synapse Analytics. ##### Related content * [dbt Learn courses](https://learn.getdbt.com) * [About continuous integration jobs](https://docs.getdbt.com/docs/deploy/continuous-integration.md) * [Deploy jobs](https://docs.getdbt.com/docs/deploy/deploy-jobs.md) * [Job notifications](https://docs.getdbt.com/docs/deploy/job-notifications.md) * [Source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) #### Load data into your Azure Synapse Analytics 1. Log in to your [Azure portal account](https://portal.azure.com/#home). 2. On the home page, select the **SQL databases** tile. 3. From the **SQL databases** page, navigate to your organization’s workspace or create a new workspace; refer to [Create a Synapse workspace](https://learn.microsoft.com/en-us/azure/synapse-analytics/quickstart-create-workspace) in the Microsoft docs for more details. 4. From the workspace's sidebar, select **Data**. Click the three dot menu on your database and select **New SQL script** to open the SQL editor. 5. Copy these statements into the SQL editor to load the Jaffle Shop example data: ```sql CREATE TABLE dbo.customers ( [ID] [bigint], [FIRST_NAME] [varchar](8000), [LAST_NAME] [varchar](8000) ); COPY INTO [dbo].[customers] FROM 'https://dbtlabsynapsedatalake.blob.core.windows.net/dbt-quickstart-public/jaffle_shop_customers.parquet' WITH ( FILE_TYPE = 'PARQUET' ); CREATE TABLE dbo.orders ( [ID] [bigint], [USER_ID] [bigint], [ORDER_DATE] [date], [STATUS] [varchar](8000) ); COPY INTO [dbo].[orders] FROM 'https://dbtlabsynapsedatalake.blob.core.windows.net/dbt-quickstart-public/jaffle_shop_orders.parquet' WITH ( FILE_TYPE = 'PARQUET' ); CREATE TABLE dbo.payments ( [ID] [bigint], [ORDERID] [bigint], [PAYMENTMETHOD] [varchar](8000), [STATUS] [varchar](8000), [AMOUNT] [bigint], [CREATED] [date] ); COPY INTO [dbo].[payments] FROM 'https://dbtlabsynapsedatalake.blob.core.windows.net/dbt-quickstart-public/stripe_payments.parquet' WITH ( FILE_TYPE = 'PARQUET' ); ``` [![Example of loading data](/img/quickstarts/dbt-platform/example-load-data-azure-syn-analytics.png?v=2 "Example of loading data")](#)Example of loading data #### Connect dbt to Azure Synapse Analytics 1. Create a new project in dbt. Click on your account name in the left side menu, select **Account settings**, and click **+ New Project**. 2. Enter a project name and click **Continue**. 3. Choose **Synapse** as your connection and click **Next**. 4. In the **Configure your environment** section, enter the **Settings** for your new project: * **Server** — Use the service principal's **Synapse host name** value (without the trailing `, 1433` string) for the Synapse test endpoint. * **Port** — 1433 (which is the default). * **Database** — Use the service principal's **database** value for the Synapse test endpoint. 5. Enter the **User credentials** for your new project: * **Authentication** — Choose **Service Principal** from the dropdown. * **Tenant ID** — Use the service principal’s **Directory (tenant) id** as the value. * **Client ID** — Use the service principal’s **application (client) ID id** as the value. * **Client secret** — Use the service principal’s **client secret** (not the **client secret id**) as the value. 6. Click **Test connection**. This verifies that dbt can access your Azure Synapse Analytics account. 7. Click **Next** when the test succeeds. If it failed, you might need to check your Microsoft service principal. #### Set up a dbt managed repository When you develop in dbt, you can leverage [Git](https://docs.getdbt.com/docs/platform/git/git-version-control.md) to version control your code. To connect to a repository, you can either set up a dbt-hosted [managed repository](https://docs.getdbt.com/docs/platform/git/managed-repository.md) or directly connect to a [supported git provider](https://docs.getdbt.com/docs/platform/git/connect-github.md). Managed repositories are a great way to trial dbt without needing to create a new repository. In the long run, it's better to connect to a supported git provider to use features like automation and [continuous integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md). To set up a managed repository: 1. Under "Setup a repository", select **Managed**. 2. Type a name for your repo such as `bbaggins-dbt-quickstart` 3. Click **Create**. It will take a few seconds for your repository to be created and imported. 4. Once you see the "Successfully imported repository," click **Continue**. #### Initialize your dbt project​ and start developing Now that you have a repository configured, you can initialize your project and start development in dbt: 1. Click **Start developing in the Studio IDE**. It might take a few minutes for your project to spin up for the first time as it establishes your git connection, clones your repo, and tests the connection to the warehouse. 2. Above the file tree to the left, click **Initialize dbt project**. This builds out your folder structure with example models. 3. Make your initial commit by clicking **Commit and sync**. Use the commit message `initial commit` and click **Commit Changes**. This creates the first commit to your managed repo and allows you to open a branch where you can add new dbt code. 4. You can now directly query data from your warehouse and execute `dbt run`. You can try this out now: * In the command line bar at the bottom, enter `dbt run` and click **Enter**. You should see a `dbt run succeeded` message. #### Build your first model 1. Under **Version Control** on the left, click **Create branch**. You can name it `add-customers-model`. You need to create a new branch since the main branch is set to read-only mode. 2. Click the three dot menu (**...**) next to the `models` directory, then select **Create file**. 3. Name the file `customers.sql`, then click **Create**. 4. Copy the following query into the file and click **Save**. customers.sql ```sql with customers as ( select ID as customer_id, FIRST_NAME as first_name, LAST_NAME as last_name from dbo.customers ), orders as ( select ID as order_id, USER_ID as customer_id, ORDER_DATE as order_date, STATUS as status from dbo.orders ), customer_orders as ( select customer_id, min(order_date) as first_order_date, max(order_date) as most_recent_order_date, count(order_id) as number_of_orders from orders group by customer_id ), final as ( select customers.customer_id, customers.first_name, customers.last_name, customer_orders.first_order_date, customer_orders.most_recent_order_date, coalesce(customer_orders.number_of_orders, 0) as number_of_orders from customers left join customer_orders on customers.customer_id = customer_orders.customer_id ) select * from final ``` 5. Enter `dbt run` in the command prompt at the bottom of the screen. You should get a successful run and see the three models. Later, you can connect your business intelligence (BI) tools to these views and tables so they only read cleaned up data rather than raw data in your BI tool. ###### FAQs How can I see the SQL that dbt is running? To check out the SQL that dbt is running, you can look in: * dbt: * Within the run output, click on a model name, and then select "Details" * dbt v1: * The `target/compiled/` directory for compiled `select` statements * The `target/run/` directory for compiled `create` statements * The `logs/dbt.log` file for verbose logging. How did dbt choose which schema to build my models in? By default, dbt builds models in your target schema. To change your target schema: * If you're developing in **dbt**, these are set for each user when you first use a development environment. * If you're developing with **dbt v1**, this is the `schema:` parameter in your `profiles.yml` file. If you wish to split your models across multiple schemas, check out the docs on [using custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md). Note: on BigQuery, `dataset` is used interchangeably with `schema`. Do I need to create my target schema before running dbt? Nope! dbt will check if the schema exists when it runs. If the schema does not exist, dbt will create it for you. If I rerun dbt, will there be any downtime as models are rebuilt? Nope! The SQL that dbt generates behind the scenes ensures that any relations are replaced atomically (i.e. your business users won't experience any downtime). The implementation of this varies on each warehouse, check out the [logs](https://docs.getdbt.com/faqs/Runs/checking-logs.md) to see the SQL dbt is executing. What happens if the SQL in my query is bad or I get a database error? If there's a mistake in your SQL, dbt will return the error that your database returns. ```shell $ dbt run --select customers Running with dbt=1.9.0 Found 3 models, 9 tests, 0 snapshots, 0 analyses, 133 macros, 0 operations, 0 seed files, 0 sources 14:04:12 | Concurrency: 1 threads (target='dev') 14:04:12 | 14:04:12 | 1 of 1 START view model dbt_alice.customers.......................... [RUN] 14:04:13 | 1 of 1 ERROR creating view model dbt_alice.customers................. [ERROR in 0.81s] 14:04:13 | 14:04:13 | Finished running 1 view model in 1.68s. Completed with 1 error and 0 warnings: Database Error in model customers (models/customers.sql) Syntax error: Expected ")" but got identifier `your-info-12345` at [13:15] compiled SQL at target/run/jaffle_shop/customers.sql Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1 ``` Any models downstream of this model will also be skipped. Use the error message and the [compiled SQL](https://docs.getdbt.com/faqs/Runs/checking-logs.md) to debug any errors. #### Change the way your model is materialized One of the most powerful features of dbt is that you can change the way a model is materialized in your warehouse, simply by changing a configuration value. You can change things between tables and views by changing a keyword rather than writing the data definition language (DDL) to do this behind the scenes. By default, everything gets created as a view. You can override that at the directory level so everything in that directory will materialize to a different materialization. 1. Edit your `dbt_project.yml` file. * Update your project `name` to: dbt\_project.yml ```yaml name: 'jaffle_shop' ``` * Configure `jaffle_shop` so everything in it will be materialized as a table; and configure `example` so everything in it will be materialized as a view. Update your `models` config in the project YAML file to: dbt\_project.yml ```yaml models: jaffle_shop: +materialized: table example: +materialized: view ``` * Click **Save**. 2. Enter the `dbt run` command. Your `customers` model should now be built as a table! info To do this, dbt had to first run a `drop view` statement (or API call on BigQuery), then a `create table as` statement. 3. Edit `models/customers.sql` to override the `dbt_project.yml` for the `customers` model only by adding the following snippet to the top, and click **Save**: models/customers.sql ```sql {{ config( materialized='view' ) }} with customers as ( select id as customer_id ... ) ``` 4. Enter the `dbt run` command. Your model, `customers`, should now build as a view. * BigQuery users need to run `dbt run --full-refresh` instead of `dbt run` to full apply materialization changes. 5. Enter the `dbt run --full-refresh` command for this to take effect in your warehouse. ##### FAQs What materializations are available in dbt? dbt ships with five built-in materializations: `view`, `table`, `incremental`, `ephemeral`, and `materialized_view`. Check out the documentation on [materializations](https://docs.getdbt.com/docs/build/materializations.md) for more information on each of these options. You can also create your own [custom materializations](https://docs.getdbt.com/guides/create-new-materializations.md). This is an advanced feature of dbt. Which materialization should I use for my model? Start out with views, and then change models to tables when required for performance reasons (i.e. downstream queries have slowed). Check out the [docs on materializations](https://docs.getdbt.com/docs/build/materializations.md) for advice on when to use each materialization. What model configurations exist? You can also configure: * [tags](https://docs.getdbt.com/reference/resource-configs/tags.md) to support easy categorization and graph selection * [custom schemas](https://docs.getdbt.com/reference/resource-properties/schema.md) to split your models across multiple schemas * [aliases](https://docs.getdbt.com/reference/resource-configs/alias.md) if your view/table name should differ from the filename * Snippets of SQL to run at the start or end of a model, known as [hooks](https://docs.getdbt.com/docs/build/hooks-operations.md) * Warehouse-specific configurations for performance (e.g. `sort` and `dist` keys on Redshift, `partitions` on BigQuery) Check out the docs on [model configurations](https://docs.getdbt.com/reference/model-configs.md) to learn more. #### Delete the example models You can now delete the files that dbt created when you initialized the project: 1. Delete the `models/example/` directory. 2. Delete the `example:` key from your `dbt_project.yml` file, and any configurations that are listed under it. dbt\_project.yml ```yaml # before models: jaffle_shop: +materialized: table example: +materialized: view ``` dbt\_project.yml ```yaml # after models: jaffle_shop: +materialized: table ``` 3. Save your changes. ###### FAQs How do I remove deleted models from my data warehouse? If you delete a model from your dbt project, dbt does not automatically drop the relation from your schema. This means that you can end up with extra objects in schemas that dbt creates, which can be confusing to other users. (This can also happen when you switch a model from being a view or table, to ephemeral) When you remove models from your dbt project, you should manually drop the related relations from your schema. I got an "unused model configurations" error message, what does this mean? You might have forgotten to nest your configurations under your project name, or you might be trying to apply configurations to a directory that doesn't exist. Check out this [article](https://discourse.getdbt.com/t/faq-i-got-an-unused-model-configurations-error-message-what-does-this-mean/112) to understand more. #### Build models on top of other models As a best practice in SQL, you should separate logic that cleans up your data from logic that transforms your data. You have already started doing this in the existing query by using common table expressions (CTEs). Now you can experiment by separating the logic out into separate models and using the [ref](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) function to build models on top of other models: [![The DAG we want for our dbt project](/img/dbt-dag.png?v=2 "The DAG we want for our dbt project")](#)The DAG we want for our dbt project 1. Create a new SQL file, `models/stg_customers.sql`, with the SQL from the `customers` CTE in our original query. 2. Create a second new SQL file, `models/stg_orders.sql`, with the SQL from the `orders` CTE in our original query. models/stg\_customers.sql ```sql select ID as customer_id, FIRST_NAME as first_name, LAST_NAME as last_name from dbo.customers ``` models/stg\_orders.sql ```sql select ID as order_id, USER_ID as customer_id, ORDER_DATE as order_date, STATUS as status from dbo.orders ``` 3. Edit the SQL in your `models/customers.sql` file as follows: models/customers.sql ```sql with customers as ( select * from {{ ref('stg_customers') }} ), orders as ( select * from {{ ref('stg_orders') }} ), customer_orders as ( select customer_id, min(order_date) as first_order_date, max(order_date) as most_recent_order_date, count(order_id) as number_of_orders from orders group by customer_id ), final as ( select customers.customer_id, customers.first_name, customers.last_name, customer_orders.first_order_date, customer_orders.most_recent_order_date, coalesce(customer_orders.number_of_orders, 0) as number_of_orders from customers left join customer_orders on customers.customer_id = customer_orders.customer_id ) select * from final ``` 4. Execute `dbt run`. This time, when you performed a `dbt run`, separate views/tables were created for `stg_customers`, `stg_orders` and `customers`. dbt inferred the order to run these models. Because `customers` depends on `stg_customers` and `stg_orders`, dbt builds `customers` last. You do not need to explicitly define these dependencies. ###### FAQs How do I run one model at a time? To run one model, use the `--select` flag (or `-s` flag), followed by the name of the model: ```shell $ dbt run --select customers ``` Check out the [model selection syntax documentation](https://docs.getdbt.com/reference/node-selection/syntax.md) for more operators and examples. Do ref-able resource names need to be unique? Within one project: yes! To build dependencies between resources (such as models, seeds, and snapshots), you need to use the `ref` function, and pass in the resource name as an argument. dbt uses that resource name to uniquely resolve the `ref` to a specific resource. As a result, these resource names need to be unique, *even if they are in distinct folders*. A resource in one project can have the same name as a resource in another project (installed as a dependency). dbt uses the project name to uniquely identify each resource. We call this "namespacing." If you `ref` a resource with a duplicated name, it will resolve to the resource within the same namespace (package or project), or raise an error because of an ambiguous reference. Use [two-argument `ref`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models) to disambiguate references by specifying the namespace. Those resource will still need to land in distinct locations in the data warehouse. Read the docs on [custom aliases](https://docs.getdbt.com/docs/build/custom-aliases.md) and [custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md) for details on how to achieve this. As I create more models, how should I keep my project organized? What should I name my models? There's no one best way to structure a project! Every organization is unique. If you're just getting started, check out how we (dbt Labs) [structure our dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md). #### Add tests to your models Adding [data tests](https://docs.getdbt.com/docs/build/data-tests.md) to a project helps validate that your models are working correctly. To add data tests to your project: 1. Create a new YAML file in the `models` directory, named `models/schema.yml` 2. Add the following contents to the file: models/schema.yml ```yaml version: 2 models: - name: customers columns: - name: customer_id data_tests: - unique - not_null - name: stg_customers columns: - name: customer_id data_tests: - unique - not_null - name: stg_orders columns: - name: order_id data_tests: - unique - not_null - name: status data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['placed', 'shipped', 'completed', 'return_pending', 'returned'] - name: customer_id data_tests: - not_null - relationships: arguments: to: ref('stg_customers') field: customer_id ``` 3. Run `dbt test`, and confirm that all your tests passed. When you run `dbt test`, dbt iterates through your YAML files, and constructs a query for each test. Each query will return the number of records that fail the test. If this number is 0, then the test is successful. ###### FAQs What tests are available for me to use in dbt? Can I add my own custom tests? Out of the box, dbt ships with the following data tests: * `unique` * `not_null` * `accepted_values` * `relationships` (for example, referential integrity) You can also write your own [custom generic tests](https://docs.getdbt.com/docs/build/data-tests.md#generic-data-tests). Some additional generic tests have been open-sourced in the [dbt-utils package](https://github.com/dbt-labs/dbt-utils#generic-tests). Check out the docs on [packages](https://docs.getdbt.com/docs/build/packages.md) to learn how to make these tests available in your project. How do I test one model at a time? Running tests on one model looks very similar to running a model: use the `--select` flag (or `-s` flag), followed by the name of the model: ```shell dbt test --select customers ``` Check out the [model selection syntax documentation](https://docs.getdbt.com/reference/node-selection/syntax.md) for full syntax, and [test selection examples](https://docs.getdbt.com/reference/node-selection/test-selection-examples.md) in particular. One of my tests failed, how can I debug it? To debug a failing test, find the SQL that dbt ran by: * dbt: * Within the test output, click on the failed test, and then select "Details". * dbt v1: * Open the file path returned as part of the error message. * Navigate to the `target/compiled/schema_tests` directory for all compiled test queries. Copy the SQL into a query editor (in dbt, you can paste it into a new `Statement`), and run the query to find the records that failed. Does my test file need to be named \`schema.yml\`? No! You can name this file whatever you want (including `whatever_you_want.yml`), so long as: * The file is in your `models/` directory¹ * The file has `.yml` extension Check out the [docs](https://docs.getdbt.com/reference/configs-and-properties.md) for more information. ¹If you're declaring properties for seeds, snapshots, or macros, you can also place this file in the related directory — `seeds/`, `snapshots/` and `macros/` respectively. Why do model and source YAML files always start with \`version: 2\`? Once upon a time, the structure of these `.yml` files was very different (s/o to anyone who was using dbt back then!). Adding `version: 2` allowed us to make this structure more extensible. From [dbt v1.5](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.5.md#quick-hits>), the top-level `version:` key is optional in all resource YAML files. If present, only `version: 2` is supported. Also starting in v1.5, both the [`config-version: 2`](https://docs.getdbt.com/reference/project-configs/config-version.md) and the top-level `version:` key in the `dbt_project.yml` are optional. Resource YAML files do not currently require this config. We only support `version: 2` if it's specified. Although we do not expect to update YAML files to `version: 3` soon, having this config will make it easier for us to introduce new structures in the future What data tests should I add to my project? We recommend that every model has a data test on a primary key, that is, a column that is `unique` and `not_null`. We also recommend that you test any assumptions on your source data. For example, if you believe that your payments can only be one of three payment methods, you should test that assumption regularly — a new payment method may introduce logic errors in your SQL. In advanced dbt projects, we recommend using [sources](https://docs.getdbt.com/docs/build/sources.md) and running these source data-integrity tests against the sources rather than models. When should I run my data tests? You should run your data tests whenever you are writing new code (to ensure you haven't broken any existing models by changing SQL), and whenever you run your transformations in production (to ensure that your assumptions about your source data are still valid). #### Document your models Adding [documentation](https://docs.getdbt.com/docs/build/documentation.md) to your project allows you to describe your models in rich detail, and share that information with your team. Here, we're going to add some basic documentation to our project. Update your `models/schema.yml` file to include some descriptions, such as those below. models/schema.yml ```yaml version: 2 models: - name: customers description: One record per customer columns: - name: customer_id description: Primary key data_tests: - unique - not_null - name: first_order_date description: NULL when a customer has not yet placed an order. - name: stg_customers description: This model cleans up customer data columns: - name: customer_id description: Primary key data_tests: - unique - not_null - name: stg_orders description: This model cleans up order data columns: - name: order_id description: Primary key data_tests: - unique - not_null - name: status data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['placed', 'shipped', 'completed', 'return_pending', 'returned'] - name: customer_id data_tests: - not_null - relationships: arguments: to: ref('stg_customers') field: customer_id ``` ##### View in Catalog [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) provides powerful tools to interact with your dbt projects, including documentation: 1. Run one of the following commands: * `dbt docs generate` if you're on dbt v1 * `dbt build` if you're on dbt v2 2. Click **Catalog** in the navigation menu to launch Catalog. 3. Catalog reflects **Production** by default. If your account has additional environments (for example, **Staging**), you can select them from the environment dropdown. [![Select an environment in Catalog.](/img/docs/collaborate/dbt-explorer/catalog-nav-dropdown.png?v=2 "Select an environment in Catalog.")](#)Select an environment in Catalog. 4. Select your project from the file tree. 5. Use the search bar or browse the resource list to find the `customers` model. 6. Click the model to view its details, including the descriptions you added. [![View your model's documentation and lineage in Catalog.](/img/docs/collaborate/dbt-explorer/example-model-details.png?v=2 "View your model's documentation and lineage in Catalog.")](#)View your model's documentation and lineage in Catalog. Catalog displays your model's description, column documentation, data tests, and lineage graph. You can also see which columns are missing documentation and track test coverage across your project. ##### View in Studio IDE You can view docs directly from the IDE if you're on **v1 Latest** or another version of dbt v1. Keep in mind that this is a legacy view and doesn't offer the same level of interactivity as Catalog. 1. In the IDE, run `dbt docs generate`. 2. From the navigation bar, click the **View docs** icon located to the right of the **branch name**. [![The View docs icon in the Studio IDE.](/img/docs/collaborate/dbt-explorer/docs-icon.png?v=2 "The View docs icon in the Studio IDE.")](#)The View docs icon in the Studio IDE. 3. From **Projects**, select your project name and expand the folders. 4. Click **models** > **marts** > **customers**. [![View your model's documentation in the legacy docs view.](/img/docs/collaborate/dbt-explorer/legacy-docs-view.png?v=2 "View your model's documentation in the legacy docs view.")](#)View your model's documentation in the legacy docs view. ###### FAQs How do I write long-form explanations in my descriptions? If you need more than a sentence to explain a model, you can: 1. Split your description over multiple lines using `>`. Interior line breaks are removed and Markdown can be used. This method is recommended for simple, single-paragraph descriptions: ```yml models: - name: customers description: > Lorem ipsum **dolor** sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. ``` 2. Split your description over multiple lines using `|`. Interior line breaks are maintained and Markdown can be used. This method is recommended for more complex descriptions: ```yml models: - name: customers description: | ### Lorem ipsum * dolor sit amet, consectetur adipisicing elit, sed do eiusmod * tempor incididunt ut labore et dolore magna aliqua. ``` 3. Use a [docs block](https://docs.getdbt.com/docs/build/documentation.md#using-docs-blocks) to write the description in a separate Markdown file. How do I access documentation in dbt Catalog? If you're using dbt to deploy your project and have a [Starter, Enterprise, or Enterprise+ plan](https://www.getdbt.com/pricing/), you can use Catalog to view your project's [resources](https://docs.getdbt.com/docs/build/projects.md) (such as models, tests, and metrics) and their lineage to gain a better understanding of its latest production state. Access Catalog in dbt by clicking the **Catalog** link in the navigation. Read-only users can access the documentation for your project — for license availability by plan, refer to [dbt pricing](https://www.getdbt.com/pricing). dbt developer plan and dbt v1 users can use [dbt Docs](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md#dbt-docs), which generates basic documentation but it doesn't offer the same speed, metadata, or visibility as Catalog. #### Commit your changes Now that you've built your customer model, you need to commit the changes you made to the project so that the repository has your latest code. **If you edited directly in the protected primary branch:**<br /> 1. Click the **Commit and sync git** button. This action prepares your changes for commit. 2. A modal titled **Commit to a new branch** will appear. 3. In the modal window, name your new branch `add-customers-model`. This branches off from your primary branch with your new changes. 4. Add a commit message, such as "Add customers model, tests, docs" and and commit your changes. 5. Click **Merge this branch to main** to add these changes to the main branch on your repo. **If you created a new branch before editing:**<br /> 1. Since you already branched out of the primary protected branch, go to **Version Control** on the left. 2. Click **Commit and sync** to add a message. 3. Add a commit message, such as "Add customers model, tests, docs." 4. Click **Merge this branch to main** to add these changes to the main branch on your repo. #### Deploy dbt Use dbt's Scheduler to deploy your production jobs confidently and build observability into your processes. You'll learn to create a deployment environment and run a job in the following steps. ##### Create a deployment environment 1. From the main menu, go to **Orchestration** > **Environments**. 2. Click **Create environment**. 3. In the **Name** field, write the name of your deployment environment. For example, "Production." 4. The **dbt version** will default to the latest available. We recommend all new projects run on the latest version of dbt. 5. Under **Deployment connection**, enter the name of the dataset you want to use as the target, such as "Analytics". This will allow dbt to build and work with that dataset. For some data warehouses, the target dataset may be referred to as a "schema". 6. Click **Save**. ##### Create and run a job Jobs are a set of dbt commands that you want to run on a schedule. For example, `dbt build`. As the `jaffle_shop` business gains more customers, and those customers create more orders, you will see more records added to your source data. Because you materialized the `customers` model as a table, you'll need to periodically rebuild your table to ensure that the data stays up-to-date. This update will happen when you run a job. 1. After creating your deployment environment, you should be directed to the page for a new environment. If not, select **Orchestration** from the main menu, then click **Jobs**. 2. Click **Create job** > **Deploy job**. 3. Provide a job name (for example, "Production run") and select the environment you just created. 4. Scroll down to the **Execution settings** section. 5. Under **Commands**, add this command as part of your job if you don't see it: * `dbt build` 6. Select the **Generate docs on run** option to automatically [generate updated project docs](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) each time your job runs. 7. For this exercise, do *not* set a schedule for your project to run — while your organization's project should run regularly, there's no need to run this example project on a schedule. Scheduling a job is sometimes referred to as *deploying a project*. 8. Click **Save**, then click **Run now** to run your job. 9. Click the run and watch its progress under **Run summary**. 10. Once the run is complete, click **View Documentation** to see the docs for your project. Congratulations 🎉! You've just deployed your first dbt project! ###### FAQs What happens if one of my runs fails? If you're using dbt, we recommend setting up email and Slack notifications (`Account Settings > Notifications`) for any failed runs. Then, debug these runs the same way you would debug any runs in development. --- ### Quickstart for dbt and BigQuery [Back to guides](https://docs.getdbt.com/guides.md) BigQuery Platform Quickstart Beginner [Menu ]() #### Introduction In this quickstart guide, you'll learn how to use dbt with BigQuery. It will show you how to: * Create a Google Cloud Platform (GCP) project. * Access sample data in a public dataset. * Connect dbt to BigQuery. * Take a sample query and turn it into a model in your dbt project. A model in dbt is a select statement. * Add tests to your models. * Document your models. * Schedule a job to run. Videos for you You can check out [dbt Fundamentals](https://learn.getdbt.com/courses/dbt-fundamentals) for free if you're interested in course learning with videos. ##### Prerequisites​ * You have a [dbt account](https://www.getdbt.com/signup/). * You have a [Google account](https://support.google.com/accounts/answer/27441?hl=en). * You can use a personal or work account to set up BigQuery through [Google Cloud Platform (GCP)](https://cloud.google.com/free). ##### Related content * Learn more with [dbt Learn courses](https://learn.getdbt.com) * [CI jobs](https://docs.getdbt.com/docs/deploy/continuous-integration.md) * [Deploy jobs](https://docs.getdbt.com/docs/deploy/deploy-jobs.md) * [Job notifications](https://docs.getdbt.com/docs/deploy/job-notifications.md) * [Source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) #### Create a new GCP project​ 1. Go to the [BigQuery Console](https://console.cloud.google.com/bigquery) after you log in to your Google account. If you have multiple Google accounts, make sure you’re using the correct one. 2. Create a new project from the [Manage resources page](https://console.cloud.google.com/projectcreate?previousPage=%2Fcloud-resource-manager%3Fwalkthrough_id%3Dresource-manager--create-project%26project%3D%26folder%3D%26organizationId%3D%23step_index%3D1\&walkthrough_id=resource-manager--create-project). For more information, refer to [Creating a project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#creating_a_project) in the Google Cloud docs. GCP automatically populates the Project name field for you. You can change it to be more descriptive for your use. For example, `dbt Learn - BigQuery Setup`. #### Create BigQuery datasets 1. From the [BigQuery Console](https://console.cloud.google.com/bigquery), click **Editor**. Make sure to select your newly created project, which is available at the top of the page. 2. Verify that you can run SQL queries. Copy and paste these queries into the Query Editor: ```sql select * from `dbt-tutorial.jaffle_shop.customers`; select * from `dbt-tutorial.jaffle_shop.orders`; select * from `dbt-tutorial.stripe.payment`; ``` Click **Run**, then check for results from the queries. For example: [![Bigquery Query Results](/img/bigquery/query-results.png?v=2 "Bigquery Query Results")](#)Bigquery Query Results 3. Create new datasets from the [BigQuery Console](https://console.cloud.google.com/bigquery). For more information, refer to [Create datasets](https://cloud.google.com/bigquery/docs/datasets#create-dataset) in the Google Cloud docs. Datasets in BigQuery are equivalent to schemas in a traditional database. On the **Create dataset** page: * **Dataset ID** — Enter a name that fits the purpose. This name is used like schema in fully qualified references to your database objects such as `database.schema.table`. As an example for this guide, create one for `jaffle_shop` and another one for `stripe` afterward. * **Data location** — Leave it blank (the default). It determines the GCP location of where your data is stored. The current default location is the US multi-region. All tables within this dataset will share this location. * **Enable table expiration** — Leave it unselected (the default). The default for the billing table expiration is 60 days. Because billing isn’t enabled for this project, GCP defaults to deprecating tables. * **Google-managed encryption key** — This option is available under **Advanced options**. Allow Google to manage encryption (the default). [![Bigquery Create Dataset ID](/img/bigquery/create-dataset-id.png?v=2 "Bigquery Create Dataset ID")](#)Bigquery Create Dataset ID 4. After you create the `jaffle_shop` dataset, create one for `stripe` with all the same values except for **Dataset ID**. #### Generate BigQuery credentials In order to let dbt connect to your warehouse, you'll need to generate a keyfile. This is analogous to using a database username and password with most other data warehouses. 1. Start the [GCP credentials wizard](https://console.cloud.google.com/apis/credentials/wizard). Make sure your new project is selected in the header. If you do not see your account or project, click your profile picture to the right and verify you are using the correct email account. For **Credential Type**: * From the **Select an API** dropdown, choose **BigQuery API** * Select **Application data** for the type of data you will be accessing * Click **Next** to create a new service account. 2. Create a service account for your new project from the [Service accounts page](https://console.cloud.google.com/projectselector2/iam-admin/serviceaccounts?supportedpurview=project). For more information, refer to [Create a service account](https://developers.google.com/workspace/guides/create-credentials#create_a_service_account) in the Google Cloud docs. As an example for this guide, you can: * Type `dbt-user` as the **Service account name** * From the **Select a role** dropdown, choose **BigQuery Job User** and **BigQuery Data Editor** roles and click **Continue** * Leave the **Grant users access to this service account** fields blank * Click **Done** 3. Create a service account key for your new project from the [Service accounts page](https://console.cloud.google.com/iam-admin/serviceaccounts?walkthrough_id=iam--create-service-account-keys\&start_index=1#step_index=1). For more information, refer to [Create a service account key](https://cloud.google.com/iam/docs/creating-managing-service-account-keys#creating) in the Google Cloud docs. When downloading the JSON file, make sure to use a filename you can easily remember. For example, `dbt-user-creds.json`. For security reasons, dbt Labs recommends that you protect this JSON file like you would your identity credentials; for example, don't check the JSON file into your version control software. #### Connect dbt to BigQuery​ 1. In [dbt](https://docs.getdbt.com/docs/platform/about-platform/access-regions-ip-addresses.md), click your account name in the left side menu. 2. Navigate to **Account settings** and click **+ New project**. 3. Enter a project name and click **Continue**. 4. For the warehouse, click **BigQuery** then **Next** to set up your connection. 5. Click **Upload a Service Account JSON File** in settings. 6. Select the JSON file you downloaded in Generate BigQuery credentials and dbt will fill in all the necessary fields. 7. (Optional) dbt Enterprise plans can configure developer OAuth with BigQuery, providing an additional layer of security. For more information, refer to [Set up BigQuery OAuth](https://docs.getdbt.com/docs/platform/manage-access/set-up-bigquery-oauth.md). 8. Set up your personal user credentials by navigating to **Your profile** > **Credentials**. 9. Select your project that uses the BigQuery connection. 10. Click **Edit**. 11. Enter your **User credentials** for BigQuery with: * **Authentication Method** — Select **Service Account JSON**. This uses the service account you uploaded when you set up the project connection. * **Dataset** — You may notice that the dataset name has been auto-created for you. By convention, this is `dbt_<first-initial><last-name>`. This is the dataset connected directly to your development environment, and it's where your models will be built when running dbt within the Studio IDE. * **Target name** — Leave as the default. * **Threads** — Leave as the default (6). This is the number of simultaneous connections that dbt will make to build models concurrently. [![BigQuery User credentials](/img/bigquery/dbt_platform_bigquery_development_credentials.png?v=2 "BigQuery User credentials")](#)BigQuery User credentials 12. Click **Test Connection**. This verifies that dbt can access your BigQuery account. 13. Click **Next** if the test succeeded. If it failed, you might need to go back and regenerate your BigQuery credentials. #### Set up a dbt managed repository When you develop in dbt, you can leverage [Git](https://docs.getdbt.com/docs/platform/git/git-version-control.md) to version control your code. To connect to a repository, you can either set up a dbt-hosted [managed repository](https://docs.getdbt.com/docs/platform/git/managed-repository.md) or directly connect to a [supported git provider](https://docs.getdbt.com/docs/platform/git/connect-github.md). Managed repositories are a great way to trial dbt without needing to create a new repository. In the long run, it's better to connect to a supported git provider to use features like automation and [continuous integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md). To set up a managed repository: 1. Under "Setup a repository", select **Managed**. 2. Type a name for your repo such as `bbaggins-dbt-quickstart` 3. Click **Create**. It will take a few seconds for your repository to be created and imported. 4. Once you see the "Successfully imported repository," click **Continue**. #### Initialize your dbt project​ and start developing Now that you have a repository configured, you can initialize your project and start development in dbt: 1. Click **Start developing in the Studio IDE**. It might take a few minutes for your project to spin up for the first time as it establishes your git connection, clones your repo, and tests the connection to the warehouse. 2. Above the file tree to the left, click **Initialize dbt project**. This builds out your folder structure with example models. 3. Make your initial commit by clicking **Commit and sync**. Use the commit message `initial commit` and click **Commit**. This creates the first commit to your managed repo and allows you to open a branch where you can add new dbt code. 4. You can now directly query data from your warehouse and execute `dbt run`. You can try this out now: * Click **+ Create new file**, add this query to the new file, and click **Save as** to save the new file: ```sql select * from `dbt-tutorial.jaffle_shop.customers` ``` * In the command line bar at the bottom, enter `dbt run` and click **Enter**. You should see a `dbt run succeeded` message. #### Build your first model You have two options for working with files in the Studio IDE: * Create a new branch (recommended) — Create a new branch to edit and commit your changes. Navigate to **Version Control** on the left sidebar and click **Create branch**. * Edit in the protected primary branch — If you prefer to edit, format, or lint files and execute dbt commands directly in your primary git branch. The Studio IDE prevents commits to the protected branch, so you will be prompted to commit your changes to a new branch. Name the new branch `add-customers-model`. 1. Click the **...** next to the `models` directory, then select **Create file**. 2. Name the file `customers.sql`, then click **Create**. 3. Copy the following query into the file and click **Save**. ```sql with customers as ( select id as customer_id, first_name, last_name from `dbt-tutorial`.jaffle_shop.customers ), orders as ( select id as order_id, user_id as customer_id, order_date, status from `dbt-tutorial`.jaffle_shop.orders ), customer_orders as ( select customer_id, min(order_date) as first_order_date, max(order_date) as most_recent_order_date, count(order_id) as number_of_orders from orders group by 1 ), final as ( select customers.customer_id, customers.first_name, customers.last_name, customer_orders.first_order_date, customer_orders.most_recent_order_date, coalesce(customer_orders.number_of_orders, 0) as number_of_orders from customers left join customer_orders using (customer_id) ) select * from final ``` 4. Enter `dbt run` in the command prompt at the bottom of the screen. You should get a successful run and see the three models. Later, you can connect your business intelligence (BI) tools to these views and tables so they only read cleaned up data rather than raw data in your BI tool. ###### FAQs How can I see the SQL that dbt is running? To check out the SQL that dbt is running, you can look in: * dbt: * Within the run output, click on a model name, and then select "Details" * dbt v1: * The `target/compiled/` directory for compiled `select` statements * The `target/run/` directory for compiled `create` statements * The `logs/dbt.log` file for verbose logging. How did dbt choose which schema to build my models in? By default, dbt builds models in your target schema. To change your target schema: * If you're developing in **dbt**, these are set for each user when you first use a development environment. * If you're developing with **dbt v1**, this is the `schema:` parameter in your `profiles.yml` file. If you wish to split your models across multiple schemas, check out the docs on [using custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md). Note: on BigQuery, `dataset` is used interchangeably with `schema`. Do I need to create my target schema before running dbt? Nope! dbt will check if the schema exists when it runs. If the schema does not exist, dbt will create it for you. If I rerun dbt, will there be any downtime as models are rebuilt? Nope! The SQL that dbt generates behind the scenes ensures that any relations are replaced atomically (i.e. your business users won't experience any downtime). The implementation of this varies on each warehouse, check out the [logs](https://docs.getdbt.com/faqs/Runs/checking-logs.md) to see the SQL dbt is executing. What happens if the SQL in my query is bad or I get a database error? If there's a mistake in your SQL, dbt will return the error that your database returns. ```shell $ dbt run --select customers Running with dbt=1.9.0 Found 3 models, 9 tests, 0 snapshots, 0 analyses, 133 macros, 0 operations, 0 seed files, 0 sources 14:04:12 | Concurrency: 1 threads (target='dev') 14:04:12 | 14:04:12 | 1 of 1 START view model dbt_alice.customers.......................... [RUN] 14:04:13 | 1 of 1 ERROR creating view model dbt_alice.customers................. [ERROR in 0.81s] 14:04:13 | 14:04:13 | Finished running 1 view model in 1.68s. Completed with 1 error and 0 warnings: Database Error in model customers (models/customers.sql) Syntax error: Expected ")" but got identifier `your-info-12345` at [13:15] compiled SQL at target/run/jaffle_shop/customers.sql Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1 ``` Any models downstream of this model will also be skipped. Use the error message and the [compiled SQL](https://docs.getdbt.com/faqs/Runs/checking-logs.md) to debug any errors. #### Change the way your model is materialized One of the most powerful features of dbt is that you can change the way a model is materialized in your warehouse, simply by changing a configuration value. You can change things between tables and views by changing a keyword rather than writing the data definition language (DDL) to do this behind the scenes. By default, everything gets created as a view. You can override that at the directory level so everything in that directory will materialize to a different materialization. 1. Edit your `dbt_project.yml` file. * Update your project `name` to: dbt\_project.yml ```yaml name: 'jaffle_shop' ``` * Configure `jaffle_shop` so everything in it will be materialized as a table; and configure `example` so everything in it will be materialized as a view. Update your `models` config in the project YAML file to: dbt\_project.yml ```yaml models: jaffle_shop: +materialized: table example: +materialized: view ``` * Click **Save**. 2. Enter the `dbt run` command. Your `customers` model should now be built as a table! info To do this, dbt had to first run a `drop view` statement (or API call on BigQuery), then a `create table as` statement. 3. Edit `models/customers.sql` to override the `dbt_project.yml` for the `customers` model only by adding the following snippet to the top, and click **Save**: models/customers.sql ```sql {{ config( materialized='view' ) }} with customers as ( select id as customer_id ... ) ``` 4. Enter the `dbt run` command. Your model, `customers`, should now build as a view. * BigQuery users need to run `dbt run --full-refresh` instead of `dbt run` to full apply materialization changes. 5. Enter the `dbt run --full-refresh` command for this to take effect in your warehouse. ##### FAQs What materializations are available in dbt? dbt ships with five built-in materializations: `view`, `table`, `incremental`, `ephemeral`, and `materialized_view`. Check out the documentation on [materializations](https://docs.getdbt.com/docs/build/materializations.md) for more information on each of these options. You can also create your own [custom materializations](https://docs.getdbt.com/guides/create-new-materializations.md). This is an advanced feature of dbt. Which materialization should I use for my model? Start out with views, and then change models to tables when required for performance reasons (i.e. downstream queries have slowed). Check out the [docs on materializations](https://docs.getdbt.com/docs/build/materializations.md) for advice on when to use each materialization. What model configurations exist? You can also configure: * [tags](https://docs.getdbt.com/reference/resource-configs/tags.md) to support easy categorization and graph selection * [custom schemas](https://docs.getdbt.com/reference/resource-properties/schema.md) to split your models across multiple schemas * [aliases](https://docs.getdbt.com/reference/resource-configs/alias.md) if your view/table name should differ from the filename * Snippets of SQL to run at the start or end of a model, known as [hooks](https://docs.getdbt.com/docs/build/hooks-operations.md) * Warehouse-specific configurations for performance (e.g. `sort` and `dist` keys on Redshift, `partitions` on BigQuery) Check out the docs on [model configurations](https://docs.getdbt.com/reference/model-configs.md) to learn more. #### Delete the example models You can now delete the files that dbt created when you initialized the project: 1. Delete the `models/example/` directory. 2. Delete the `example:` key from your `dbt_project.yml` file, and any configurations that are listed under it. dbt\_project.yml ```yaml # before models: jaffle_shop: +materialized: table example: +materialized: view ``` dbt\_project.yml ```yaml # after models: jaffle_shop: +materialized: table ``` 3. Save your changes. ###### FAQs How do I remove deleted models from my data warehouse? If you delete a model from your dbt project, dbt does not automatically drop the relation from your schema. This means that you can end up with extra objects in schemas that dbt creates, which can be confusing to other users. (This can also happen when you switch a model from being a view or table, to ephemeral) When you remove models from your dbt project, you should manually drop the related relations from your schema. I got an "unused model configurations" error message, what does this mean? You might have forgotten to nest your configurations under your project name, or you might be trying to apply configurations to a directory that doesn't exist. Check out this [article](https://discourse.getdbt.com/t/faq-i-got-an-unused-model-configurations-error-message-what-does-this-mean/112) to understand more. #### Build models on top of other models As a best practice in SQL, you should separate logic that cleans up your data from logic that transforms your data. You have already started doing this in the existing query by using common table expressions (CTEs). Now you can experiment by separating the logic out into separate models and using the [ref](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) function to build models on top of other models: [![The DAG we want for our dbt project](/img/dbt-dag.png?v=2 "The DAG we want for our dbt project")](#)The DAG we want for our dbt project 1. Create a new SQL file, `models/stg_customers.sql`, with the SQL from the `customers` CTE in our original query. 2. Create a second new SQL file, `models/stg_orders.sql`, with the SQL from the `orders` CTE in our original query. models/stg\_customers.sql ```sql select id as customer_id, first_name, last_name from `dbt-tutorial`.jaffle_shop.customers ``` models/stg\_orders.sql ```sql select id as order_id, user_id as customer_id, order_date, status from `dbt-tutorial`.jaffle_shop.orders ``` 3. Edit the SQL in your `models/customers.sql` file as follows: models/customers.sql ```sql with customers as ( select * from {{ ref('stg_customers') }} ), orders as ( select * from {{ ref('stg_orders') }} ), customer_orders as ( select customer_id, min(order_date) as first_order_date, max(order_date) as most_recent_order_date, count(order_id) as number_of_orders from orders group by 1 ), final as ( select customers.customer_id, customers.first_name, customers.last_name, customer_orders.first_order_date, customer_orders.most_recent_order_date, coalesce(customer_orders.number_of_orders, 0) as number_of_orders from customers left join customer_orders using (customer_id) ) select * from final ``` 4. Execute `dbt run`. This time, when you performed a `dbt run`, separate views/tables were created for `stg_customers`, `stg_orders` and `customers`. dbt inferred the order to run these models. Because `customers` depends on `stg_customers` and `stg_orders`, dbt builds `customers` last. You do not need to explicitly define these dependencies. #### Build models on top of sources Sources make it possible to name and describe the data loaded into your warehouse by your extract and load tools. By declaring these tables as sources in dbt, you can: * select from source tables in your models using the `{{ source() }}` function, helping define the lineage of your data * test your assumptions about your source data * calculate the freshness of your source data 1. Create a new YML file `models/sources.yml`. 2. Declare the sources by copying the following into the file and clicking **Save**. models/sources.yml ```yml sources: - name: jaffle_shop description: This is a replica of the Postgres database used by our app database: dbt-tutorial schema: jaffle_shop tables: - name: customers description: One record per customer. - name: orders description: One record per order. Includes cancelled and deleted orders. ``` 3. Edit the `models/stg_customers.sql` file to select from the `customers` table in the `jaffle_shop` source. models/stg\_customers.sql ```sql select id as customer_id, first_name, last_name from {{ source('jaffle_shop', 'customers') }} ``` 4. Edit the `models/stg_orders.sql` file to select from the `orders` table in the `jaffle_shop` source. models/stg\_orders.sql ```sql select id as order_id, user_id as customer_id, order_date, status from {{ source('jaffle_shop', 'orders') }} ``` 5. Execute `dbt run`. The results of your `dbt run` will be exactly the same as the previous step. Your `stg_customers` and `stg_orders` models will still query from the same raw data source in BigQuery. By using `source`, you can test and document your raw data and also understand the lineage of your sources. ###### FAQs How do I run one model at a time? To run one model, use the `--select` flag (or `-s` flag), followed by the name of the model: ```shell $ dbt run --select customers ``` Check out the [model selection syntax documentation](https://docs.getdbt.com/reference/node-selection/syntax.md) for more operators and examples. Do ref-able resource names need to be unique? Within one project: yes! To build dependencies between resources (such as models, seeds, and snapshots), you need to use the `ref` function, and pass in the resource name as an argument. dbt uses that resource name to uniquely resolve the `ref` to a specific resource. As a result, these resource names need to be unique, *even if they are in distinct folders*. A resource in one project can have the same name as a resource in another project (installed as a dependency). dbt uses the project name to uniquely identify each resource. We call this "namespacing." If you `ref` a resource with a duplicated name, it will resolve to the resource within the same namespace (package or project), or raise an error because of an ambiguous reference. Use [two-argument `ref`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models) to disambiguate references by specifying the namespace. Those resource will still need to land in distinct locations in the data warehouse. Read the docs on [custom aliases](https://docs.getdbt.com/docs/build/custom-aliases.md) and [custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md) for details on how to achieve this. As I create more models, how should I keep my project organized? What should I name my models? There's no one best way to structure a project! Every organization is unique. If you're just getting started, check out how we (dbt Labs) [structure our dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md). #### Add tests to your models Adding [data tests](https://docs.getdbt.com/docs/build/data-tests.md) to a project helps validate that your models are working correctly. To add data tests to your project: 1. Create a new YAML file in the `models` directory, named `models/schema.yml` 2. Add the following contents to the file: models/schema.yml ```yaml version: 2 models: - name: customers columns: - name: customer_id data_tests: - unique - not_null - name: stg_customers columns: - name: customer_id data_tests: - unique - not_null - name: stg_orders columns: - name: order_id data_tests: - unique - not_null - name: status data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['placed', 'shipped', 'completed', 'return_pending', 'returned'] - name: customer_id data_tests: - not_null - relationships: arguments: to: ref('stg_customers') field: customer_id ``` 3. Run `dbt test`, and confirm that all your tests passed. When you run `dbt test`, dbt iterates through your YAML files, and constructs a query for each test. Each query will return the number of records that fail the test. If this number is 0, then the test is successful. ###### FAQs What tests are available for me to use in dbt? Can I add my own custom tests? Out of the box, dbt ships with the following data tests: * `unique` * `not_null` * `accepted_values` * `relationships` (for example, referential integrity) You can also write your own [custom generic tests](https://docs.getdbt.com/docs/build/data-tests.md#generic-data-tests). Some additional generic tests have been open-sourced in the [dbt-utils package](https://github.com/dbt-labs/dbt-utils#generic-tests). Check out the docs on [packages](https://docs.getdbt.com/docs/build/packages.md) to learn how to make these tests available in your project. How do I test one model at a time? Running tests on one model looks very similar to running a model: use the `--select` flag (or `-s` flag), followed by the name of the model: ```shell dbt test --select customers ``` Check out the [model selection syntax documentation](https://docs.getdbt.com/reference/node-selection/syntax.md) for full syntax, and [test selection examples](https://docs.getdbt.com/reference/node-selection/test-selection-examples.md) in particular. One of my tests failed, how can I debug it? To debug a failing test, find the SQL that dbt ran by: * dbt: * Within the test output, click on the failed test, and then select "Details". * dbt v1: * Open the file path returned as part of the error message. * Navigate to the `target/compiled/schema_tests` directory for all compiled test queries. Copy the SQL into a query editor (in dbt, you can paste it into a new `Statement`), and run the query to find the records that failed. Does my test file need to be named \`schema.yml\`? No! You can name this file whatever you want (including `whatever_you_want.yml`), so long as: * The file is in your `models/` directory¹ * The file has `.yml` extension Check out the [docs](https://docs.getdbt.com/reference/configs-and-properties.md) for more information. ¹If you're declaring properties for seeds, snapshots, or macros, you can also place this file in the related directory — `seeds/`, `snapshots/` and `macros/` respectively. Why do model and source YAML files always start with \`version: 2\`? Once upon a time, the structure of these `.yml` files was very different (s/o to anyone who was using dbt back then!). Adding `version: 2` allowed us to make this structure more extensible. From [dbt v1.5](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.5.md#quick-hits>), the top-level `version:` key is optional in all resource YAML files. If present, only `version: 2` is supported. Also starting in v1.5, both the [`config-version: 2`](https://docs.getdbt.com/reference/project-configs/config-version.md) and the top-level `version:` key in the `dbt_project.yml` are optional. Resource YAML files do not currently require this config. We only support `version: 2` if it's specified. Although we do not expect to update YAML files to `version: 3` soon, having this config will make it easier for us to introduce new structures in the future What data tests should I add to my project? We recommend that every model has a data test on a primary key, that is, a column that is `unique` and `not_null`. We also recommend that you test any assumptions on your source data. For example, if you believe that your payments can only be one of three payment methods, you should test that assumption regularly — a new payment method may introduce logic errors in your SQL. In advanced dbt projects, we recommend using [sources](https://docs.getdbt.com/docs/build/sources.md) and running these source data-integrity tests against the sources rather than models. When should I run my data tests? You should run your data tests whenever you are writing new code (to ensure you haven't broken any existing models by changing SQL), and whenever you run your transformations in production (to ensure that your assumptions about your source data are still valid). #### Document your models Adding [documentation](https://docs.getdbt.com/docs/build/documentation.md) to your project allows you to describe your models in rich detail, and share that information with your team. Here, we're going to add some basic documentation to our project. Update your `models/schema.yml` file to include some descriptions, such as those below. models/schema.yml ```yaml version: 2 models: - name: customers description: One record per customer columns: - name: customer_id description: Primary key data_tests: - unique - not_null - name: first_order_date description: NULL when a customer has not yet placed an order. - name: stg_customers description: This model cleans up customer data columns: - name: customer_id description: Primary key data_tests: - unique - not_null - name: stg_orders description: This model cleans up order data columns: - name: order_id description: Primary key data_tests: - unique - not_null - name: status data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['placed', 'shipped', 'completed', 'return_pending', 'returned'] - name: customer_id data_tests: - not_null - relationships: arguments: to: ref('stg_customers') field: customer_id ``` ##### View in Catalog [Catalog](https://docs.getdbt.com/docs/explore/explore-projects.md) provides powerful tools to interact with your dbt projects, including documentation: 1. Run one of the following commands: * `dbt docs generate` if you're on dbt v1 * `dbt build` if you're on dbt v2 2. Click **Catalog** in the navigation menu to launch Catalog. 3. Catalog reflects **Production** by default. If your account has additional environments (for example, **Staging**), you can select them from the environment dropdown. [![Select an environment in Catalog.](/img/docs/collaborate/dbt-explorer/catalog-nav-dropdown.png?v=2 "Select an environment in Catalog.")](#)Select an environment in Catalog. 4. Select your project from the file tree. 5. Use the search bar or browse the resource list to find the `customers` model. 6. Click the model to view its details, including the descriptions you added. [![View your model's documentation and lineage in Catalog.](/img/docs/collaborate/dbt-explorer/example-model-details.png?v=2 "View your model's documentation and lineage in Catalog.")](#)View your model's documentation and lineage in Catalog. Catalog displays your model's description, column documentation, data tests, and lineage graph. You can also see which columns are missing documentation and track test coverage across your project. ##### View in Studio IDE You can view docs directly from the IDE if you're on **v1 Latest** or another version of dbt v1. Keep in mind that this is a legacy view and doesn't offer the same level of interactivity as Catalog. 1. In the IDE, run `dbt docs generate`. 2. From the navigation bar, click the **View docs** icon located to the right of the **branch name**. [![The View docs icon in the Studio IDE.](/img/docs/collaborate/dbt-explorer/docs-icon.png?v=2 "The View docs icon in the Studio IDE.")](#)The View docs icon in the Studio IDE. 3. From **Projects**, select your project name and expand the folders. 4. Click **models** > **marts** > **customers**. [![View your model's documentation in the legacy docs view.](/img/docs/collaborate/dbt-explorer/legacy-docs-view.png?v=2 "View your model's documentation in the legacy docs view.")](#)View your model's documentation in the legacy docs view. ###### FAQs How do I write long-form explanations in my descriptions? If you need more than a sentence to explain a model, you can: 1. Split your description over multiple lines using `>`. Interior line breaks are removed and Markdown can be used. This method is recommended for simple, single-paragraph descriptions: ```yml models: - name: customers description: > Lorem ipsum **dolor** sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. ``` 2. Split your description over multiple lines using `|`. Interior line breaks are maintained and Markdown can be used. This method is recommended for more complex descriptions: ```yml models: - name: customers description: | ### Lorem ipsum * dolor sit amet, consectetur adipisicing elit, sed do eiusmod * tempor incididunt ut labore et dolore magna aliqua. ``` 3. Use a [docs block](https://docs.getdbt.com/docs/build/documentation.md#using-docs-blocks) to write the description in a separate Markdown file. How do I access documentation in dbt Catalog? If you're using dbt to deploy your project and have a [Starter, Enterprise, or Enterprise+ plan](https://www.getdbt.com/pricing/), you can use Catalog to view your project's [resources](https://docs.getdbt.com/docs/build/projects.md) (such as models, tests, and metrics) and their lineage to gain a better understanding of its latest production state. Access Catalog in dbt by clicking the **Catalog** link in the navigation. Read-only users can access the documentation for your project — for license availability by plan, refer to [dbt pricing](https://www.getdbt.com/pricing). dbt developer plan and dbt v1 users can use [dbt Docs](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md#dbt-docs), which generates basic documentation but it doesn't offer the same speed, metadata, or visibility as Catalog. #### Commit your changes Now that you've built your customer model, you need to commit the changes you made to the project so that the repository has your latest code. **If you edited directly in the protected primary branch:**<br /> 1. Click the **Commit and sync git** button. This action prepares your changes for commit. 2. A modal titled **Commit to a new branch** will appear. 3. In the modal window, name your new branch `add-customers-model`. This branches off from your primary branch with your new changes. 4. Add a commit message, such as "Add customers model, tests, docs" and and commit your changes. 5. Click **Merge this branch to main** to add these changes to the main branch on your repo. **If you created a new branch before editing:**<br /> 1. Since you already branched out of the primary protected branch, go to **Version Control** on the left. 2. Click **Commit and sync** to add a message. 3. Add a commit message, such as "Add customers model, tests, docs." 4. Click **Merge this branch to main** to add these changes to the main branch on your repo. #### Deploy dbt Use dbt's Scheduler to deploy your production jobs confidently and build observability into your processes. You'll learn to create a deployment environment and run a job in the following steps. ##### Create a deployment environment 1. From the main menu, go to **Orchestration** > **Environments**. 2. Click **Create environment**. 3. In the **Name** field, write the name of your deployment environment. For example, "Production." 4. The **dbt version** will default to the latest available. We recommend all new projects run on the latest version of dbt. 5. Under **Deployment connection**, enter the name of the dataset you want to use as the target, such as "Analytics". This will allow dbt to build and work with that dataset. For some data warehouses, the target dataset may be referred to as a "schema". 6. Click **Save**. ##### Create and run a job Jobs are a set of dbt commands that you want to run on a schedule. For example, `dbt build`. As the `jaffle_shop` business gains more customers, and those customers create more orders, you will see more records added to your source data. Because you materialized the `customers` model as a table, you'll need to periodically rebuild your table to ensure that the data stays up-to-date. This update will happen when you run a job. 1. After creating your deployment environment, you should be directed to the page for a new environment. If not, select **Orchestration** from the main menu, then click **Jobs**. 2. Click **Create job** > **Deploy job**. 3. Provide a job name (for example, "Production run") and select the environment you just created. 4. Scroll down to the **Execution settings** section. 5. Under **Commands**, add this command as part of your job if you don't see it: * `dbt build` 6. Select the **Generate docs on run** option to automatically [generate updated project docs](https://docs.getdbt.com/docs/explore/build-and-view-your-docs.md) each time your job runs. 7. For this exercise, do *not* set a schedule for your project to run — while your organization's project should run regularly, there's no need to run this example project on a schedule. Scheduling a job is sometimes referred to as *deploying a project*. 8. Click **Save**, then click **Run now** to run your job. 9. Click the run and watch its progress under **Run summary**. 10. Once the run is complete, click **View Documentation** to see the docs for your project. Congratulations 🎉! You've just deployed your first dbt project! ###### FAQs What happens if one of my runs fails? If you're using dbt, we recommend setting up email and Slack notifications (`Account Settings > Notifications`) for any failed runs. Then, debug these runs the same way you would debug any runs in development. --- ### Quickstart for dbt and Databricks [Back to guides](https://docs.getdbt.com/guides.md) Platform Quickstart Databricks Beginner [Menu ]() #### Introduction In this quickstart guide, you'll learn how to use dbt with Databricks. It will show you how to: * Create a Databricks workspace. * Load sample data into your Databricks account. * Connect dbt to Databricks. * Take a sample query and turn it into a model in your dbt project. A model in dbt is a select statement. * Add tests to your models. * Document your models. * Schedule a job to run. Videos for you You can check out [dbt Fundamentals](https://learn.getdbt.com/courses/dbt-fundamentals) for free if you're interested in course learning with videos. ##### Prerequisites​ * You have a [dbt account](https://www.getdbt.com/signup/). * You have an account with a cloud service provider (such as AWS, GCP, and Azure) and have permissions to create an S3 bucket with this account. For demonstrative purposes, this guide uses AWS as the cloud service provider. ##### Related content * Learn more with [dbt Learn courses](https://learn.getdbt.com) * [CI jobs](https://docs.getdbt.com/docs/deploy/continuous-integration.md) * [Deploy jobs](https://docs.getdbt.com/docs/deploy/deploy-jobs.md) * [Job notifications](https://docs.getdbt.com/docs/deploy/job-notifications.md) * [Source freshness](https://docs.getdbt.com/docs/deploy/source-freshness.md) #### Create a Databricks workspace 1. Use your existing account or [sign up for a Databricks account](https://databricks.com/). Complete the form with your user information and click **Continue**. [![Sign up for Databricks](/img/databricks_tutorial/images/signup_form.png?v=2 "Sign up for Databricks")](#)Sign up for Databricks 2. On the next screen, select your cloud provider. This tutorial uses AWS as the cloud provider, but if you use Azure or GCP internally, please select your platform. The setup process will be similar. Do not select the **Get started with Community Edition** option, as this will not provide the required compute for this guide. [![Choose cloud provider](/img/databricks_tutorial/images/choose_provider.png?v=2 "Choose cloud provider")](#)Choose cloud provider 3. Check your email and complete the verification process. 4. After completing the verification processes, you will be brought to the first setup screen. Databricks defaults to the `Premium` plan and you can change the trial to `Enterprise` on this page. [![Choose Databricks Plan](/img/databricks_tutorial/images/choose_plan.png?v=2 "Choose Databricks Plan")](#)Choose Databricks Plan 5. Now, it's time to create your first workspace. A Databricks workspace is an environment for accessing all of your Databricks assets. The workspace organizes objects like notebooks, SQL warehouses, clusters, and more so into one place. Provide the name of your workspace, choose the appropriate AWS region, and click **Start Quickstart**. You might get the checkbox of **I have data in S3 that I want to query with Databricks**. You do not need to check this off for this tutorial. [![Create AWS resources](/img/databricks_tutorial/images/start_quickstart.png?v=2 "Create AWS resources")](#)Create AWS resources 6. By clicking on `Start Quickstart`, you will be redirected to AWS and asked to log in if you haven’t already. After logging in, you should see a page similar to this. [![Create AWS resources](/img/databricks_tutorial/images/quick_create_stack.png?v=2 "Create AWS resources")](#)Create AWS resources tip If you get a session error and don’t get redirected to this page, you can go back to the Databricks UI and create a workspace from the interface. All you have to do is click **create workspaces**, choose the quickstart, fill out the form and click **Start Quickstart**. 7. There is no need to change any of the pre-filled out fields in the Parameters. Just add in your Databricks password under **Databricks Account Credentials**. Check off the Acknowledgement and click **Create stack**. [![Parameters](/img/databricks_tutorial/images/parameters.png?v=2 "Parameters")](#)Parameters [![Capabilities](/img/databricks_tutorial/images/create_stack.png?v=2 "Capabilities")](#)Capabilities 8. Go back to the Databricks tab. You should see that your workspace is ready to use. [![A Databricks Workspace](/img/databricks_tutorial/images/workspaces.png?v=2 "A Databricks Workspace")](#)A Databricks Workspace 9. Now let’s jump into the workspace. Click **Open** and log into the workspace using the same login as you used to log into the account. #### Load data 1. Download these CSV files (the Jaffle Shop sample data) that you will need for this guide: * [jaffle\_shop\_customers.csv](https://dbt-tutorial-public.s3-us-west-2.amazonaws.com/jaffle_shop_customers.csv) * [jaffle\_shop\_orders.csv](https://dbt-tutorial-public.s3-us-west-2.amazonaws.com/jaffle_shop_orders.csv) * [stripe\_payments.csv](https://dbt-tutorial-public.s3-us-west-2.amazonaws.com/stripe_payments.csv) 2. First we need a SQL warehouse. Find the drop down menu and toggle into the SQL space. [![SQL space](/img/databricks_tutorial/images/go_to_sql.png?v=2 "SQL space")](#)SQL space 3. We will be setting up a SQL warehouse now. Select **SQL Warehouses** from the left hand side console. You will see that a default SQL Warehouse exists. 4. Click **Start** on the Starter Warehouse. This will take a few minutes to get the necessary resources spun up. 5. Once the SQL Warehouse is up, click **New** and then **File upload** on the dropdown menu. [![New File Upload Using Databricks SQL](/img/databricks_tutorial/images/new_file_upload_using_databricks_SQL.png?v=2 "New File Upload Using Databricks SQL")](#)New File Upload Using Databricks SQL 6. Let's load the Jaffle Shop Customers data first. Drop in the `jaffle_shop_customers.csv` file into the UI. [![Databricks Table Loader](/img/databricks_tutorial/images/databricks_table_loader.png?v=2 "Databricks Table Loader")](#)Databricks Table Loader 7. Update the Table Attributes at the top: * **data\_catalog** = hive\_metastore * **database** = default * **table** = jaffle\_shop\_customers * Make sure that the column data types are correct. The way you can do this is by hovering over the datatype icon next to the column name. * **ID** = bigint * **FIRST\_NAME** = string * **LAST\_NAME** = string [![Load jaffle shop customers](/img/databricks_tutorial/images/jaffle_shop_customers_upload.png?v=2 "Load jaffle shop customers")](#)Load jaffle shop customers 8. Click **Create** on the bottom once you’re done. 9. Now let’s do the same for `Jaffle Shop Orders` and `Stripe Payments`. [![Load jaffle shop orders](/img/databricks_tutorial/images/jaffle_shop_orders_upload.png?v=2 "Load jaffle shop orders")](#)Load jaffle shop orders [![Load stripe payments](/img/databricks_tutorial/images/stripe_payments_upload.png?v=2 "Load stripe payments")](#)Load stripe payments 10. Once that's done, make sure you can query the training data. Navigate to the `SQL Editor` through the left hand menu. This will bring you to a query editor. 11. Ensure that you can run a `select *` from each of the tables with the following code snippets. ```sql select * from default.jaffle_shop_customers select * from default.jaffle_shop_orders select * from default.stripe_payments ``` [![Query Check](/img/databricks_tutorial/images/query_check.png?v=2 "Query Check")](#)Query Check 12. To ensure any users who might be working on your dbt project has access to your object, run this command. ```sql grant all privileges on schema default to users; ``` #### Connect dbt to Databricks There are two ways to connect dbt to Databricks. The first option is Partner Connect, which provides a streamlined setup to create your dbt account from within your new Databricks trial account. The second option is to create your dbt account separately and build the Databricks connection yourself (connect manually). If you want to get started quickly, dbt Labs recommends using Partner Connect. If you want to customize your setup from the very beginning and gain familiarity with the dbt setup flow, dbt Labs recommends connecting manually. #### Set up the integration from Partner Connect note Partner Connect is intended for trial partner accounts. If your organization already has a dbt account, connect manually. Refer to [Connect to dbt manually](https://docs.databricks.com/partners/prep/dbt-cloud.html#connect-to-dbt-cloud-manually) in the Databricks docs for instructions. To connect dbt to Databricks using Partner Connect, do the following: 1. In the sidebar of your Databricks account, click **Partner Connect**. 2. Click the **dbt tile**. 3. Select a catalog from the drop-down list, and then click **Next**. The drop-down list displays catalogs you have read and write access to. If your workspace isn't `<UC>-enabled`, the legacy Hive metastore (`hive_metastore`) is used. 4. If there are SQL warehouses in your workspace, select a SQL warehouse from the drop-down list. If your SQL warehouse is stopped, click **Start**. 5. If there are no SQL warehouses in your workspace: 1. Click **Create warehouse**. A new tab opens in your browser that displays the **New SQL Warehouse** page in the Databricks SQL UI. 2. Follow the steps in [Create a SQL warehouse](https://docs.databricks.com/en/sql/admin/create-sql-warehouse.html#create-a-sql-warehouse) in the Databricks docs. 3. Return to the Partner Connect tab in your browser, and then close the **dbt tile**. 4. Re-open the **dbt tile**. 5. Select the SQL warehouse you just created from the drop-down list. 6. Select a schema from the drop-down list, and then click **Add**. The drop-down list displays schemas you have read and write access to. You can repeat this step to add multiple schemas. Partner Connect creates the following resources in your workspace: * A Databricks service principal named **DBT\_CLOUD\_USER**. * A Databricks personal access token that is associated with the **DBT\_CLOUD\_USER** service principal. Partner Connect also grants the following privileges to the **DBT\_CLOUD\_USER** service principal: * (Unity Catalog) **USE CATALOG**: Required to interact with objects within the selected catalog. * (Unity Catalog) **USE SCHEMA**: Required to interact with objects within the selected schema. * (Unity Catalog) **CREATE SCHEMA**: Grants the ability to create schemas in the selected catalog. * (Hive metastore) **USAGE**: Required to grant the **SELECT** and **READ\_METADATA** privileges for the schemas you selected. * **SELECT**: Grants the ability to read the schemas you selected. * (Hive metastore) **READ\_METADATA**: Grants the ability to read metadata for the schemas you selected. * **CAN\_USE**: Grants permissions to use the SQL warehouse you selected. 7. Click **Next**. The **Email** box displays the email address for your Databricks account. dbt Labs uses this email address to prompt you to create a trial dbt account. 8. Click **Connect to dbt**. A new tab opens in your web browser, which displays the getdbt.com website. 9. Complete the on-screen instructions on the getdbt.com website to create your trial dbt account. #### Set up a dbt managed repository When you develop in dbt, you can leverage [Git](https://docs.getdbt.com/docs/platform/git/git-version-control.md) to version control your code. To connect to a repository, you can either set up a dbt-hosted [managed repository](https://docs.getdbt.com/docs/platform/git/managed-repository.md) or directly connect to a [supported git provider](https://docs.getdbt.com/docs/platform/git/connect-github.md). Managed repositories are a great way to trial dbt without needing to create a new repository. In the long run, it's better to connect to a supported git provider to use features like automation and [continuous integration](https://docs.getdbt.com/docs/deploy/continuous-integration.md). To set up a managed repository: 1. Under "Setup a repository", select **Managed**. 2. Type a name for your repo such as `bbaggins-dbt-quickstart` 3. Click **Create**. It will take a few seconds for your repository to be created and imported. 4. Once you see the "Successfully imported repository," click **Continue**. #### Initialize your dbt project​ and start developing Now that you have a repository configured, you can initialize your project and start development in dbt: 1. Click **Start developing in the Studio IDE**. It might take a few minutes for your project to spin up for the first time as it establishes your git connection, clones your repo, and tests the connection to the warehouse. 2. Above the file tree to the left, click **Initialize dbt project**. This builds out your folder structure with example models. 3. Make your initial commit by clicking **Commit and sync**. Use the commit message `initial commit` and click **Commit**. This creates the first commit to your managed repo and allows you to open a branch where you can add new dbt code. 4. You can now directly query data from your warehouse and execute `dbt run`. You can try this out now: * Click **+ Create new file**, add this query to the new file, and click **Save as** to save the new file: ```sql select * from default.jaffle_shop_customers ``` * In the command line bar at the bottom, enter `dbt run` and click **Enter**. You should see a `dbt run succeeded` message. #### Build your first model You have two options for working with files in the Studio IDE: * Create a new branch (recommended) — Create a new branch to edit and commit your changes. Navigate to **Version Control** on the left sidebar and click **Create branch**. * Edit in the protected primary branch — If you prefer to edit, format, or lint files and execute dbt commands directly in your primary git branch. The Studio IDE prevents commits to the protected branch, so you will be prompted to commit your changes to a new branch. Name the new branch `add-customers-model`. 1. Click the **...** next to the `models` directory, then select **Create file**. 2. Name the file `customers.sql`, then click **Create**. 3. Copy the following query into the file and click **Save**. ```sql with customers as ( select id as customer_id, first_name, last_name from jaffle_shop_customers ), orders as ( select id as order_id, user_id as customer_id, order_date, status from jaffle_shop_orders ), customer_orders as ( select customer_id, min(order_date) as first_order_date, max(order_date) as most_recent_order_date, count(order_id) as number_of_orders from orders group by 1 ), final as ( select customers.customer_id, customers.first_name, customers.last_name, customer_orders.first_order_date, customer_orders.most_recent_order_date, coalesce(customer_orders.number_of_orders, 0) as number_of_orders from customers left join customer_orders using (customer_id) ) select * from final ``` 4. Enter `dbt run` in the command prompt at the bottom of the screen. You should get a successful run and see the three models. Later, you can connect your business intelligence (BI) tools to these views and tables so they only read cleaned up data rather than raw data in your BI tool. ###### FAQs How can I see the SQL that dbt is running? To check out the SQL that dbt is running, you can look in: * dbt: * Within the run output, click on a model name, and then select "Details" * dbt v1: * The `target/compiled/` directory for compiled `select` statements * The `target/run/` directory for compiled `create` statements * The `logs/dbt.log` file for verbose logging. How did dbt choose which schema to build my models in? By default, dbt builds models in your target schema. To change your target schema: * If you're developing in **dbt**, these are set for each user when you first use a development environment. * If you're developing with **dbt v1**, this is the `schema:` parameter in your `profiles.yml` file. If you wish to split your models across multiple schemas, check out the docs on [using custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md). Note: on BigQuery, `dataset` is used interchangeably with `schema`. Do I need to create my target schema before running dbt? Nope! dbt will check if the schema exists when it runs. If the schema does not exist, dbt will create it for you. If I rerun dbt, will there be any downtime as models are rebuilt? Nope! The SQL that dbt generates behind the scenes ensures that any relations are replaced atomically (i.e. your business users won't experience any downtime). The implementation of this varies on each warehouse, check out the [logs](https://docs.getdbt.com/faqs/Runs/checking-logs.md) to see the SQL dbt is executing. What happens if the SQL in my query is bad or I get a database error? If there's a mistake in your SQL, dbt will return the error that your database returns. ```shell $ dbt run --select customers Running with dbt=1.9.0 Found 3 models, 9 tests, 0 snapshots, 0 analyses, 133 macros, 0 operations, 0 seed files, 0 sources 14:04:12 | Concurrency: 1 threads (target='dev') 14:04:12 | 14:04:12 | 1 of 1 START view model dbt_alice.customers.......................... [RUN] 14:04:13 | 1 of 1 ERROR creating view model dbt_alice.customers................. [ERROR in 0.81s] 14:04:13 | 14:04:13 | Finished running 1 view model in 1.68s. Completed with 1 error and 0 warnings: Database Error in model customers (models/customers.sql) Syntax error: Expected ")" but got identifier `your-info-12345` at [13:15] compiled SQL at target/run/jaffle_shop/customers.sql Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1 ``` Any models downstream of this model will also be skipped. Use the error message and the [compiled SQL](https://docs.getdbt.com/faqs/Runs/checking-logs.md) to debug any errors. #### Change the way your model is materialized One of the most powerful features of dbt is that you can change the way a model is materialized in your warehouse, simply by changing a configuration value. You can change things between tables and views by changing a keyword rather than writing the data definition language (DDL) to do this behind the scenes. By default, everything gets created as a view. You can override that at the directory level so everything in that directory will materialize to a different materialization. 1. Edit your `dbt_project.yml` file. * Update your project `name` to: dbt\_project.yml ```yaml name: 'jaffle_shop' ``` * Configure `jaffle_shop` so everything in it will be materialized as a table; and configure `example` so everything in it will be materialized as a view. Update your `models` config in the project YAML file to: dbt\_project.yml ```yaml models: jaffle_shop: +materialized: table example: +materialized: view ``` * Click **Save**. 2. Enter the `dbt run` command. Your `customers` model should now be built as a table! info To do this, dbt had to first run a `drop view` statement (or API call on BigQuery), then a `create table as` statement. 3. Edit `models/customers.sql` to override the `dbt_project.yml` for the `customers` model only by adding the following snippet to the top, and click **Save**: models/customers.sql ```sql {{ config( materialized='view' ) }} with customers as ( select id as customer_id ... ) ``` 4. Enter the `dbt run` command. Your model, `customers`, should now build as a view. * BigQuery users need to run `dbt run --full-refresh` instead of `dbt run` to full apply materialization changes. 5. Enter the `dbt run --full-refresh` command for this to take effect in your warehouse. ##### FAQs What materializations are available in dbt? dbt ships with five built-in materializations: `view`, `table`, `incremental`, `ephemeral`, and `materialized_view`. Check out the documentation on [materializations](https://docs.getdbt.com/docs/build/materializations.md) for more information on each of these options. You can also create your own [custom materializations](https://docs.getdbt.com/guides/create-new-materializations.md). This is an advanced feature of dbt. Which materialization should I use for my model? Start out with views, and then change models to tables when required for performance reasons (i.e. downstream queries have slowed). Check out the [docs on materializations](https://docs.getdbt.com/docs/build/materializations.md) for advice on when to use each materialization. What model configurations exist? You can also configure: * [tags](https://docs.getdbt.com/reference/resource-configs/tags.md) to support easy categorization and graph selection * [custom schemas](https://docs.getdbt.com/reference/resource-properties/schema.md) to split your models across multiple schemas * [aliases](https://docs.getdbt.com/reference/resource-configs/alias.md) if your view/table name should differ from the filename * Snippets of SQL to run at the start or end of a model, known as [hooks](https://docs.getdbt.com/docs/build/hooks-operations.md) * Warehouse-specific configurations for performance (e.g. `sort` and `dist` keys on Redshift, `partitions` on BigQuery) Check out the docs on [model configurations](https://docs.getdbt.com/reference/model-configs.md) to learn more. #### Delete the example models You can now delete the files that dbt created when you initialized the project: 1. Delete the `models/example/` directory. 2. Delete the `example:` key from your `dbt_project.yml` file, and any configurations that are listed under it. dbt\_project.yml ```yaml # before models: jaffle_shop: +materialized: table example: +materialized: view ``` dbt\_project.yml ```yaml # after models: jaffle_shop: +materialized: table ``` 3. Save your changes. ###### FAQs How do I remove deleted models from my data warehouse? If you delete a model from your dbt project, dbt does not automatically drop the relation from your schema. This means that you can end up with extra objects in schemas that dbt creates, which can be confusing to other users. (This can also happen when you switch a model from being a view or table, to ephemeral) When you remove models from your dbt project, you should manually drop the related relations from your schema. I got an "unused model configurations" error message, what does this mean? You might have forgotten to nest your configurations under your project name, or you might be trying to apply configurations to a directory that doesn't exist. Check out this [article](https://discourse.getdbt.com/t/faq-i-got-an-unused-model-configurations-error-message-what-does-this-mean/112) to understand more. #### Build models on top of other models As a best practice in SQL, you should separate logic that cleans up your data from logic that transforms your data. You have already started doing this in the existing query by using common table expressions (CTEs). Now you can experiment by separating the logic out into separate models and using the [ref](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md) function to build models on top of other models: [![The DAG we want for our dbt project](/img/dbt-dag.png?v=2 "The DAG we want for our dbt project")](#)The DAG we want for our dbt project 1. Create a new SQL file, `models/stg_customers.sql`, with the SQL from the `customers` CTE in our original query. 2. Create a second new SQL file, `models/stg_orders.sql`, with the SQL from the `orders` CTE in our original query. models/stg\_customers.sql ```sql select id as customer_id, first_name, last_name from jaffle_shop_customers ``` models/stg\_orders.sql ```sql select id as order_id, user_id as customer_id, order_date, status from jaffle_shop_orders ``` 3. Edit the SQL in your `models/customers.sql` file as follows: models/customers.sql ```sql with customers as ( select * from {{ ref('stg_customers') }} ), orders as ( select * from {{ ref('stg_orders') }} ), customer_orders as ( select customer_id, min(order_date) as first_order_date, max(order_date) as most_recent_order_date, count(order_id) as number_of_orders from orders group by 1 ), final as ( select customers.customer_id, customers.first_name, customers.last_name, customer_orders.first_order_date, customer_orders.most_recent_order_date, coalesce(customer_orders.number_of_orders, 0) as number_of_orders from customers left join customer_orders using (customer_id) ) select * from final ``` 4. Execute `dbt run`. This time, when you performed a `dbt run`, separate views/tables were created for `stg_customers`, `stg_orders` and `customers`. dbt inferred the order to run these models. Because `customers` depends on `stg_customers` and `stg_orders`, dbt builds `customers` last. You do not need to explicitly define these dependencies. ###### FAQs How do I run one model at a time? To run one model, use the `--select` flag (or `-s` flag), followed by the name of the model: ```shell $ dbt run --select customers ``` Check out the [model selection syntax documentation](https://docs.getdbt.com/reference/node-selection/syntax.md) for more operators and examples. Do ref-able resource names need to be unique? Within one project: yes! To build dependencies between resources (such as models, seeds, and snapshots), you need to use the `ref` function, and pass in the resource name as an argument. dbt uses that resource name to uniquely resolve the `ref` to a specific resource. As a result, these resource names need to be unique, *even if they are in distinct folders*. A resource in one project can have the same name as a resource in another project (installed as a dependency). dbt uses the project name to uniquely identify each resource. We call this "namespacing." If you `ref` a resource with a duplicated name, it will resolve to the resource within the same namespace (package or project), or raise an error because of an ambiguous reference. Use [two-argument `ref`](https://docs.getdbt.com/reference/dbt-jinja-functions/ref.md#ref-project-specific-models) to disambiguate references by specifying the namespace. Those resource will still need to land in distinct locations in the data warehouse. Read the docs on [custom aliases](https://docs.getdbt.com/docs/build/custom-aliases.md) and [custom schemas](https://docs.getdbt.com/docs/build/custom-schemas.md) for details on how to achieve this. As I create more models, how should I keep my project organized? What should I name my models? There's no one best way to structure a project! Every organization is unique. If you're just getting started, check out how we (dbt Labs) [structure our dbt projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview.md). #### Add tests to your models Adding [data tests](https://docs.getdbt.com/docs/build/data-tests.md) to a project helps validate that your models are working correctly. To add data tests to your project: 1. Create a new YAML file in the `models` directory, named `models/schema.yml` 2. Add the following contents to the file: models/schema.yml ```yaml version: 2 models: - name: customers columns: - name: customer_id data_tests: - unique - not_null - name: stg_customers columns: - name: customer_id data_tests: - unique - not_null - name: stg_orders columns: - name: order_id data_tests: - unique - not_null - name: status data_tests: - accepted_values: arguments: # available in v1.10.5 and higher. Older versions can set the <argument_name> as the top-level property. values: ['placed', 'shipped', 'completed', 'return_pending', 'returned'] - name: customer_id data_tests: - not_null - relationships: arguments: to: ref('stg_customers') field: customer_id ``` 3. Run `dbt test`, and confirm that all your tests passed. When you run `dbt test`, dbt iterates through your YAML files, and constructs a query for each test. Each query will return the number of records that fail the test. If this number is 0, then the test is successful. ###### FAQs What tests are available for me to use in dbt? Can I add my own custom tests? Out of the box, dbt ships with the following data tests: * `unique` * `not_null` * `accepted_values` * `relationships` (for example, referential integrity) You can also write your own [custom generic tests](https://docs.getdbt.com/docs/build/data-tests.md#generic-data-tests). Some additional generic tests have been open-sourced in the [dbt-utils package](https://github.com/dbt-labs/dbt-utils#generic-tests). Check out the docs on [packages](https://docs.getdbt.com/docs/build/packages.md) to learn how to make these tests available in your project. How do I test one model at a time? Running tests on one model looks very similar to running a model: use the `--select` flag (or `-s` flag), followed by the name of the model: ```shell dbt test --select customers ``` Check out the [model selection syntax documentation](https://docs.getdbt.com/reference/node-selection/syntax.md) for full syntax, and [test selection examples](https://docs.getdbt.com/reference/node-selection/test-selection-examples.md) in particular. One of my tests failed, how can I debug it? To debug a failing test, find the SQL that dbt ran by: * dbt: * Within the test output, click on the failed test, and then select "Details". * dbt v1: * Open the file path returned as part of the error message. * Navigate to the `target/compiled/schema_tests` directory for all compiled test queries. Copy the SQL into a query editor (in dbt, you can paste it into a new `Statement`), and run the query to find the records that failed. Does my test file need to be named \`schema.yml\`? No! You can name this file whatever you want (including `whatever_you_want.yml`), so long as: * The file is in your `models/` directory¹ * The file has `.yml` extension Check out the [docs](https://docs.getdbt.com/reference/configs-and-properties.md) for more information. ¹If you're declaring properties for seeds, snapshots, or macros, you can also place this file in the related directory — `seeds/`, `snapshots/` and `macros/` respectively. Why do model and source YAML files always start with \`version: 2\`? Once upon a time, the structure of these `.yml` files was very different (s/o to anyone who was using dbt back then!). Adding `version: 2` allowed us to make this structure more extensible. From [dbt v1.5](<https://docs.getdbt.com/docs/dbt-versions/dbt-upgrade/Older versions/upgrading-to-v1.5.md#quick-hits>), the top-level `version:` key is optional in all resource YAML files. If present, only `version: 2` is supported. Also starting in v1.5, both the [`config-version: 2`](https://docs.getdbt.com/reference/project-configs/config-version.md) and the top-level `version:` key in the `dbt_project.yml` are optional. Resource YAML files do not currently require this config. We only support `version: 2` if it's specified. Although we do not expect to update YAML files to `version: 3` soon, having this config will make it easier for us to introduce new structures in the future What data tests should I add to my project? We recommend that every model has a data test on a primary key, that is, a column that is `unique` and `not_null`. We also recommend that you test any assumptions on your source data. For example, if you believe that your payments can only be one of three payment methods, you should test that assumption regularly — a new payment method may introduce logic errors in your SQL. In advanced dbt projects, we recommend using [sources](https://docs.getdbt.com/docs/build/sources.md) and running these source data-integrity tests against the sources rather than models. When should I run my data tests? You should run your data tests whenever you are writing new code (to ensure you haven't broken any existing models by changing SQL), and whenever you run your transformations in production (to ensure that your assumptions about your source data are still valid). #### Document your models Adding [documentation](https://docs.getdbt.com/docs/build/documentation.md) to your project allows you to describe your models in rich detail, and share that information with your team. Here, we're going to add some basic documentation to our project. Update your `models/schema.yml` file to include some descriptions, such as those below. models/schema.yml ```yaml version: 2 models: - name: customers description: One record per customer columns: - name: customer_id description: Primary key data_tests: - unique - not_null - name: first_order_date description: NULL when a customer has not yet placed an order. - name: stg_customers description: This model cleans up customer data columns: - name: customer_id description: Primary key data_tests: - unique - not_null - name: stg_orders description: This model cleans up order data columns: - name: order_id description: Primary key data_tests: - unique - not_null - name: status data_tests: - accepted_valu